Sharing types between frontend (ghcjs) and backend (ghc)

[imalsogreg]

Groundhog's encoding of foreign keys makes relationships between tables typesafe; the price to pay is yaml parsing (which depends on a c library), code generation with template haskell, and injecting groundhog-specific type constructors (i.e. DefaultKey) into the user's types.

This makes it hard to a library of application types with few dependencies - something we like to do if we want to share types between backend server compiled by ghc and frontend application compiled by ghcjs.

@mightybyte and I have been comparing solutions to this problem - I wonder if you have your own ideas of how to fix this?

CPP + custom type family in ghcjs case

In user code, chop out mkPersist ghCodegen [groundhog| ... |] from the code with CPP when the compiler is GHCJS, and instead include type instance DefaultKey MyType = Int64. Somewhere at the top-level define type family DefaultKey a :: * This works well, but is a bit of a burden on user's code.

Parse YAML description in ghcjs

By using a different yaml parser when the compiler is ghcjs, the yaml can be parsed and template haskell can then happily make the associated types. groundhog and groundhog-th can themselves be compiled with ghcjs, as can the user's code with the quasiquotes in it. I tried this in a patch to groundhog (master...imalsogreg:ghcjs) that relies on a small, not-yet-on-hackage ghcjs yaml parser binding (https://github.com/imalsogreg/yaml-ghcjs). groundhog-postgres (and probably the other backends) still can't be compiled though - they fail at the linking phase because they depend on a c library. So there is no way to get the captive Postgresql newtype in the client code, which we sometimes need in order to turn Key a BackendSpecific into Int64 and back. My workaround here is to make a NullDatabese empty data type and DbDescriptor instance for it, and use those in the client code to turn Keys into Ints.

Those are a couple of options - if you play with ghcjs, do you have any alternative ideas here, or a preference to go forward? Would you be open to pull requests that introduce CPP into groundhog-th to handle the ghcjs case? Thanks for reading a long message!

[lykahb]

Groundhog is not tied to YAML although now it is the standard way to use it. Function mkPersist needs just PersistDefinitions. You can pass it as just as a Haskell value https://github.com/lykahb/groundhog/blob/master/examples/withoutQuasiQuotes.hs. Surely that use case can be improved with default values and maybe lenses.

The current YAML library gives nice error messages with exact position. It is a high priority when considering a parsing library. I am open to including other ways to specify dependencies - nicer Haskell for 'PersistDefinitions', JSON, etc. - as long as they share the similar names and structure which would keep translation straightforward. It would be nice if CPP in groundhog-th is not tied to ghcjs at all. Or if there were a Haskell library that does not use C bindings. Speed does not matter in this case. Do I understand correctly that the only thing that prevents you from using TH is YAML and if you get TH working you do not need to define DefaultKey yourself?

NullDatabase trick is ok. Actually I think that I will remove proxy from to/from*PersistValue functions and deal with BackendSpecific keys in another way.

[mightybyte]

I like the sounds of @imalsogreg's second solution. The core problem here is that in a web app with GHCJS you usually will have some version of the following code structure:

project/
|-  backend/
|-  frontend/
|-  shared/

The backend and frontend directories each contain cabal files for building the backend (with GHC) and the frontend (with GHCJS). Both projects also depend on the code in shared. You usually want the groundhog data types to live in shared because they will probably also be used with JSON serialization to send data between the client and server. You don't want the front end to depend on groundhog because it doesn't really make much sense, and because the C bindings can't be built anyway.

In the project I'm working on we have been using the CPP + type family solution. It works ok, but is frustrating and not an obvious solution for less experienced Haskell programmers. I think the second solution would work great if we create a separate package (maybe called groundhog-ghcjs) that has its own GHCJS-specific definition for mkPersist. The backend would depend on groundhog and groundhog-th, while the front end would depend on groundhog-ghcjs. Having the groundhog-ghcjs package eliminates the need for any GHCJS-specific code in groundhog-th. The best case scenario would be if groundhog-ghcjs exports all the modules exported by groundhog and groundhog-th so there is no need for any CPP at all! But if that's not possible I think a little CPP around module imports would be ok.

[lykahb]

I see... So if shared references DefaultKey or just Key ghcjs needs to build groundhog.

[mightybyte]

No, I think groundhog-ghcjs would export its own DefaultKey and Key so GHCJS does not need to build groundhog.

It might be necessary to create one more package groundhog-types that defines anything used by both groundhog-th and groundhog-ghcjs (and maybe groundhog). After a brief look it seems like groundhog-types would contain CodegenConfig, PersistDefinitions, and maybe a few things from groundhog that would be common to both.

[lykahb]

Groundhog-ghcjs would be quite a heavyweight solution to this problem. I would prefer to keep the packages as is with perhaps some CPP.

[lykahb]

The comment about the DefaultKey above was referring to the current state.

[mightybyte]

Yes, this does sound a bit heavy. But I think it would create a fantastic user experience.

We've used this pattern before back when we were using haste. We ended up creating a string-compat package that allowed haste code to use backend code by exporting its own version of text and bytestring (https://github.com/Soostone/string-compat/blob/master/string-compat.cabal).

[imalsogreg]

shared references DefaultKey because we might have types that have foreign keys embedded in them. In that case, I follow the pattern in the documentation, where that is a DefaultKey OtherType.

So here is another solution, as long as we are just brainstorming: any user who wants their type to be compiled by ghcjs should avoid DefaultKey. It's possible to replaces all uses of DefaultKey by something like Key OtherType BackendSpecific or something like that, in most cases?

[mightybyte]

@imalsogreg Doesn't that just shift the problem from DefaultKey to Key?

[lykahb]

@imalsogreg Yes, it is totally possible. It will be less flexible and more verbose though.

In an email @mightybyte showed a snippet

#ifdef GHCJS
type family DefaultKey a :: *
#else
import Database.Groundhog (DefaultKey)
#endif

I think that we can avoid defining instances for each datatype with

#ifdef GHCJS
type DefaultKey a u = Int64
#else
import Database.Groundhog (DefaultKey)
#endif

[lykahb]

Though not every key is Int64. You can have a unique key with string or even a composite key.

[mightybyte]

Yeah, that might work. The reason we used the type family was just because that matched most closely with what groundhog was doing.

[mightybyte]

I just threw together something that might pass for a minimal compatibility package. What do you think?

https://github.com/mightybyte/groundhog-ghcjs

[lykahb]

Thank you. It is somewhat hacky but if it solves the problem it's great.

[mightybyte]

I don't know yet how well it will work. We'll try it in some real projects and see...

This solution should be good when all your keys are Int64. If you have different key types, then we would need to expand the library with a version of mkPersist that properly figures out what type to use.

[imalsogreg]

The solution here is that the user would put CPP in their cabal file to conditionally bring in groundhog-ghcjs instead of groundhog-th and groundhog, right? This is a nice, pretty lightweight solution for the shared types problem. My only criticism is that, if anyone wanted to use groundhog with ghcjs down the road, the package name may be misleading, as people would expect that they're getting all of the TH-built types, when really they are getting an empty splice.

[mightybyte]

You don't even need to do anything conditionally in the cabal file. The frontend cabal file just depends on groundhog-ghcjs, while the backend cabal file depends on groundhog and groundhog-th.

I can certainly change the package name. Maybe something like groundhog-ghcjs-shim?

[imalsogreg]

Ah, ok. For a use case like ours I see that would work. Only if you wanted to put the common package on hackage, then you would have to pick a concrete groundhog-{th | ghcjs} dependency in common.cabal. If we're just sym-linking to it rather that treating it as a bona-fide package, then only frontend- or backend ever refers to common. With groundhog-ghcjs-shim I'm 100% happy.

[lykahb]

Does ghcjs often require shims? If a datatype has Text, ByteString or JSON Value there may be some C involved too.

[lykahb]

Would we still need groundhog-ghcjs if groundhog-th does not have C library dependency?

[ryantrinkle]

I think that if it did not have a C dependency (or if there were appropriate C shims in the ghcjs shims library), it would be fine. I would prefer to use code that's as similar as possible between ghc and ghcjs, personally.

[imalsogreg]

Yep, exactly. ghcjs comes with its own base that includes replacements for Text and ByteString https://github.com/ghcjs/ghcjs-base. There's somehow an js replacement for gmp. C depending libraries generally require some kind of shim, yep.

[lykahb]

Can we avoid groundhog-ghcjs and have a YAML shim?

[imalsogreg]

@lykahb Correct - the c yaml parser is the only place where groundhog-th breaks under ghcjs. groundhog compiles as is. groundhog-postgresql won't compile at all. We could have a fully functioning groundhog-th including yaml parsing, if we CPP out the calls to c. This is 'option 2' in my original message.

[imalsogreg]

It would make sense to put the ghcjs yaml shim in the haskell yaml package itself, except that the yaml package provides a better interface with better error reporting than I expect I can find in a js library. I could be wrong there though.

[lykahb]

Right, groundhog-th does not use C directly. You can open a ticket at https://github.com/snoyberg/yaml/

[andrewthad]

Just stumbled across this issue while struggling with a similar problem. I have my model defined in one library that gets imported by two other projects: and ghcjs-compiled client and a ghc-compiled server. But I don't really want the client to incur a groundhog dependency at all. I'm doing to little bit of manual marshaling of auto-increment primary keys to make this possible, which I don't mind, but there is no way to get foreign keys to work. It would be nice if I could do something like this:

- entity: Foo
  constructors:
    - name: Foo
      foreigns:
        - name: foo_bar_fk
          target: Bar
          fields:
            - [bar, identifier]

And the foreigns clause would have no effect on any of the generated code except that the migration would now contain the foreign key constraint as well.

[lykahb]

@andrewthad
The foreign keys can be defined with the tables
https://github.com/lykahb/groundhog/blob/master/groundhog-th/Database/Groundhog/TH.hs#L496
If this is a composite key, it must be in a single record field. Without the groundhog dependency that may be a tuple or any other data type that implements EmbeddedField.

[mchaver]

@andrewthad
@lykahb
I think it would be nice if there were a solution to allow auto-increment for an existing field in a product type. I was thinking when you do an insert you provide a dummy value for the id, it gets ignored and allows the db to handle it the value. When you select a row then the id is provided in the appropriate id field.

[imalsogreg]

There's now a pure haskell yaml parser: http://hackage.haskell.org/package/HsYAML-0.1.0.0