Discuss easing compile times by splitting up types module

[MichaelXavier]

Carrying over discussion from #221 @bitemyapp

I've also noticed the types module takes an extremely long time to compile and it sounds like it is teetering on the edge of making CI not work correctly. First, do we know that splitting up the modules but keeping all the same code will even help? Doesn't the compiler still have to derive all those instances in CI anyway?

[bitemyapp]

https://ghc.haskell.org/trac/ghc/wiki/Performance/Compiler

https://ghc.haskell.org/trac/ghc/ticket/5642

Then from:

https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/GenericDeriving

Also: tfausak/tfausak.github.io#127

My understanding that the more work GHC had to do in a single module, the slower the build got overall. I'm going to ask some folks that I think have kicked this can around and see if my thought that we could split up the modules to save compile-time was misbegotten or not.

[bitemyapp]

Part of the thinking here is that there's a lot of excessive allocation and retained heap for compiling one-big-module and that some speed benefits would be obtained from splitting that up in addition to whatever computational nastiness arises from having all the Generics derived and used in one big module.

I'm willing to eat the risk of wasted time in running out a branch for this and comparing the compile times.

[bitemyapp]

Follow-up thought: TH is reportedly much faster than Generics for almost all purposes of deriving class instances.

Example: tfausak/rattletrap#58 (comment)

Taylor's been very helpful in all this btw.

Another option I am entertaining: just write them out.

[MichaelXavier]

I'm pretty sure I added all the Generic instances not for anything to do with the library itself but to use generics SOP to derive arbitrary instances to smoke test our JSON instances. So that's actually kind of a bad tradeoff that everyone installing the package, whether or not they'll actually run tests has to pay this big cost for Generics. And if you'll recall, I went down the road of generics for Arbitrary because the author of QuickCheck disagrees on some fundamental level that Arbitrary instances should ever be derived.

So I can think of some additional approaches here:

1. Drop Generic deriving entirely from the Types modules. Allow orphans in the test module and standalone derive them or whatever there. They are then immediately used to derive Arbitrary. This means still probably slow CI builds but for the most part everyone else should get faster compile times.
2. I enter a trance, drop all Generic deriving from Types, and rework the test suite to not use Generics to get Arbitrary. Fair warning, if I'm going to rewrite all that code anyways and not try to save on boilerplate, I would probably toss out QuickCheck in favor of hedgehog which doesn't even use typeclasses for its generators and as a result is a lot more straightforward to fine tune the test suite for reasonable performance and coverage than a typeclass approach.

[bitemyapp]

I've been working on http://hackage.haskell.org/package/hedgehog-checkers and http://hackage.haskell.org/package/hedgehog-checkers-lens — I think I'd prefer #2, which is also something I'm willing to help out with.

I'll start a branch for us tonight if that's all right.

[AlexeyRaga]

We are experiencing issues with building Bloodhound in CircleCI, although the issues are not around compilation time (yes, it is annoying but whatever) but around memory usage.
Basically I cannot build our project that uses the latest Bloodhound right now:

We are happy to do the work and split modules if you are willing to accept this PR?