Introduce an alternative to .cabal

[ndmitchell]

There are lots of nasty things in .cabal files, including specifying libraries multiple times, a custom syntax (rather than YAML), a flag mechanism that works backwards etc. Once stack can do stack upload then you could take the stack-friendly alternative and produce a .cabal file automatically. Stack would have to be able to work with both its simplified/improved format and the existing Cabal format.

Not sure if this is a good idea, but it occurred to me, so I thought I'd write it down (at the very least so you can point at this ticket and the reasons its a bad idea when the next person asks).

[gregwebs]

stack upload assume a cabal file, so I don't know what it has to do with generating one.

YAML does not seem like a good choice for expressing things such a flag conditional. Perhap you have an approach in mind?

In general right now this idea is lacking any detail on what a benefit would be. Examples would help.

[ndmitchell]

Let me rephrase: Perhaps stack could have a native format that was different to .cabal. If so, Stack could automatically convert it to .cabal for upload. Such a file format would have the obvious drawback of being less compatible, but since it is designed from scratch, could have some benefits:

• It could use an existing syntax such as YAML, to make parsing easier for a variety of tools without depending on the Cabal library.
• The flags in Cabal are often quite unwieldy - they are inferred and make basic things much more complex than they should be. Stack could go with a more traditional semantics closer to #ifdef and then convert these to Cabal flags later on.
• Often Cabal requires listing pieces multiple times, in both libraries and executable sections.
• Things like test executables could be given additional arguments.
• You could omit the list of hidden modules to include in a .cabal file and instead infer it from the build graph.

[gregwebs]

It could use an existing syntax such as YAML, to make parsing easier for a variety of tools without depending on the Cabal library.

Unfortunately because in the general case .cabal files will need to be supported it may be hard to reap this benefit

The flags in Cabal are often quite unwieldy - they are inferred and make basic things much more complex than they should be. Stack could go with a more traditional semantics closer to #ifdef and then convert these to Cabal flags later on.

Sorry for being dense, I would still love to see a more concrete example to understand this better.

Things like test executables could be given additional arguments.

Probably this could be added to the existing .cabal file system?

Often Cabal requires listing pieces multiple times, in both libraries and executable sections.

You could omit the list of hidden modules to include in a .cabal file and instead infer it from the build graph.

Sounds good.

There are existing package managers (npm and the javascript ones are what come to mind first) that use JSON as their format. YAML is really just JSON for humans instead of computers, so I view them two as equivalent. So we can look to these for examples of how to do project configuration.

[snoyberg]

To throw in a justification or two from the implementation side: relying on the Cabal library means that we're subject to all the bugs and quirks of the Cabal library, of which there are many. If we have a file format like this, not only can we make the format itself more user friendly, but stack can do more intelligent things about how it builds (e.g., no need to reconfigure in order to build your test suites). It will also theoretically make builds faster, as stack can do more intelligent parallelization.

If you want to go really crazy: stack could use GHC as a persistent process and hand it files to compile, which would allow GHC to keep .hi files loaded in memory between package builds, which has the potential to dramatically speed up compiles and make an autorecompile mode which is truly wonderful.

All this, if we just get rid of Cabal :)

[NathanHowell]

In our codebase we have hundreds of packages, so there is a huge huge amount of boilerplate and duplication between .cabal files. Having some notion of import or #include would be nice. I was looking at using an Applicative eDSL an alternative to .cabal files (with a "render as .cabal" evaluator), which is why our build tool uses Haskell source files instead of yaml as the build description language.

[snoyberg]

In the world of stack, explicitly listing packages isn't a hard requirement actually. We could do something intelligent like:

• Try to compile
• If it fails, look at the module that's missing
• Go to the build plan and determine which package it comes from, and install it

Not saying this is a good idea, there are downsides, but pointing out that there are entire regions of the design space that we have open to us.

[NathanHowell]

I will always want to explicitly list what to build. Often times we'll have portions of the tree that don't build (old utilities usually) while libraries are being upgraded, or to decrease incremental build times in feature branches (e.g. branch-x-wip only needs to build feature y, but master builds everything).

Some other options:

• Support (explicitly) inheriting configuration from the parent directory
  • this is how razzle worked for MS Windows builds
• Use a single giant file to describe all packages in tree (not as friendly for large projects and teams)

[ndmitchell]

@gregwebs Agreed on adding the executable arguments to the existing Cabal stuff. Unfortunately mixing new cabal.exe and old Cabal library is a disaster, so it takes quite a while to get rolled out, while Stack has the advantage of being very hackable and more aggressively developed.

For flags, I really want to say: if not windows, include the unix package. In my ideal world, that would test whether I was on Windows first, then look at the condition. In Cabal, it can sometimes infer the Windows flag based on whether the unix package exists, sometimes (for certain situations) inferring Windows on Linux platforms. Basically, I want the flags to be fixed, not inferred.

[acowley]

I totally support this direction, but it's probably a touch too early in the Stack era for it to happen. Things like build-tool dependencies are basically a disaster at the moment, with tooling either just punting on them (like cabal-install), or having to special case them. With package descriptions finally moving to separate markdown files, the information content of package specifications is naturally winding down to a skeletal structure currently captured in an unwieldy format.

It will also help to have a usable specification format if and when we start considering hashes of packages for Nix-like building. So this very issue is something I was going to raise next week, and it's very helpful to start sorting out what we want in terms of flags and imports. We're dealing with a much, much narrower use-case than the Nix language, and we're developing the tooling that is to be driven by these specifications, so it's quite possible all we'd need could comfortably fit in YAML.

[gregwebs]

Michael managed to make stack not need to reconfigure in order to build your test suites.
He still had a motivation of faster compilation techniques that are limited by Cabal the library.

However, the rest of the goals seem to be accomplishable by using Cabal the library but having a new format that stack can output as a .cabal file. Neil mentioned generating a .cabal file before upload, but I think you would want to always be generating the cabal file for use with the Cabal library and compatibility with existing tools.

[radix]

Is there a way we can obsolete the need for the purely boilerplate Setup.hs files with Stack? Would that fit into this ticket, or a different one?

I've never needed to write anything other than the default code in my Setup.hs files, so maybe someone with that use case would be able to offer more input there.

[snoyberg]

Custom Setup.hs files allow you to do things like have pre-build and post-build hooks. It's uncommon to need them, but they do have a purpose. In any event, stack has never required that a Setup.hs file be present (it will write one for you as necessary).

[bitemyapp]

@snoyberg I agree, but I would submit that writing Setup.hs files that work for doctests is extremely hairy. I had a lot of trouble getting the one in Bloodhound and very few people knew what to do except for people that worked on lens or another non-trivial project with doctests.

I know this has been brought up, but - If anything can be done to have nicer/more automatic integration with doctests that could save a lot of programmer-hours.

[gregwebs]

Setup.hs should probably be discussed on the stack new issue

[bitemyapp]

@gregwebs my concern is even apart from the question of Setup.hs (I do think they're useful/necessary in some cases), so I've opened an issue specific to doctests.

[sol]

I've been pointed here by @soenkehahn.

@ndmitchell we have been hacking on something that is very close to what you suggest https://github.com/sol/hpack

(No documentation yet, but there are examples and links to slides from my talk at the SG Haskell user group in the README).

It

• allows you to specify many things (e.g. dependencies top-level and on a per-section basis, removing redundancy
• infers other-modules
• supports file globs for extra-source-files

It's experimental, I'm very open to feedback and at the current state of the project not afraid to make breaking changes.

Currently it maps to Cabal, but I like the idea of having a custom builder.

[mgsloan]

I've taken a look at hpack, and I think it looks like a great alternative package markup, it addresses most things that I can think of improving over the current format.

I've opened this issue on hpack discussing integration with stack, and opened this issue on the stack tracker, for discovereability.