Packages with !include in them don't work with Pantry.

[myShoggoth]

With the new Pantry system taking package dependencies that don't have a top level .cabal or package.yaml file fail.

Example:

- git: [email protected]:kim/opentracing.git
  commit: b6e12de671f51f39ea2a9230e43cef2badd993f4
  subdirs:
    - opentracing
    - opentracing-zipkin-common
    - opentracing-zipkin-v2

This project has a stack.yaml that references packages, but doesn't have a .cabal or package.yaml file for the top directory.

The error looks like:

/private/var/folders/6y/_mvyr6tj4px_52pkvychlw1r0000gp/T/hpack-pkg-dir23692/package.yaml: Yaml file not found: /private/var/folders/6y/_mvyr6tj4px_52pkvychlw1r0000gp/T/share/package.yaml

[qrilka]

@myShoggoth the problem is not about top-level directory missing package.yaml it's about shared yaml file, see the following line in opentracin/package.yaml:

_include: !include '../share/package.yaml'

@snoyberg it looks like Pantry broke this feature of yaml sharing, does it make sense to bring it back or maybe it contradicts Pantry logic in some way?

[snoyberg]

I'd shy away from such a change to how Pantry works. In order to make this use case work, we would need to store every file in an archive or Git repo on the off chance that something is using an !include or similar feature.

[myShoggoth]

Is there a path forward for this? A pre-processor step on the yaml, as gross as that is?

[snoyberg]

I don't see a path forward like that. The closest I can imagine is a setting to Pantry which says "cache the entire repository, not just the subdirectory in question." I don't intend to work on that, but if someone is interested in approaching it I can provide guidance. For now, I'd say generating a tarball with a cabal file instead would be a decent workaround.

[asivitz]

I have no real knowledge of Pantry's implementation, so I apologize if this is completely off base, but would it be reasonable to generate the cabal files after the clone but before it is cached? Unless there is value in caching the subdirectory exactly as it is cloned. I guess I'm suggesting something similar to @myShoggoth but it might not be gross because it's just moving the hpack generation step earlier. Maybe there's even a performance benefit to be gained here. 🤷‍♂

I'm assuming of course that the full repo is cloned, and not just the specified subdirs.

[maksbotan]

Hello, @snoyberg.
To clarify -- would you accept a patch to Pantry that would cache the generated *.cabal files instead of source package.yamls?

[snoyberg]

I'm not really in favor of that, it will still break the reproducibility guarantees of pantry. Overall, I think the problem is that we shouldn't have allowed package.yaml files in Git repos to be used as they're being used. The original design of pantry required a .cabal file to be present in the repo, and that overall is the correct solution. My advice remains to refer to tarballs or Git repos that include the cabal file.

I'm strongly considering deprecating support in pantry in the future for using package.yaml files as they are supported today.

[snoyberg]

Since we'll be deprecating this workflow (see #5210), I'm closing this issue as wontfix.