-
Notifications
You must be signed in to change notification settings - Fork 843
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backpack for Stack #2540
Comments
This would be awesome. |
@ezyang First let me thank you for all your work on this. TL;DR. Does stack need this shipped tomorrow, in a month or in a year? Before I can look into details and ask more substantive questions: is there a timeline for Backpack being merged/released/used, and an estimate how much work was required on cabal-install?
I agree this is probably essential. Having feature parity also matters—having to choose between stack and Backpack, or stack and sane strings, would be pretty bad. |
@mgsloan, I think you know the most about the parts of Stack that @ezyang is talking about. Once you're back from BM, can you weigh in? @ezyang: how is cabal-install going to handle Backpack? Both cabal-install and Stack use the Cabal library to actually build packages, so there may be opportunity to share much of the code. |
Definitely not tomorrow or a month. If all goes well Backpack will get merged to GHC and Cabal, and people can start experimenting with it, but there were certainly be bugs and I won't claim that anyone should start using it for real immediately. Backpack'ing base is at least blocked on teaching GHC's build system how to deal, which will take some time and I am not planning for 8.2. On the other hand, base is probably the most plausible, widely used package that will get Backpack'ed, because (1) the potential upside is so great, and (2) it closely tracks GHC versions, so we don't have to worry about supporting old GHCs.
There are some key library functions for configuring and instantiating packages, and cabal-install (new-build only; the legacy codepath doesn't support it) just calls out to them. At present, there are some things that are duplicated because I couldn't figure out an API that could be reused in both cases, but this is definitely a case where I can try harder. The two key functions have these signatures:
and a |
Stack does this. We use |
Great. Another question I have (since I am looking into this) is how difficult it would be to get Stack to support a per-component build model. I'm hoping that I can take anywhere we talk about 'Package' and actually it's talking about a 'Component', but I am not familiar enough with the code to say if this is the case or not. |
@ezyang Could be tricky! The current code is more package centric than component centric. This is for a variety of reasons. Off the top of my head:
And no doubt, more. So, no, when the stack code is talking about a 'Package' it's definitely not talking about a 'Component'. |
OK. So perhaps a reasonable approach to start with is to assume that any package which exposes a Backpack library, ONLY contains that library, so that if I need to instantiate a package multiple times, this would only involve creating duplicate copies of the package (another thing where I'm not sure how well Stack will deal with this change.) |
I've been fishing around the codebase, and here are some initial architectural thoughts on what Backpack needs.
If Stack was modified in this way, would you take the patch? How would you like the design to be different? What is unclear? |
I'm just coming up to speed on this now Edward. It seems relevant to work I'm already doing upgrading to Cabal 2.0. I'm also in favor of pushing Stack towards a component base build system. My only concern with this is making sure we retain support for building packages with older versions of Cabal, as all existing snapshots include such an older library version. I realize this issue is a bit dated now, and perhaps you have different thoughts. Maybe we could jump on a text chat and synch up? |
@snoyberg Sure thing; ping me on IRC some time? As far as I am concerned, the thinking on this ticket hasn't changed much (although maybe Stack's internals have!)
Yeah, so in cabal new-build, this is done by treating these packages as non-per-component, single packages in the plan. It's a little bit of complexity but not too bad, and keeps BC. |
Just chatted with @ezyang on IRC about this. Here's my summary:
I'll ping this issue with updates when Cabal 2.0 is merged in |
Pinging appropriately: Cabal 2.0 support has just been merged to master :) |
This should resolve both #3364 and #3361. There is a test case included that should address both of them as well. This is still relatively hacky. We will eventually be overhauling the component system more dramatically to support Backpack with #2540 (CC @ezyang). This may still have some problems due to the upstream issue haskell/cabal#4763, but at least in theory we're matching the behavior of upstream. We can consider workarounds after that issue comes to a conclusion.
What work needs to be done to finish adding Backpack support to stack? |
There's still quite a bit of work to be done, see my comment above #2540 (comment). All that we've done so far is get support for Cabal 2.0 into Stack. |
I'm working on per-component build for Stack in https://github.com/ezyang/stack/tree/per-component-build It doesn't build yet, but I've finally gotten it to typecheck to |
Great, thanks @ezyang ! I skimmed it, and it looks good so far. Feel free to open a PR before its ready, as soon as you are ready to start getting review feedback. |
How's this come/coming along? |
Very, very slowly XD |
any progress? 🙂 |
I noticed that this ticket has a call for help via "A year into Backpack." Since this is needed for the hasktorch project to get into the LTSs, I've posted a bounty for this issue. |
From @ezyang's blog.
Hey @ezyang, it was nice of you to volunteer, but when you realize you don't have time for a ticket it's good etiquette to say so in the ticket thread. That way people know it's unassigned and up for grabs. Don't lick the cookie—pass the plate! You've done great work in characterizing this issue so far and I don't want to diminish that. But as they say, the last responsibility of a maintainer is to find a replacement, and it's good to do that in a centralized way. Many developers are on tenterhooks by this feature, so it's important that its status—viz. is somebody working on it / does it need a maintainer—is clearly advertised. |
@matthew-piziak Cookie licking sounds pretty incongruous to this particular situation. @ezyang did a tremendous amount of work to actually get Backpack merged into GHC, and also on this particular ticket. This task isn't assigned to anyone and @ezyang makes it clear in his blog post that he'd be happy to see someone else pick this up. |
@mboes You're right, I realized now that I had a poor understanding of the ticket's history. I apologize for the misleading language in my comment. In any case I am happy that this issue's status has been clarified. |
I don't know if its related but I faced the same issue (Invalid package ID) on a backpacked project and fixed it by adding |
this seems like a pretty egregious bug. why isn't it fixed? |
@cartazio For what it's worth, For reference, please note the following tracking issues showing that support for |
while i can empathize, i dont sympathize. |
if they support ghc, this is a bug. If they just support haskell 2010, fine. but this means any post Haskell 2010 feature support is underspecified. Do these tools support extensions/changes relative to Haskell 2010? What reference implementation is guiding them? I think you're saying "not ghc/cabal, but something more nebulous" |
I suppose my point here is that we've gone several years with all of the alternative build tools to
My point was more that there doesn't seem to be much pressure on any build tools to support To be fair, this is one of those tricky things in the space of open source community management. If the compiler implements a feature, but not enough people care to use it (or push for its adoption), is it really a bug not to support that? EDIT: In the |
Most of the projects can move forward without backpack. Just that it offers really nice features which people like to use. "Not a pressing problem" doesn't justify the attention this issue got and the nearly $500 bounty on it |
I’m starting some work in progress evaluations on how / if folks such as
backpack can dramatically improve the experience of both maintaining
vector and making it easier for users to write their own factor families.
It’s still up In the air if I can get it to work the way I want.
But if it pays for itself in the ways I hope, the longest I’m going to sit
on such a release to allow other tools to catchup is 6 months.
I’m pretty confident that if I get all the pieces lined up that It can
essentially just be a minor version bump of vector. But if so, I’ll have to
give y’all a grace period to fix this bug. I guess this is your 6 month
notice. Since waiting to be certain it pays off will just drag this ticket
out further.
…On Mon, Feb 3, 2020 at 1:14 PM flip111 ***@***.***> wrote:
it's certainly not a pressing problem for consumers of these tools
Most of the projects can move forward without backpack. Just that it
offers really nice features which people like to use. "Not a pressing
problem" doesn't justify the attention this issue got and the nearly $500
bounty on it
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2540?email_source=notifications&email_token=AAABBQQ2GUDBLLN35GVIGNTRBBNJJA5CNFSM4CN45YHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKU22VA#issuecomment-581545300>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAABBQSBFUPIVGEOSBJA5ODRBBNJJANCNFSM4CN45YHA>
.
|
to be clear: (and a stack contributor nicely pm'd me on IRC to help me clarify what i mean)
many packages depend on vector directly or indirectly. If vector starts using backpack, this impacts users of tools that dont support it. If backpack is an effective tool to support evolving vector aggressively while maintaing super stable user interfaces and much more friendly "write your own new vector instances " workflow for users who want, idk, memmory mapped or gpu or tape-drive fusioned vector data types, i want to be able to do this. Or I want YOU to be able to this. Many people use stack as a personal choice, and I dont want stack users or vector users to suffer. Accordingly it seems pretty important for backpack to be supported should this design mix pan out. Otherwise any library using vector directly or indirectly will not be buildable with stack. |
Summarized and paraphrased information so far.
At the moment stack does not have a graph for components. "mix-in" relates to which types are used for a package from which other package. See: #2540 (comment)
Open question: Does stack know how to register multiple libraries produced by a setup script? --> If it can then mechanisms there could be re-used to do components (as one library would be one component)
An approach to start getting a proof of concept? Maybe better to have real components to get "architecturally sound implementation". Best plan for implementation:
Feel free to correct me :) @ezyang what did you mean with "register" in your opening post? |
@snoyberg can you explain how far this file has the type of component that is needed to support backpack ?? https://github.com/commercialhaskell/stack/commits/master/src/Stack/Types/NamedComponent.hs Is this in principle the same "component" but might need some enhancement to store backpack specific information ? EDIT: also is it ok for stack to use types like https://github.com/haskell/cabal/blob/1a31242dfe1f0376051057370e25cc846e319ea8/Cabal/Distribution/Types/Component.hs I'm not sure what parts of cabal can being pulled in or not |
Well, for what it's worth, I've been recently discussing with colleagues switching back from Of course, there is not much pressure at the moment, because there are few Backpacked packages, because package authors avoid Backpack, because it cannot be built by one of two main Haskell build tools. It is a pity that packages cannot explore features, merged into GHC four versions ago. Changing this status quo would be a massive shift IMO. On the other hand, this is open source: people are free to pop in, fix an issue and get away with praise/glory/bounty. It would be unfair to blame maintainers of |
I agree with @Bodigrim. Very few Haskell packages take advantage of Backpack because a major build tool ( |
Note that there are several GHC bugs related to backpack, although I don't see anything too bad. |
@ezyang are the
My guess would be that these "units" are not the same kind of units, because one includes PackageIdentifier and the other doesn't. Confirmation would be appreciated. |
Is it possible that such an instantiation requires a new package? Or is it fine to do these instantiations after package dependencies have been calculated ? |
On how to blend backpack code with stack code ... Searching in cabal's codebase for On the one hand for stack's purposes it seems useful to do the expansion of tasks to build that is needed for backpack as late as possible. So after tasks are made expand them in yet even more tasks. stack/src/Stack/Build/ConstructPlan.hs Line 206 in a7a6c35
On the other hand toConfiguredComponent and toLinkedComponent need information that will only be available earlier than First step seems to be to figure out where the arguments from After that see if the return value of |
As a very new Haskell user, I was exposed to backpack (while still getting my bearings straight with regard to Haskell build tools), only to realize it would be a no-go if I wanted to use it with Stack. I've since been tempted to use it again, but realized the issue was still open. Had I been a user of backpack all along (which has only been a little over a year), I'm sure I'd have thought of many more places I could have used it. Just my 2c - I understand all too well that development time is limited. |
Sorry about the delay, I saw this in my GH notifications but sat on it for a week.
referring to:
My recollection from the last time I looked at the Stack code, was that everything in Stack revolves around installing packages (there's a dep graph of packages, you install them one by one). The comment on private Backpack use is simply pointing out that sometimes installing one package results in multiple libraries getting installed, and as long as Stack can deal with this situation, all is good.
Yeah, they're not the same, precisely because the proposed type here refers to packages rather than components. Components are a pretty big deal for usability of Backpack, because a lot of things you want to do involve having to write many small libraries and then recombining them together. If you'd have to make (and distribute) a separate package for each of these libraries, it would be a serious source of friction.
It's fine to do it after depsolving; that's how Cabal-install does it. Depsolve the package structure, and then elaborate it into a more fine-grained component structure. As for the last few comments, the kit in the Cabal library is what stack can usefully use, the stuff in cabal-install is a useful reference for how to use the APIs but I expect none of it is salvageable. That's OK; I have a "mini" implementation of the Cabal-install logic for Setup.hs and it was mostly fine. To me, the problem was always how to drag Stack into the component world. That required more time investment learning about Stack that, sadly, I never found time to do in the end. |
@snoyberg is this on the todo queue of y’all ? |
Following:
I think for now, it's better to follow the progress here, for anyone interested: |
Hello all,
I want to open up the discussion about how Backpack can be integrated with Stack; in particular, I'd be willing to work on and submit a patchset adding support for it in Stack. But before I do that, I want to hash out some general architectural concerns. The specification for the GHC and Cabal support for Backpack is here: https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst ; please read it for more motivation on why you might like to support it.
The design process over two years lead us to a design where Backpack requires the package manager (Stack) to know how to instantiate packages that have required signatures in them (I spent the better part of six months trying to figure out how to do it all in GHC; if you are interested in the sordid details I can let you know). Things seem to work most smoothly if:
PackageDescription
); we can't do Backpack instantiation until this finalization has occurred. I don't know where in the Stack codebase one would have to engineer this.I'll note that private use of Backpack (i.e., the public library does not have any signatures) should be supported out-of-the-box, as there is no change to the outward-facing API of Setup scripts in this case (Stack will need to understand how to register multiple libraries produced by a Setup script, however). In the meantime, public libraries which have signatures would not be supported.
One of the major applications of Backpack I would like to see in the near future is the Backpack-ification of GHC's boot libraries to solve the string problem. Backpack would make it easy to write modules which are parametric over a particular desired string representation; the original package names would continue to export
String
-based interfaces, but versions usingByteString
etc could be made available, and the availability of a package abstracted over holes means that an end-user could specify their own. Absent full support for Backpack, Stack would at least have to be able to handle traditional-looking packages which internally depended on libraries that made use of Backpack. Maybe this would be easy.Please let me know if you have any questions or concerns; I'd be happy to answer them.
CC @mgsloan, @snoyberg, @Blaisorblade , @borsboom , @harendra-kumar
The text was updated successfully, but these errors were encountered: