Plan to be distribution friendly

[bastien-roucaries]

Hi,

Two years ago you planned to be distribution friendly (https://docs.google.com/document/d/1VuYv41U_Esjrl0umu-uO3hUK4WNnCiG0pXZgjliHfTw/)

On the debian side we are dropping IA package (tensorflow) because we could not get bazel on debian.

Did you have some plan to be distrib friendly ?

Thanks

[email protected]

[bastien-roucaries]

Related to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=782654

[bastien-roucaries]

Note that we have outdated patch here https://github.com/kmoffett/bazel/tree/debian/sid/debian/patches

[bastien-roucaries]

Any plan and delay for this ?

[dslomov]

We are not actively pursuing getting into Debian atm. However, we would welcome community efforts.

[bastien-roucaries]

Le lun. 21 oct. 2019 à 10:58, Dmitry Lomov <[email protected]> a écrit :

We are not actively pursuing getting into Debian atm. However, we would welcome community efforts. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9408?email_source=notifications&email_token=ABE5UHCYOSFSBMQVXP5EULTQPVVMFA5CNFSM4IYP6V32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBZSWII#issuecomment-544418593>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABE5UHHLN3I5HOY7NOLYPQDQPVVMFANCNFSM4IYP6V3Q> .

Coule you at least write a plan for getting distri friendly with incremental step ?

…

[bastien-roucaries]

@dslomov Could you please give us some guidance to be step by step distrib friendly and upstream includable ?

I can code myself but I need plan

[philwo]

Hi @bastien-roucaries,

thank you for offering to help. I'll follow up in the next days with action items that we should address. I think we have some notes about what we have to do here, I'll try to find them and will post everything here. If you don't hear from me soon, please ping me here on the issue.

Some things out of my head:

• The Bazel binary should not be self-extracting, instead we should package the install_base so that it's present in /usr/libexec/bazel (?) and Bazel just uses it.
• Bazel should not use an embedded JDK, but the one provided by the distro by default.
• We have to figure out how to deal with dependencies - we probably have to build Bazel against third-party libraries provided by the distro, instead of the versions in our third_party folder.

[olekw]

Hi @philwo,

Ping. :) I'm working with the Debian Med team right now to get some much-needed software packaged and available for users in the medical community to help with the COVID-19 pandemic. At least one of the packages we desperately need requires Bazel to build.

Clearly this is an unusual and very critical situation. I don't think it's an exaggeration to say that lives may literally depend on us getting better tools to the medical professionals out there, and quickly.

I appreciate that you likely have your own struggles right now but there are a number of us who are willing to assist. The entire international community would be extraordinarily grateful if @google and the @bazelbuild team could prioritize helping with this!

Thanks in advance and I'm waiting and eager to help however I can!

-Olek

[olekw]

Also, it probably goes without saying but we'd love to be able to highlight @google support when we summarize the impacts of the international effort towards the COVID-19 Biohackathon! All of us doing a little can make a huge difference for people out there!

[meisterT]

Hi @olekw,

while I understand the desire to move this issue forward, please understand that it is major work which will take a lot of time to complete. And even after making Bazel distribution friendly, it will take some time to get it into the stable debian distribution.

If you need something in the short term, consider using the debian packages that we already provide (https://docs.bazel.build/versions/master/install-ubuntu.html) or use bazelisk (https://github.com/bazelbuild/bazelisk). Let us know if any of those help you to satisfy your current needs.

Tobi

[olekw]

@meisterT,

Thank you for your reply. However, I'm afraid neither of the options you presented can be packaged in Debian and therefore the medical packages we need will also not be able to be included in Debian. It's not that an individual user can't install Bazel itself, it's that we're trying to build critical medical packages in Debian using Bazel. And we are targeting Debian Unstable at the moment, not Stable. I hope that clarifies what we're trying to do.

To be clear, we're moving forward on making @bazelbuild work in Debian. This is too important to just wait. But we would really love to have support from the upstream team and from @google in this effort!

[meisterT]

Thanks for this clarification. May I ask which package(s) you want to build with Bazel?

Probably the first thing one should do to move this issue forward is to go over our (vendored) third party dependencies (see https://github.com/bazelbuild/bazel/tree/master/third_party) and for each

• check whether we can drop it,
• and if dropping is infeasible check if there's already a debian package that provides the dependency (in ~the right version).

[olekw]

Hi @meisterT,

The highest-priority package we're trying to add to Debian is TensorFlow (https://salsa.debian.org/science-team/tensorflow).

Thanks! Yes, I've been working on that. My personal perspective (I'm a DD but not on the FTP Master team) is that we could leave certain third-party dependencies bundled. Here's how I see that:

1. Dependency is in Debian and usable: Drop and use Debian version
2. Dependency is in Debian and not usable: Update Debian version so it works with Bazel
3. Dependency is not in Debian and licensing allows it to be included in the Bazel source package: leave it in (if other programs need to use that dependency in the future we could always spin it off into its own package)
4. Dependency is not in Debian and licensing does not allow it to be included in the Bazel source package: work on separately packaging (although I'm not sure if any dependencies fall into this category)

I would love your inputs as to which dependencies could theoretically be dropped (and the impact of dropping them) so that we don't spend tons of time on something that doesn't appreciably impact Bazel functionality. I've looked through the old Dependencies Google Sheet, and the Debianization Google Doc although those seem old and I'm not sure how updated they are. I might recreate those on salsa.debian.org so that we can have everything together and update as needed.

[hicksjoseph]

Hi Olek, I am the Product Manager for Bazel. I have sent you an email with my contact information so that we can have a quick call so that I can fully understand your needs and the urgency of your request. Please feel free to contact me any time via the contact methods in my email to discuss this directly. Thanks, Joe

[Ericson2314]

With supreme difficulty the NixOS distro has managed to package software built with Bazel, including tensorflow right https://github.com/NixOS/nixpkgs/blob/29eddfc36d720dcc4822581175217543b387b1e8/pkgs/top-level/python-packages.nix#L6561-L6623 . If our code, or our explaining of code could be of some assistance, do let us know.

Bazel really insists on wanting to control the world like only a distro should, so there is still rather more vendoring of random things here than we would like, but it does build and run.

[Ericson2314]

For the long term beyondn Covid-19, I would recommend following the lead of Meson with their "wrap dependencies" https://mesonbuild.com/Wrap-dependency-system-manual.html . Basically, the purpose of unblocking devs on all platforms with vendoring is laudible, but it is also synonymous with being a package manger. So own it and be a de jure not de facto package manager, and share the vendoring code between projects while one is at it. But make all that vendoring optional so there is a vendor-nothing version of the build distros can use.

[Full disclosure, I have contributed a bunch to Meson, so perhaps I am biased. But I began contributing only long after I already was packaging things for Nixpkgs/NixOS, and was inspired to do so because it was one of the only new build systems which did play nice with distros.]

[hicksjoseph]

The Bazel team spoke to Olek and others and are working together on an assessment of work and plan per the request above. Ping this thread if you have questions / concerns.

[olekw]

Hi @Ericson2314 and thanks for the extra info! I'm not very familiar with NixOS therefore I'm not sure how much of your work we'd be able to reuse but I'll definitely take a look. I appreciate the pointer!

[Ericson2314]

@olekw Nixos/Nixpkgs is quite different from Debian, but one ignore all that and just look at the bash templating, which should be useful prior art for any disto cherry-pick. I am happy to link the relevant bits so you don't have to search for them if you like. [I linked roughly a list of the tensorflow-related packages we had, separate from the details of any of the,.]

[davido]

So, Error Prone upgrade was submitted, that depends on Caffeine Cache.
However, it's outdated on Debian. I figured, to contact Java Maintainers
the email address is: pkg-java-maintainers @ lists.alioth.debian.org.
But would it be enough to kindly ask to upgrade Caffeine Cache version?

https://packages.debian.org/source/sid/caffeine-cache

[olekw]

Hi @davido. The best way of requesting a package upgrade in Debian is by submitting a bug report. The reportbug program is the easiest way of doing so if you're on a Debian-based OS.

However, can you give me some more info since I happen to the packager/maintainer for Error Prone in Debian? I currently have Error Prone 2.4.0 in Debian unstable. That version depends on Caffeine 2.8.0 which is the same requirement as HEAD. So I don't see an issue there. Or do you mean that you would like the Error Prone package to include additional modules? I currently only build annotation, annotations, and type_annotations. The others have dependency issues and, as far as I'm aware, are not required by Bazel.

[davido]

@olekw See: #11426 (comment).

third_party/caffeine/caffeine-2.8.0.jar: OK
third_party/checker_framework_annotations/checker-qual-3.2.0-sources.jar: OK
third_party/checker_framework_annotations/checker-qual-3.2.0.jar: OK
third_party/checker_framework_dataflow/dataflow-shaded-3.1.2-sources.jar: OK
third_party/checker_framework_dataflow/dataflow-shaded-3.1.2.jar: OK
third_party/checker_framework_javacutil/javacutil-3.2.0-sources.jar: OK
third_party/checker_framework_javacutil/javacutil-3.2.0.jar: OK
third_party/error_prone/error_prone_annotation-2.4.0.jar: OK
third_party/error_prone/error_prone_annotations-2.4.0.jar: OK
third_party/error_prone/error_prone_check_api-2.4.0.jar: OK
third_party/error_prone/error_prone_core-2.4.0.jar: OK
third_party/error_prone/error_prone_type_annotations-2.4.0.jar: OK
third_party/error_prone/threeten-extra-1.5.0.jar: OK

Particularly, these were new dependencies in Bazel:

third_party/caffeine/caffeine-2.8.0.jar
third_party/checker_framework_dataflow/dataflow-shaded-3.1.2.jar
third_party/error_prone/threeten-extra-1.5.0.jar

Are these libraries available in Debian unstable?

[davido]

OK, answering my own question:

You have uploaded Error Prone upstream code base 2 weeks ago to Debian Salsa, and the new EP transitive dependencies, I mentioned in my previous comment seems to be there:

https://salsa.debian.org/java-team/error-prone-java/-/blob/master/check_api/pom.xml#L65
https://salsa.debian.org/java-team/error-prone-java/-/blob/master/core/pom.xml#L356

And I also found:

https://tracker.debian.org/pkg/threeten-extra
https://salsa.debian.org/java-team/checker-framework-java/-/tree/master/dataflow

[olekw]

Hi @davido. So those artifacts are not currently built in the Debian package since I was trying to keep the new package as simple as possible and only include the minimum needed for Bazel. As I mentioned earlier, Error Prone only builds three artifacts in Debian and Checker Framework (once it clears the NEW queue) will only build checker-qual and checker-qual-android.

Please let me know right away if additional artifacts are mandatory for Bazel to build! Last I checked with @meteorcloudy, the currently-packaged artifacts were the only ones we needed. For all of the Bazel dependencies that I packaged in Debian I typically added additional (not required by Bazel) artifacts if there were no dependency problems with including them. Building additional artifacts will likely require software not currently packaged in Debian and that will likely add significant time and effort to this project.

[sgowroji]

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.

[olekw]

@sgowroji, no, since this has not yet been completed I do not think it should be closed. This is still an ongoing effort and therefore this bug should remain open to track that effort.

[meteorcloudy]

Bazel's third_party directory is much smaller now since we have migrated all checked-in jar dependencies to rules_jvm_external, which should make it easier to package Bazel for Linux distributions in future (for example, we only need to override the @maven repository to make use of existing jar files under /usr/share/maven-repo on Debian)

[meisterT]

@olekw What are the main blockers from your POV to make Bazel distribution friendly?