Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple Regolith from Debian package management #540

Open
2 of 3 tasks
marc0der opened this issue Jan 21, 2021 · 48 comments
Open
2 of 3 tasks

Decouple Regolith from Debian package management #540

marc0der opened this issue Jan 21, 2021 · 48 comments
Labels
enhancement New feature or request

Comments

@marc0der
Copy link

marc0der commented Jan 21, 2021

Is your feature request related to a problem? Please describe.
I love using Regolith, but like many other users, do not feel the same way about Ubuntu. At the moment, Regolith is very tightly coupled to the Debian package management, which makes porting Regolith to other distributions very difficult and cumbersome.

Describe the solution you'd like
It would be fantastic if all Regolith configuration lived in a separate repo, decoupled from all the Debian packaging config. This would allow other spins (such as an Arch or Fedora spin) to be created from the centralised configuration. The current Arch (and Manjaro) packages rely on transforming the Debian artefacts, which works but is an error-prone and labour intensive process.

Describe alternatives you've considered
I've been using the regolith-de package provided by @gardotd426 (which is a great first attempt to get this working, thank you!), but this package tends to fall behind and not on the Arch AUR. Even though this works in the short term, it is not an optimal solution and will require lots of effort to be maintained in the future.

Proposed solution
Having a separate repo for the Regolith configuration could allow this project to have multiple automated builds that produce releases for all popular Linux distros. This would definitely further the reach of this fantastic project and help in adoption by many tiling window manager fans like myself who do not want to be tied to Ubuntu.

I am willing to get involved in such a project and would love to know your thoughts around such a proposal. It certainly means quite a bit of work, but could potentially open up many new avenues and bring lots of growth for this project in the future.

@marc0der marc0der added enhancement New feature or request needs-triage Any bug that requires classification by a contributor labels Jan 21, 2021
@kgilmer kgilmer removed the needs-triage Any bug that requires classification by a contributor label Jan 22, 2021
@kgilmer
Copy link
Member

kgilmer commented Jan 22, 2021

Hi @marc0der , I completely agree with your points. We look to address the existing coupling to Debian packaging in Regolith 2.0. We don't have much in the way of details yet on this particular item but I believe this will be in scope for the next major release. There are two issues ~ reducing the coupling between packages such that a smaller subset can be more easily ported to another Linux distro without having to "boil the ocean" and port all packages to have a working system. Second, (and more minor I believe but things always start out seeming to be minor) is to remove any existing logic that refers to dpkg, apt, or anything else specific to Debian (or Ubunutu).

I would greatly appreciate your help! What package manager are you familiar with? The question on the top of my mind is, do all popular package management systems support the following features:

  1. "virtual" packages...multiple packages can satisfy a given dependency
  2. "one and only one" of a given virtual package
  3. specifying that a virtual package must be installed and if one currently isn't, specifying a default
  4. soft dependency declarations...a transitive package is installed along with a specified package, but the transitive package can be cleanly removed by the user without breaking the dependency relationships in the package manager.

Regolith relies on all of these features from apt/dpkg. In fact, a primary configuration mechanism that Regolith utilizes is the package manager itself. For example, we have compositor packages and they contain package metadata such that one compositor must be installed, and all of the status indicators exist as packages that inject conf.d style configuration snippets at install time.

Much of the coupling I mention above is due to the monolithic nature of the i3 config file. As-is, any program referenced in that file becomes a hard dependency of Regolith. Another theme in 2.0 is adding a simple pre-processor that will generate the i3 config file from conf.d style partials. This allows us to break up these dependency relationships such that porting can start small, be functional, and be incrementally augmented over time.

@kgilmer
Copy link
Member

kgilmer commented Jan 22, 2021

Oh and I happen to be an sdkman user, thanks for your efforts @marc0der !

@marc0der
Copy link
Author

marc0der commented Jan 24, 2021

Thanks for the reply @kgilmer! Sounds great as a target for Regolith 2.0, and I totally agree with your suggested approach.

I've worked with several package managers in the past. I did some packaging work for some Ubuntu/Debian projects but have now been a longtime Arch (and Manjaro) user. I've also done some work with rpm, and I'm sure I'd be able to pick that up again too.

I hear what you are saying about all the virtual packaging considerations, but this does lead to a very Debian-centric way of thinking about packaging. Have you ever considered a radically different approach? I'm a guy who loves automating things, so have you ever considered using a build tool/system to do the packaging for you?

Consider if you had all the configuration for Regolith in a single repo (neatly modularised), and had some build tool that sat on top of this. You would be able to run different targets to produce whatever packaging you liked. You could produce a set of Debian/Ubuntu .deb packages, some .rpms (that might even have a completely different granularity or dependency graph to your .debs), or produce a PKGBUILD that produces a single package to be pushed to the AUR. Delegating this to a build tool could even eventually lead to the building of custom ISOs that represent different spins of Regolith. I know it's radical, but stay with me 😄

Along these lines, have you heard of jdeb? In case you haven't, it's a Java library that provides the ability to create Debian packages through an Ant task (that can easily be used from within Gradle or Maven through plugins). This is a way to get around having to use the entire Debian toolchain and be able to build packages on a CI environment. Similarly, there is an RPM builder that can be used from Maven to build .rpms. Of course, these don't give you all the bells and whistles that the native tooling provides, but you would still be able to build a fairly complex DAG of packages.

Another benefit of using a build tool is the power of templates. For instance, it could be possible to use Gradle to prepare a template of the i3 config custom for a given distro. This ties in nicely with what you mentioned about the generation of the i3 config file from partials.

These are all just thoughts and ideas, so take them with a pinch of salt. I'm sure you've already pondered many of these ideas too. But maybe some of these could lead to something useful that will help bring Regolith to the masses through many different Linux distros.

@kgilmer
Copy link
Member

kgilmer commented Jan 25, 2021

Are you saying that the notion of a Virtual Package is a Debian-only concept @marc0der ?

Regarding pivoting to a development model in which the Regolith DE is expressed in package-independent way and that a build system takes that as input and produces sets of packages for target Package Managers and Distributions is something I've briefly thought about. I find myself sticking to well-beaten paths simply as a matter of coping with a project of relatively large scope yet little time to devote to (being a side project). As I only know (and honestly not very well) how Debian packaging works I personally don't feel the risk is acceptably low that I would be able to produce a package abstraction and build system out of the gate that would fit well with other package systems, simply because I do not know how they differ from Debian.

I have briefly looked around to see how other DEs may handle this (package/distro agnostic) but haven't found anything that is general purpose in the sense that I'd be able to use use something out of the box. I've played around with the Opensuse build system which claims to offer this but didn't find documentation that I could get to work.

One approach I plan to look into once 2.0 is stable is fpm which is able to use the Debian packaging format as a source model and generate packages in other target forms. I've done a bit of testing with this and it seems to work.

Another concern w/ a custom build solution is that it would require would-be contributors to learn whatever abstract packaging format that was used, vs Debian which is at least popular and well documented (assuming you're reading the right docs).

And lastly, there is just a lot of long tail stuff that ends up being hard to do. By using Debian packaging I can usually find someone else having a similar problem and am able to adapt something like their strategy.

But, yes, the general idea makes sense. I just don't have the time to build it myself and haven't found anything that would be suitable for Regolith that doesn't have a steep learning curve and/or large risk that it ends up just not working. Last year I devoted a lot of time to unraveling the mystery of how Ubuntu LiveCD installers are produced, and ultimately threw in the towel with nothing to show for it. I'm still somewhat recovering from that and focusing more on goals with higher impact w/ less risk.

@marc0der
Copy link
Author

marc0der commented Jan 25, 2021

Are you saying that the notion of a Virtual Package is a Debian-only concept

By no means, but it is handled a bit differently by other distributions. For instance, an Arch package in the AUR can't depend on other AUR packages (you can only specify dependencies on packages in the main repos). However, on Arch, we can have a PKGBUILD that builds multiple packages (like regolith-de does), so closer to how a build script would build artefacts. In fact, the command to build the PKGBUILD is called makepkg. You then need to install these packages individually. You will notice that @gardotd426 has far fewer packages than your debs because of this flat dependency graph.

As you can see, this is a departure from Ubuntu/Debian where a PPA's packages become part of the greater pool and allows for far more complex DAGs. The package management on Arch is purposefully simpler in keeping with their philosophy of KISS.

Thus, making Ubuntu the base packaging enforces the Debian packaging onto other package management systems that aren't really compatible, or have very different ways of achieving their purposes.

So my personal gut feel is that:

source -> deb -> rpm
              -> pacman
              -> xxx

feels more error-prone, and prescriptive than:

source -> deb
       -> rpm
       -> pacman
       -> xxx

Then again, I'm known to be a purist 😁

I also hear your concerns about this stuff being very time-consuming, but here I'd be happy to help by getting involved (and I'm sure others are also keen to get in). I don't think anyone expects you to do all this work by yourself or get on top of all the types of packaging systems out there. I'm also not saying that what you've done so far isn't great work, because it's awesome! I'm merely saying that the current setup does limit the scope of who the project could reach going forward.

My idea of using a build tool for the packaging is exactly for the reason that you mentioned before. I don't want to know the intricacies of multiple platform packages. Others have done the work already. I would rather work on top of what they have already achieved. Gradle with the Netflix nebula ospackage plugin could save lots of time. I'm sure there are many other alternatives too.

Here's my suggestion: why don't we do a small spike to test viability?
I would choose some fairly isolated packages that can be drawn together via a virtual package (maybe themes are a good candidate, but whatever you would suggest).

  • import the repo(s) into a new repo via git submodules.
  • use a build tool in this new repo to try to build .deb packages.
  • new packages should be mostly equivalent to the ones that you are currently producing.
  • add the ability to produce the rpm analogues of the debs

If we could initially succeed in doing a simple like-for-like, it would prove the point. This would also give us a starting point to begin iterating on. I have a deep dislike for big bang changes, so I'm not proposing anything radical here. I would do something like this in a very sensible and gradual approach.

I hope you now have a clearer understanding of where I'm coming from and that I'm not thinking of any gun-slinging cowboy stuff 😄

@kgilmer
Copy link
Member

kgilmer commented Jan 27, 2021

Thanks for the kind words and well reasoned argument @marc0der 😄 . I think I have a clear understanding of where you're coming from and will take you up on your offer to collaborate! However I think the current work I'm doing in greatly reducing coupling across all packages needs to be stable before getting too far ahead, as the package layout change is drastic. Would you be able to do testing for AUR? Maybe some one else can cover RPM?

Let's move this over to the wiki: https://github.com/regolith-linux/regolith-system/wiki/Package-Manager-Agnostic-Build-System

@gardotd426 are you interested in getting involved?

@marc0der
Copy link
Author

No probs and glad that you understand my intentions here.

However I think the current work I'm doing in greatly reducing coupling across all packages needs to be stable before getting too far ahead

You are so right. Decoupling package management will give us a far better foundation to work from.

Would you be able to do testing for AUR?

Absolutely, yes. I would love to help in any way possible.

@marc0der
Copy link
Author

marc0der commented Feb 19, 2021

Just dropping a note here to say that regolith-de has not seen any subsequent releases since 25 December (1.5 was the last) and has still not been released to the AUR. Also, no further updates from @gardotd426 on this. I would love to see this move forward while we don't have the newly proposed solution in place yet.

What can I do to make this happen? Should we fork the regolith-de repo as an official part of regolith and start making it part of the regular releases? That would also involve publishing it to the AUR when it lands. As always, I'd be happy to get my hands dirty on this.

@gardotd426
Copy link
Member

gardotd426 commented Feb 19, 2021

Hey guys, sorry I had some equipment issues that kept me out of it for a while, but everything is now sorted and yeah Ken, I'm more than happy to get involved, this sounds like a good idea if you're able to do it. 

Also @marc0der, maybe contacting me on the regolith-de repo would have answered your questions, and the repository absolutely has had updates in since 12/25, actually 13 days ago was the last update, look for yourself. I don't see a reason to do GitHub "Releases" for an Arch Linux PKGBUILD, that seems nonsensical. But the repo has been getting updated as needed. And at this stage, everything is to the point where the only things that will be changing are the PKGBUILD and then maybe little patches here and there to fix Arch/Ubuntu differences, but they're rare and I've been able to remove most of them by now. So yeah, weekly/bi-weekly updates to package version numbers will be the only thing changing from here on out.

A user kindly put together a python script to help me automatically update the PKGBUILD with new .deb package versions, so I don't have to manually update it, because trying to track 40-something packages for version changes and then individually change them is insanely tedious. I'm looking to set up something that can automatically run the script at a set time (probably weekly) and notify me when there's updates so I can merge the PKGBUILD files and push the update to the repo. So I guess I anticipate a move to the AUR in the next week, as if it's this stable and only needs to be changed to track package versions (for the most part, unless something breaks where I fix it before pushing the update), then why not have it in the AUR? The main reason I've waited is for more testing, as 3-4 people isn't enough for a desktop environment for me to feel comfortable with stability, but the past few weeks I've had more and more testers, to where I'm around 20 at this point, and I feel comfortable pushing to the AUR.

But yes, a distro-agnostic build system would be a dream come true for me at this point, and am happy to help however I can.

@marc0der
Copy link
Author

@gardotd426 Well, this is fantastic news, and I'm thrilled to hear that regolith-de is still alive and kicking! I'm also happy to join your testers and will certainly engage with your regolith-de repo going forward.

It would be super nice to have some tags corresponding to the official regolith releases though, no need for full-blown Github releases. I think we are currently standing at 1.5.3, and it would be good to track this in your repo too. Also, great to hear that you are about to go live to the AUR! Have you considered using simple Github actions to automate this process? I'm happy to help you get this up and running.

But let's move over to the regolith-de repo for further discussion.

@gardotd426
Copy link
Member

Release is on 1.5, I see nothing on Regolith's website about 1.5.3, and I've had my PKGBUILD versioned 1.5 for weeks (since Regolith moved to 1.5 from 1.4.3 or 1.4.1 or whatever).

I see no point in tagging the github repo according to Regolith release version numbers, the only real important thing is to have the Arch PKGBUILD match up with the Regolith version its taken from, and it already does that. This is intended to be an AUR package.

Furthermore there's no real way to even automate versioning and I don't plan on constantly checking the Regolith home page to find out what the latest release one the "Release" PPA is. Do you? I was hoping there was a package included such as regolith-desktop-complete that came with a version in-line with Regolith's overall version (btw @kgilmer is there some reason why that's not a thing? Normally all DE packages share a version with the DE itself, you can look at Plasma as an example), but that's not the case, there are no 1.5 (or 1.5.3) versioned packages on regolith's release PPA. Well, regolith-system is on 1.5.0-1 for 18.04, but not for any other version, so idk if that's relevant or not.

But yeah, I'd written a script to handle the automatic updating of the PKGBUILD and include an updating of the pkgver, but there's no package version that corresponds so there's no easy way to do this, so for now I'm just tracking 1.5 as the documentation says.

But yeah feel free to open an issue on my repo if you want to discuss anything about regolith-de.

@kgilmer
Copy link
Member

kgilmer commented Feb 20, 2021

Hey guys 😄 . I try and keep the github release tags up to date here: https://github.com/regolith-linux/regolith-desktop/releases
Release notes are here: https://regolith-linux.org/docs/reference/releases/regolith-1.5.3-release-notes/

You can follow the @RegolithL twitter account or the regolith mailing list to get announcements for releases, if you'd like.

I was hoping there was a package included such as regolith-desktop-complete that came with a version in-line with Regolith's overall version (btw @kgilmer is there some reason why that's not a thing? Normally all DE packages share a version with the DE itself, you can look at Plasma as an example)

Well, the DE is a set of many packages. For the full ISO, a package regolith-system provides that version mapping from package to DE version. However in PPA form that package is not needed. The version of packages changes frequently and without cadence w/ the release version. The closest I can give you here is the /etc/regolith/version file which is shipped here.

Also, a bit of an update from my end, the 2.0 work is coming along well. My focus has been on reducing the package graph as much as possible to aid in porting as well as just enabling users to have light setups. Regolith 2.0 can now run on i3 or i3-gaps, and the package manager will handle all the compatibily details if a user switches between them. Also, it's running on Debian in 281Mb cold start. There is still some work remaining on the desktop metapackages but overall I think the scheme is in good shape as a PoC. I will add some documentation to the wiki and call for comments from users in the next week or so regarding the general design.

@kgilmer
Copy link
Member

kgilmer commented Feb 20, 2021

So, about the build system...is anyone of a general purpose package model in a modern format?

@marc0der
Copy link
Author

@kgilmer Awesome news about the 2.0 work, It sounds like you are making great progress. Also, thanks for clarifying about the versioning. This makes perfect sense. We can pick this discussion up again on the regolith-de repo.

So about the build system. I have been giving this some more thought and have some ideas about how we can do this in the simplest possible way. My initial thoughts about using Gradle and plugins have changed somewhat. Instead, I'm thinking of using something more native. I thought of using make in conjunction with Docker to build native targets per Linux distro. For instance, this would allow us to continue using the build-essential tooling provided by the Debian stack to build packages inside a mounted Docker container. Likewise, we could use makepkg for preparing Arch packages in an Arch container. We could repeat this for as many distros as we would like to support. What do you think of this approach?

Anyway, these are just thoughts for now, but I will report back when I have something more solid up and running over your 2.0 work.

@gardotd426
Copy link
Member

Hey @kgilmer, someone just made me aware that you aren't packaging rofication on launchpad the same way you have it on the regolith-roficiation repo. They simply pointed out that you aren't including the setup.py file, but you're also not including the i3xrocks module file, 80_rofication.

This actually means that I have to wait until it's fixed to put it live on the AUR, or just remove rofication and use dunst or some other notification manager. The problem is, AUR packages can't (at least are absolutely not supposed to) require user intervention, but the way things are right now, rofication won't work unless the user pulls the setup.py script from your repo and runs it. So we're at a bit of an impasse. I can still leave up my PKGBUILD repository, but I can't put it on the AUR until this is sorted.

@kgilmer
Copy link
Member

kgilmer commented Feb 21, 2021

They simply pointed out that you aren't including the setup.py file

Hmm, yes looking at the output setup.py (dpkg -c ~/Downloads/regolith-rofication_1.3.1-1_amd64.deb | grep setup) is not there. I checked a few other packages in Regolith for that file, and none of them have it. For example, python3-i3ipc. Why is this file required only on this one repo? I can look into fixes but python/debian packaging seems like a wonderland unto itself and I simply don't have time to become an expert on this. Is there more specfiics as to what you require other than "setup.py"? Say I make a change, how can I test that my update satisfies the requirements of Arch policy or build logic?

I'm guessing this is a case where a package agnostic build approach would shine. I'm guessing your porting is constrained by what your native packaging system needs and what the debian packages provide.

you're also not including the i3xrocks module file, 80_rofication.

This file is now shipped in the binary package i3xrocks-rofication defined in regolith-i3xrocks-config. This was done to decouple rofication from i3bar.

@gardotd426
Copy link
Member

Hmm, yes looking at the output setup.py (dpkg -c ~/Downloads/regolith-rofication_1.3.1-1_amd64.deb | grep setup) is not there. I checked a few other packages in Regolith for that file, and none of them have it. For example, python3-i3ipc. Why is this file required only on this one repo? I can look into fixes but python/debian packaging seems like a wonderland unto itself and I simply don't have time to become an expert on this. Is there more specfiics as to what you require other than "setup.py"? Say I make a change, how can I test that my update satisfies the requirements of Arch policy or build logic?

Well, that's the thing. Without running the setup.py script, I'm not really sure how it's even being correctly installed. Because just copying the files from the .deb to /usr/bin (which is literally all that dpkg does too) should fail with a Module not found: rofication/no module named "rofication" Python error. So that's why I was confused, and no, you don't need to become an expert on it, I'll explain it this way: Imagine if to install a package from a PPA, instead of just adding the PPA, sudo apt update-ing and sudo apt install PPAPACKAGE-ing, you had to do all that, then cd into some directory and run an additional script. That wouldn't generally be cool, right? Same thing for AUR packages. But see, you're not doing it w/ dpkg, so it shouldn't be necessary on Arch. The only thing we need to figure out now is why it's not working on Arch without the setup.py script running. But I know you're busy, so I'll look into it, I can spin up some Ubuntu VMs and add the PPAs and see what happens, and then I can just compare the differences.

It's not anything like python-i3ipc, I made sure to include that in the dependencies for the PKGBUILD, so that's getting installed already. But yeah the fact that you don't include it or anything special on Ubuntu (or Regolith Proper) is more than enough to get me on the right track, so don't worry about it.

This file is now shipped in the binary package i3xrocks-rofication defined in regolith-i3xrocks-config. This was done to decouple rofication from i3bar.

Ah cool, I noticed you'd added a couple new modules lately, I'll go ahead and add that to the i3xrocks package then.

I'll let you know once I figure it out, but I already have a couple ideas. Thanks for the guidance

@kgilmer
Copy link
Member

kgilmer commented Feb 21, 2021

My apologies if I came across defensive or weird there, it wasn't intentional. Somehow I have a very low patience threshold when dealing w/ python issues in general. I have no good reason, just a personality quirk I guess. In any case, thanks for your clarifications. Yes, it's true that despite the deb package simply saying it's shipping the python file, actually quite a bit more comes along for the ride:

$ dpkg -c ~/Downloads/regolith-rofication_1.3.1-1_amd64.deb 
drwxr-xr-x root/root         0 2020-12-27 11:27 ./
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/bin/
-rwxr-xr-x root/root       417 2020-12-27 11:27 ./usr/bin/rofication-daemon
-rwxr-xr-x root/root       169 2020-12-27 11:27 ./usr/bin/rofication-gui
-rwxr-xr-x root/root      1157 2020-12-27 11:27 ./usr/bin/rofication-status
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/python3/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/python3/dist-packages/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/
-rw-r--r-- root/root       414 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/__init__.py
-rw-r--r-- root/root      1536 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_client.py
-rw-r--r-- root/root      3345 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_dbus.py
-rw-r--r-- root/root      3665 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_gui.py
-rw-r--r-- root/root       214 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_metadata.py
-rw-r--r-- root/root      1260 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_notification.py
-rw-r--r-- root/root      3908 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_queue.py
-rw-r--r-- root/root      2893 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_server.py
-rw-r--r-- root/root       297 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_static.py
-rw-r--r-- root/root      1092 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_util.py
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/resources/
-rw-r--r-- root/root       163 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/resources/__init__.py
-rw-r--r-- root/root      1042 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/resources/_static.py
-rw-r--r-- root/root       533 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication-1.2.2.egg-info
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/share/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/share/doc/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/share/doc/regolith-rofication/
-rw-r--r-- root/root      1121 2020-12-27 11:27 ./usr/share/doc/regolith-rofication/changelog.Debian.gz
-rw-r--r-- root/root      1231 2020-12-27 11:27 ./usr/share/doc/regolith-rofication/copyright

Again, I can't say if this should be sufficient for a working Python module, but perhaps this provides some help. I do know that a python-specifc build system is used to generate the package, and this is integrated w/ the debian packaging. I'm not sure but maybe this is doing some magic that does whatever setup.py is doing in other packages..?

If you come up w/ anything specific you'd like to try, I'm more that happy to build some test packages w code changes, just LMK 😄

@gardotd426
Copy link
Member

Oh no worries man, it's all good.

But yeah I know about the other files aside from what's in /usr/bin, I install the whole package, I'm not just pulling deb files. There's a script in the PKGBUILD like this:

extract_deb() {
    local tmpdir="$(basename "${1%.deb}")"
    rm -Rf "$tmpdir"
    mkdir "$tmpdir"
    cd "$tmpdir"
    ar x "$1"
    tar -C "${pkgdir}" -xf data.tar.xz
}

And then for each .deb, I just have to run extract_deb "${srcdir}"/pkgname-pkgver.deb, blah blah blah. So I'm installing all those files too, I was just simplifying things earlier. I get everything that's included in the .deb package, but I don't get the setup.py because that's not with the .deb package, it's in the upstream original dev's repo, so obviously we shouldn't need it. I'll figure it out, no worries. It's probably something stupid, like I said I already have a couple ideas.

Also if necessary I could potentially just break regolith-rofication into its own separate PKGBUILD instead of being part of a meta-package, and instead of using the deb, just clone the repo, because I can clone the repo and build and install from within the PKGBUILD just fine, it's just, it's hard to explain but the way PKGBUILDs work for meta-packages, defining one git repo for one package can mess all the rest up, but honestly that's not even a remotely difficult thing to do, it would literally just be mostly a copy and paste thing, and I'd just have two PKGBUILD files.

But yeah, I think debhelper-python is what's doing it for you on the Ubuntu side, which makes a ton of sense. No worries, I have like 5 different solutions, I'm just researching which one would be the "best" one, or at least the one I'm likely to get the least complaints about lol.

@gardotd426
Copy link
Member

I got it figured out. I just had to dig deep into some Arch documentation about including Python packages in PKGBUILDs, because the normal procedure/recommendation is to use PyPi/Pip, but they do have procedures for PKGBUILDs for Python packages that aren't available there. I just tested it and it works great, I'll be updating the PKGBUILD in a few.

@kgilmer
Copy link
Member

kgilmer commented Mar 16, 2021

@marc0der,

We could repeat this for as many distros as we would like to support. What do you think of this approach?

I have started working working on something similar to this in order to support Debian and Ubuntu. It's nothing fancy, just some bash code and a JSON document that describes all packages necessary to build the Regolith DE. It's somewhat simplified at the moment as it's only building a subset. There is a repo that contains the scripts (mainly just a convention): https://github.com/regolith-linux/pkgrepo-lib and then "child" repos (currently only https://github.com/regolith-linux/pkgrepo-ubuntu-focal) that include the scripts, add their own specific package models (Debian generally needs more than Ubuntu) and any custom scripts. Rather than just straight docker, each "child" repo uses gh actions to essentially do the same thing ~ create a container and do some work. The results are committed back to the source repo in my current case, as I'm able to use the Github Pages feature to host the repositories. Unsure if this is how things will be for production but works well enough for me now.

I plan to use this approach to completely automate generation of Ubuntu Bionic, Focal, Groovy and Debian Buster, Bullseye package repos.

What do you think? Is there anything about the system I describe here that would be prohibitive to non Debian based systems?

@kgilmer
Copy link
Member

kgilmer commented Mar 18, 2021

FYI I've been adding issues to give a little more insight as to what I'm doing: https://github.com/regolith-linux/pkgrepo-lib/issues

@marc0der
Copy link
Author

marc0der commented Mar 19, 2021

Hi @kgilmer! Firstly, a big apology that I haven't replied to you sooner. I've been a bit swamped with work the last while, so I haven't had much chance to look at this stuff again. That said, I hope to have a closer look over the weekend. Happy to have a play and give some feedback regarding this.

@kgilmer
Copy link
Member

kgilmer commented Mar 20, 2021

I'm suffering from a bad case of day-job myself recently, no worries @marc0der . Come back to this when you have the time and interest 😄

@kgilmer
Copy link
Member

kgilmer commented Mar 28, 2021

Things have moved around a bit now that the names matter..

@neevparikh
Copy link

neevparikh commented Sep 8, 2021

Hey all, any updates on moving this forward since March? I'm already testing regolith-de on arch, @gardotd426, and it's working totally great.

@a-crate
Copy link

a-crate commented Nov 26, 2021

I'd like to chime in as well here, I'm looking at porting regolith to gentoo, and the first hurdle from my perspective is determining how and when releases are cut. I'm under the impression that @gardotd426 has been unpacking .deb files from the PPA but that doesn't really work for me on a source based distribution. I've had a look around and while the package model in https://github.com/regolith-linux/package-repo is great for determining where I can grab sources and what sources I need, it doesn't really help me with versioning, nor is there a consistent source of truth for what source files were used to build which version. Is there any way to add a commit field in the package model, and possibly have something like a github release on version update to show the package model for each release?

@gardotd426
Copy link
Member

Yeah I struggled for quite a while with trying to figure out how exactly to version stuff (especially since with metapackages in the AUR, there can only be one package version), before I moved most of the non-Regolith-specific packages over to Arch repo packages (like i3-gaps, picom, etc), so my PKGBUILD mostly just provides regolith-specific stuff, so I landed on using the current Regolith version. So right now it's 1.6. But yeah there seems to be little rhyme or reason on the versioning on source repos, I'm in the process now of moving whatever I can over to being built from source instead of just extracting .debs.

@a-crate I got your email, I'll email you back shortly answering some of your questions

@kgilmer
Copy link
Member

kgilmer commented Nov 27, 2021

Hi @a-crate , that's exciting news. I learned a lot from running Gentoo for years and years. It would be amazing to have Regolith run there as well.

Preamble: Only the release stage in Regolith is versioned. stable and unstable lack a cross-package versioning scheme.

the first hurdle from my perspective is determining how and when releases are cut

Things are a bit weird right now due to the transition from Regolith 1.x to 2. This major version bump constitutes the change to clean things up and do things in simpler ways while breaking with the old, launchpad.net-based build/repo model. The 2.x build stuff in part is to make it easier for efforts such as yours to port the packages to other distributions. 2.0 is still in development and so if there are ways that this approach can be improved this is certainly the time to make the right changes.

As such, you are not finding any versioning stuff because there has yet to be a version of Regolith 2.x. The general workflow for how packages go from source in git repos to installed packages on user's systems is described here.

In summary, the "version" of Regolith 2.x is the set of git URLs and branch/tag names specified in the root package model + any distro specific overlays (ex).

nor is there a consistent source of truth for what source files were used to build which version

Again, since we are before an explicit version, this information does not yet exist. My thinking here is that rather than having a single package model + overlays, rather these will be maintained per distro. So the file package-repo/regolith-2.0-package-model.json would move to something like package-repo/stages/unstable/regolith-2.0-package-model.json. And package-repo/distros/debian/bullseye/* -> package-repo/distros/debian/bullseye/unstable/*. This would essentially tie package model snapshots to particular stages. When releases are cut, the corresponding release tags will snapshot these models to a particular version. (Yes there is a problem of package models specifying (changing) branches over static tags that will need to be addressed. Each regolith package will need a release tag and the release models updated with that tag before release).

Is there any way to add a commit field in the package model

Yes the current field branch should be updated to be more general and work with branches, tags, or commits. (TBD)

and possibly have something like a github release on version update to show the package model for each release?

Yes, package-repo will have tag snapshots for public releases. I'd like to tag repos with release versions as well but am unsure of how I can easily automate that and will not be doing it by hand.

Hopefully that answers your questions to some degree @a-crate . FWIW if you were to provide a GitHub action like this for your target distro, all of the versioning would generally be handled and you'd just* need need to build and stage packages for your target distro. EG when Regolith 2.0 had it's first release, your action would be called just like the Debian and Ubuntu actions.

(*: of course it's never that simple and I'm sure there are things I'm entirely unaware of that need to be solved.)

@kgilmer
Copy link
Member

kgilmer commented Nov 27, 2021

More details here (I just refreshed this page): https://github.com/regolith-linux/regolith-system/wiki/Package-Build-Automation

@gardotd426
Copy link
Member

Things are a bit weird right now due to the transition from Regolith 1.x to 2. This major version bump constitutes the change to clean things up and do things in simpler ways while breaking with the old, launchpad.net-based build/repo model. The 2.x build stuff in part is to make it easier for efforts such as yours to port the packages to other distributions. 2.0 is still in development and so if there are ways that this approach can be improved this is certainly the time to make the right changes.

Yeah, this along the lines of what I told @a-crate in my email. It is a bit of an awkward situation right now. Personally for the Arch side of things, I've just been sticking with "Release" debs while slowly transitioning some of the packages over to building from source (and I'm using repo packages for all non-Regolith dependencies, like i3-gaps, etc). Meanwhile I have a private alpha that I'm testing using some stuff from package-repo in preparation for the move to 2.0.

I'm interested to see where 2.0 goes.

I've noticed though that one thing preventing from a full move over to source builds are that quite a few packages are missing from package-repo, like all the looks/styles other than lascaille I believe (I know Midnight, Pop OS, Dracula, Gruvbox, etc. are all missing). Is that intentional as in they're not going to be in 2.0? Or have you just not gotten around to it yet?

Is there any more you can tell me about the move to ilia and away from Rofi? I notice that the alpha still uses rofi. I've got ilia built and working on my alpha build, but I don't have it integrated into the system or anything (it's still using rofi).

@gardotd426
Copy link
Member

I'll add that I just looked over the Package Manager Agnostic Build System goals and it sounds great. Really all I would need for Arch is a way to build each project from source. The beauty of pacman/makepkg is that if I can build the project from source I can create a pacman package easily. Obviously for stuff like the regolith-desktop-config with the config files don't need buildling, I can just pull them in, patch them as needed and package them. Same would go for any styles and icon themes, and anything script based.

Currently in the AUR package I'm building regolith-rofication, i3xrocks, and remontoire from source. Seeing that you've added ayu-theme to package-repo, I'll probably use git for that too instead of extracting the .deb.

Submodules would be great. Right now if I want to use package-repo, I have to trawl through the available packages in regolith-2.0-package-model.json, manually grab git repos, see what's there, clone, etc.

With submodules where each submodule is a repo for a regolith package (or group of packages), I could easily set it up to clone it, build each in order, and package.

@a-crate
Copy link

a-crate commented Nov 27, 2021

As such, you are not finding any versioning stuff because there has yet to be a version of Regolith 2.x. The general workflow for how packages go from source in git repos to installed packages on user's systems is [described here].

Ah, okay. I see. Right now I'm moving forward with just creating ebuilds from the most recent source tarballs from launchpad for 1.6, is this generally stable for now? Will you be moving away from launchpad entirely with 2.0, or will it just become a place to host built packages for debian+ubuntu, with other distros rolling their own?

I'm personally not a fan of submodules, as I need to sandbox every build to make my package manager happy, but they don't interfere at all as long as package-model.json exists.

I will wait until I've figured out manual packaging before automating anything but this has been very helpful, thank you.

@gardotd426
Copy link
Member

gardotd426 commented Nov 27, 2021

I'm personally not a fan of submodules, as I need to sandbox every build to make my package manager happy, but they don't interfere at all as long as package-model.json exists.

You can still check out individual submodules as repositories though, right?

Like if you have a repo with say 4 submodules, sub-a, sub-b, sub-c, and sub-d, you can also just clone sub-a, sub-b, sub-c, and sub-d separately at the same tag as the parent repo provides.

Here's an example of that, Tk-Glitch's PKGBUILDs repo is full of a bunch of submodules for different PKGBUILD projects. But I don't use that PKGBUILDs repo, I just clone the ones I want to use individually. Does that not work for what you're talking about? I mean I guess if you could just use package-model.json it doesn't really matter, but I figure it'd be way easier to do it the checkout-submodules-individually way.

@kgilmer
Copy link
Member

kgilmer commented Nov 27, 2021

gardotd426:

quite a few packages are missing from package-repo, like all the looks/styles other than lascaille I believe (I know Midnight, Pop OS, Dracula, Gruvbox, etc. are all missing). Is that intentional as in they're not going to be in 2.0? Or have you just not gotten around to it yet?

I haven't gotten around to it yet. In a bit more detail: the Regolith Look "system" in Regolith 1.x ended up not being flexible enough for some use cases, like making it easy to use a different bar or launcher and have it "just work". 2.0 makes breaking changes here in an effort to bridge the gap. I'm also thinking of doing less here in terms of prepackaged themes, from an "official regolith" prespective. In that line, the 1.x themes that do carry over will likely go into an -extras package or perhaps in a package not owned by the Regolith org at all.

If you have particular looks (or other functionality) that you want to see in 2.0, please LMK.

Is there any more you can tell me about the move to ilia and away from Rofi?

Development continues :). At this point all Rofi functionality needed by Regolith has been implemented except for window switching and file search. I hope to complete window switching today. Once the core functionality works I'll considered it in a good enough state for an alpha release. At that point I'll probably spend some time in package-repo to implement the stages as discussed above so that the 3 stages are implemented for Regolith 2.0. Once any other critical bugs are addressed I think this would be a good time to introduce a first alpha "release". Note: I have yet to do much polish on the UI and so I would expect the appearance of what Ilia looks like now to be somewhat different at 2.0 release time.

I'll add that I just looked over the Package Manager Agnostic Build System goals and it sounds great.

Fantastic! Thanks for the feedback

I have to trawl through the available packages in regolith-2.0-package-model.json, manually grab git repos, see what's there, clone, etc.

Well that's what the scripts in this github action do. If you come up with something similar for Arch you would only need to implement "building and publishing a given package". To be clear, we would add a github workflow that calls your action to generate packages. If you look at the shell script associated with the action, you'll see places where debuild and reprerpo are called. Ideally (but probably not practically) equivalent commands could be substituted here for "build a package from source" and "publish a package to the repo" and that would be all that's needed.

(Ideally the build infra would not be so tightly coupled to github, but as a matter of time I think it's the best compromise and only hope that the carpet isn't pulled from under our feet.)

@kgilmer
Copy link
Member

kgilmer commented Nov 27, 2021

a-crate:

I'm moving forward with just creating ebuilds from the most recent source tarballs from launchpad for 1.6, is this generally stable for now?

This is very stable, as no future updates to 1.x are planned.

Will you be moving away from launchpad entirely with 2.0, or will it just become a place to host built packages for debian+ubuntu, with other distros rolling their own?

This question is still open. Launchpad has this special "superpower". There is a command on all Ubuntu's (add-apt-repository) which automagically works with keys hosted on launchpad. This allows users to more easily add a launchpad PPA, with just one command. Ideally I could deprecate launchpad altogether (for simplicity, I am grateful for Launchpad as a system but I think Regolith has outgrown the use cases it was designed for) but this comes at a user cost of a slightly more complex install process. I will probably opt to not go with launchpad but if it ends up that many users have difficulty, confusion, or lose trust with the github-hosted repo, I may implement something that would copy packages from a release stage to a launchpad.net PPA. This could simply considered to be a mirror.

Regarding submodules, I agree that @gardotd426 's proposal is good in that it is simple and consistent with git. However in practice I've had a trouble with submodules in past projects and prefer to avoid them as a keystone to rest other work upon.

@kgilmer
Copy link
Member

kgilmer commented Nov 27, 2021

In case it's preferred, I've created a slack channel "distro" just now. If anyone would prefer that format for discussion rather than this issue.

https://regolith-linux.slack.com/archives/C02NBTBRV9D

@a-crate
Copy link

a-crate commented Nov 27, 2021

You can still check out individual submodules as repositories though, right?

Yes but it's functionally the same as cloning the repository directly for me. Like, for example, if I were to package the modules in that repository for gentoo, I would need a package for each submodule. I could git clone --recurse-submodules but I wouldn't be allowed to re-use the sources I cloned for a multiple packages. If I package submoduleA, I have to clone the whole thing again when I package submoduleB. The only way to do so would be to create a big package that encompasses every submodule at once. This would work fine it's just generally not how you're supposed to package things in gentoo.
The difference is really that PKGBUILDs are allowed to install multiple packages, but ebuilds can only install one package.

I'm not opposed to them at all, I don't want to seem like I'm trying to dissuade from submodules. I would just prefer that they aren't the only method.

In case it's preferred, I've created a slack channel "distro" just now.

I will jump in there if I have any issues, packaging is going relatively smoothly for now.

@kgilmer
Copy link
Member

kgilmer commented Dec 6, 2021

FYI the package builder now supports stages. To reduce unnecessary variance, the stages in Regolith 2 will adopt the same names as Debian.

Before: unstable -> stable -> release
After: unstable -> testing -> release

Stages are implemented without any new concepts to the build system. The stage just becomes another dimension of the build matrix, which is now: stage x distro x codename x architecture. Each stage has a seperate copy of the package model. For any given package, the package model specifies a git ref to what to build from. When a testing event is completed, such that all packages can be promoted to another stage, the package model is synced and pushed to the repo. The package builder will be notified and via manifest checking will build new package versions as needed. The release package model should reference either release branches or specific commits.

@gardotd426
Copy link
Member

That sounds pretty awesome, actually.

About the themes and styles you mentioned here:

If you have particular looks (or other functionality) that you want to see in 2.0, please LMK.

My thinking is just that, for my port, I want it to be the whole regolith experience. That requires some editorializing on my part, but as far as the styles go, I think I should provide all styles that regolith supports. Ayu, Ayu-Dark, Ayu-Mirage, Cahuella, Dracula, Gruvbox, Lascaille, Midnight, Nord, Pop-OS, Solarized-Dark, Solarized-Light, and Ubuntu.

SOME of those looks' theme and icon theme components (as in the stuff that goes in /usr/share/themes and /usr/share/icons) I can get from repo packages, and the rest I can probably get from the ubuntu repos (a lot of them aren't in the AUR or arch repos), or somewhere other than package-repo, but what I definitely need in a repo inside package-repo is everything in /etc/regolith/styles/, if that makes sense.

Like, I can get all the components myself, but the theme files, I would like a repository for. Is that doable?*

* I know that a lot of things are changing in 2.0 and you might not even use some of those themes and might add new ones, but I"m saying whatever themes/styles you do provide /etc/regolith/styles/ files for, that's what I'd like a repo for, if that makes sehse.

@a-crate
Copy link

a-crate commented Dec 6, 2021

How often will release be updated compared to testing/stable? Is release something that will only change on the timescale of debian/ubuntu releases with testing updating in between, or is it quicker than that?
Edited to clarify: for example, the rofi 1.7.0 incompatibility. Right now that isn't going to be fixed until the next release, and that won't be a problem because ubuntu won't have rofi 1.7 until their next release. If a similar issue comes up in the future, will release be patched or will the fix stay in testing until another release?

I've not looked into the automation at all yet, I know for gentoo I really just want a system to make a tarball of the source and put it in a git repo with a tag for the version. That sounds relatively simple, I assume it's capable of that.

@kgilmer
Copy link
Member

kgilmer commented Dec 7, 2021

@gardotd426:

I definitely need in a repo inside package-repo is everything in /etc/regolith/styles/

Sure, that shouldn't be a problem. FYI in Regolith 2 the styles files are broken into two categories: style definitions (colors, fonts, etc.) and application bindings (specific Xres files for a particular user program, scripts, etc.). For the former, I only have one example at the moment: regolith-look-default https://github.com/regolith-linux/regolith-look-default. The intention of this look is to be easy to port by not straying too far from commonly available themes and fonts files, and also to be small.

Other looks will likely be stand-alone packages or possibly a -contrib or -extra style package which may contain multiple styles. Everyone has different tastes and it's impossible to please everyone. So I'd like to focus more on a handful of visual looks that each offer something compelling, and provide better docs on how to tweak, customize, and copy existing looks to make new ones. Using existing tools like pywal and so forth may be a good option as well.

@a-crate:

How often will release be updated compared to testing/stable? Is release something that will only change on the timescale of debian/ubuntu releases with testing updating in between, or is it quicker than that?

Yes, on the order of several months to years. We've been roughly tracking Ubuntu's release cadence and doesn't seem like anything else is going to replace that in the short term.

If a similar issue comes up in the future, will release be patched or will the fix stay in testing until another release?

Due to the time costs of testing, releases will not be patched. If a serious problem is found, we would do a point release, or I guess this is what you mean by a patch perhaps. Regarding problems that may impact distros other than Ubuntu, I would like to do my best to provide support regardless of the distro until/if it becomes too time costly for me personally. For now at least any issue that comes up from Arch, Gentoo, or others will be treated with the same gravity as Ubuntu or Debian bugs. The specific example you bring up is a little odd because in this case we're fully replacing one program for another, and we're in the middle of a major version bump.

That sounds relatively simple, I assume it's capable of that.

This would be an add-on to the build system for the release stage, of tagging and generating releases from build packages. I'll have to think on this a little more regarding the simplest approach but I agree that released Regolith packages should be easily trackable via releases. More to follow here. If you have more specific requirements as well please LMK.

@gardotd426
Copy link
Member

Other looks will likely be stand-alone packages or possibly a -contrib or -extra style package which may contain multiple styles. Everyone has different tastes and it's impossible to please everyone. So I'd like to focus more on a handful of visual looks that each offer something compelling, and provide better docs on how to tweak, customize, and copy existing looks to make new ones. Using existing tools like pywal and so forth may be a good option as well.

Something like a -contrib or -extra would be perfectly fine for me. Like I said, Arch+AUR doesn't provide those /etc/regolith/styles files except for my package, and of course users of my PKGBUILD can tweak and theme their install just like they would a regular i3 install, but since regolith is a full desktop environment, IMO it should provide at least some "looks," and potentially an easy way to add additional one (sort of how gnome and plasma can easily use stuff in /usr/share/themes. But yeah, as long as I have easy repo access to some of the typical styles (like dracula, pop-os, ubuntu, cahuella, etc.) I'm perfectly happy with that, if that makes sense.

Due to the time costs of testing, releases will not be patched. If a serious problem is found, we would do a point release, or I guess this is what you mean by a patch perhaps. Regarding problems that may impact distros other than Ubuntu, I would like to do my best to provide support regardless of the distro until/if it becomes too time costly for me personally. For now at least any issue that comes up from Arch, Gentoo, or others will be treated with the same gravity as Ubuntu or Debian bugs. The specific example you bring up is a little odd because in this case we're fully replacing one program for another, and we're in the middle of a major version bump.

And I'm sure you'd be willing to accept PRs for stuff like that, as in stuff that affects other distros, right? So if something came up that affected gentoo, @a-crate could file a PR to fix it?

@a-crate
Copy link

a-crate commented Dec 8, 2021

Regarding problems that may impact distros other than Ubuntu, I would like to do my best to provide support regardless of the distro until/if it becomes too time costly for me personally. For now at least any issue that comes up from Arch, Gentoo, or others will be treated with the same gravity as Ubuntu or Debian bugs. The specific example you bring up is a little odd because in this case we're fully replacing one program for another, and we're in the middle of a major version bump.

Oh, sure. I don't think it would be fair to expect you to spend tons of time working on a side project. And I figured that any patches need for gentoo issues would be coming from me anyway, if for no other reason than I'm the one running it. I figured you were likely targeting the same timescale as debian & co. and wanted to be sure fixes for other distros could still make their way to release if necessary, since the timescale for us is very different.

@kgilmer
Copy link
Member

kgilmer commented Dec 9, 2021

And I'm sure you'd be willing to accept PRs for stuff like that, as in stuff that affects other distros, right? So if something came up that affected gentoo, @a-crate could file a PR to fix it?

Yes of course. As contributors, you're able to create PRs directly against the repos, as well as review and merge them.

@kgilmer
Copy link
Member

kgilmer commented Dec 10, 2021

I took another stab at providing documentation for extending the package builder for other types: https://github.com/regolith-linux/package-repo/blob/master/CONTRIBUTING.md

@kgilmer
Copy link
Member

kgilmer commented Jan 29, 2022

To keep this issue up to date, the build system underwent another rewrite and now lives here: https://github.com/regolith-linux/voulage

I have yet to implement one but the automation (bash based) allows for extension via sourcing a script that would implement well-known functions (an implicit interface): https://github.com/regolith-linux/voulage/blob/main/.github/scripts/ext-template.sh

@gardotd426
Copy link
Member

gardotd426 commented Jan 29, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants