Outside of modifying the stage0/00-configure-apt/files/{sources,raspi}.list files, there is not way to point all following apt calls as part of the Raspbian image build at custom mirrors.

Why is that bad? It looks like the trend is to move all customisation into the config file, which is just not practical.

Well, I'm not sure that another way is bad...

I thought my suggestion was along the same lines as #391

If another change along these lines is not what you want, then maybe I should be asking something like: What is the best way to make that kind of modification in my workflow? (let me explain from a high level what that is...)

I am trying to use pi-gen to build an image using a private mirror of the upstream raspbian repo (so that I can control and know what comes in). I don't really want to also maintain a version of the pi-gen git repository, but I would like to have the ability to set those custom apt sources.

I am running the build currently using docker with build-docker.sh

Would you suggest I run a set of scripts/apply patches at some point to touch up those files or overwrite them?

Should my patches be applied before build-docker.sh or in some way during build.sh execution?

I had created a feature branch with a single commit to implement what I was originally suggesting if you are interested:
bamartin125@c4b93e0

By the way - Thanks for the quick reply!

The suggested code changes are in pull request #445 .

@XECDesign , let me know what you think. If you do lean towards rejecting the pull request, please let me know what your thoughts are on a better way to solve this issue on my end (hopefully without suggestion that I should maintain my own version of pi-gen).

Thanks!

It becomes a bit awkward because I want to merge the arm64 branch into master and have an ARCH parameter which selects which set of source files to use. It may also make sense to have an additional repo added in a later stage.

Pi-gen was never written as a tool to generate custom Raspberry Pi OS images, but people kept moving things from the stages into the config file. Despite some initial resistance, I'm okay with it, but I don't see a way to do it in this case without making things quite messy.

I can't come up with a clean way to do this without getting in the way of what pi-gen is intended for (and how I have to use it) and still allow you to use it as a tool to create Raspberry Pi OS variants without editing any files other than config.

Thanks for the background info. It always helps when there is a little more context to see where things fit in the bigger picture. Especially regarding the ARCH parameter that you are looking to add, I am actually starting to side with you I think. I think the ARCH parameter is a perfect solution for allowing people to easily choose to build the 32 or 64 bit arch. With that said, I am not honestly sure how to solve the simple ARCH parameter problem and the custom mirror problem at the same time without one effectively overriding the other. Overrides in that sense could become quite confusing and a little harder to troubleshoot.

I will say as a side note that I was having trouble building an image for a while because the mirror that I was getting pushed to (local to me?) was timing out occasionally (even though the speeds were fine?). I was able to fix that when I started using a custom mirror.

I'd like a little more time to think about what you've said before we close the issue. If not a code change to make it easier to make these kinds of project specific deviations, maybe I can come up with at least a suggestion for how (from the outside) to make these kinds of changes which are counter the configurations that the project would like to expose.

Lastly, I certainly don't want to get in the way of how you guys have to use the tool.

Internally, we use apt-cacher-ng to make things a little more reliable.

When I was trying to come up with a solution, I was thinking of a CUSTOM_LIST_DIR parameter which is normally unset. But if it's set, it pulls the whole sources.list directory from CUSTOM_LIST_DIR instead. That doesn't help with the bootstrap stage though. Maybe an additional BOOTSTRAP_URL variable... But it gets a bit messy.

Coming back to this now a few months later to try and get this resolved.

@XECDesign, I took your suggestions for the most part to see how they'd work. Your BOOTSTRAP_URL suggestion was easily implemented actually through a build.sh addition and a change to stage0/prerun.sh - something like I had before.

I think I have most of the CUSTOM_LIST/CUSTOM_LIST_DIR pieces are right, but I am having a hard time figuring out how to map the files/dirs specified in the config file into the resulting docker container from running build-docker.sh. Additionally, I cannot test build.sh directly (I can only run the build through docker as I am not allowed by my administrator to install the additional packages required to run the build manually among other things...).

I think I'm going to have to create a new issues against build-docker.sh in order to allow arbitrary files/dirs to be volume-mounted into the pi-gen container. Before I do, I was going to let you speak into what I'm doing and let me know if I'm doing something wrong here.

My workflow right now looks like the following:

I have a wrapping project folder to pi-gen that I am currently calling pi-gen-bootstrap which I use to make a docker image which contains the pi-gen tool along with the requirements to run it (docker, bash, git).

Dir structure:

pi-gen-bootstrap/
|-- Dockerfile
|-- pi-gen

Dockerfile contents:

FROM docker
RUN apk add bash git
COPY pi-gen /root/
WORKDIR /root

I docker build and then take the resulting image and run it using a command like:

docker run --rm -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $HOME/deploy:/root/deploy \
    -v $HOME/config:/root/config \
    -v $HOME/sources.list:/root/sources.list \
    -v $HOME/sources.list.d:/root/sources.list.d \
    pi-gen-bootstrap \
    /root/build-docker.sh

Which doesn't really work because my custom sources.list and sources.list.d entries don't actually get mounted into the pi-gen container. Because build-docker.sh calls docker run under the hood on the pi-gen image, I have no way to actually get my desired files/dirs volume-mounted at that point.

I realize that you aren't obligated to support my particular workflow (I hate it myself, but this is what I'm allowed to do at my org for now), but within the docker-in-docker confines and all, do you see that I could be doing something else to get arbitrary files/dirs volume-mounted or otherwise accessible to the container that is running build.sh?.

This last 20% is feeling more like a support issue now rather than a real issue against the pi-gen tool. And I am sorry for that. But, any suggestions you could offer would be much appreciated. Thanks again.

Okay, I think I've figured out how best to make this stuff work and I have just now pushed up my latest changes to the PR.

Please let me know if there are any issues with the proposed changes.