Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: modify buildsystem docs in README.md #825

Closed
wants to merge 1 commit into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
202 changes: 141 additions & 61 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,36 @@
# Freifunk Berlin Firmware
https://wiki.freifunk.net/Berlin:Firmware

This is the build-system for the Firmware of Freifunk Berlin.
The firmware is based on vanilla [OpenWrt](https://openwrt.org/start) with some modifications (to fix
broken stuff in OpenWrt itself or for example LuCI) and additional default packages/configuration settings.
This is the build-system for the Firmware of Freifunk Berlin. The firmware is
based on vanilla [OpenWrt](https://openwrt.org/start) with some modifications
(to fix broken stuff in OpenWrt itself or for example LuCI) and additional
default packages/configuration settings.


## Contact / More information

More user relevant information about the firmware are on the wiki page at: https://wiki.freifunk.net/Berlin:Firmware. There you can also find the
* [ReleaseNotes](https://wiki.freifunk.net/Berlin:Firmware/v1.0.2)
* a tutorial ([en](https://wiki.freifunk.net/Berlin:Firmware:En:Howto) / [de](https://wiki.freifunk.net/Berlin:Firmware:Howto)) on router configuration
More user relevant information about the firmware are on the wiki page at: https://wiki.freifunk.net/Berlin:Firmware.
There you can also find the
* [ReleaseNotes](https://wiki.freifunk.net/Berlin:Firmware/v1.0.3)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To reflect the current state: It feels to me, that we should not reference the wiki here, as nobody takes care of updating the pages. It seems best to reference the Github-release pages (https://github.com/freifunk-berlin/firmware/releases) here, which link to the correct ReleaseNotes of the corresponding release.

* a tutorial ([en](https://wiki.freifunk.net/Berlin:Firmware/en/HowTo) / [de](https://wiki.freifunk.net/Berlin:Firmware/HowTo)) on router configuration

For questions write a mail to <[email protected]> or come to our
weekly meetings. If you find bugs please report them at: https://github.com/freifunk-berlin/firmware/issues

For questions write a mail to <[email protected]> or come to our weekly meetings.
If you find bugs please report them at: https://github.com/freifunk-berlin/firmware/issues

## Development

### Info

For the Berlin Freifunk firmware we use vanilla OpenWrt with additional patches
and packages. The Makefile automates firmware
creation and apply patches / integrates custom freifunk packages. All custom
patches are located in *patches/* and all additional packages can be found at
http://github.com/freifunk-berlin/packages_berlin.
How does this stuff work?

The idea is to download OpenWrt via git, patch it with all patches in the
`patches/` folder and configure it according to the configuration snippets in
`configs`. Then use the OpenWrt buildsystem to build all packages and the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be configs/ , as it's a folder

Imagebuilder and use the latter to assemble the final image with lists of
packages in packages for the router.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"with lists of packages in packaglists/ for the router.

All additional packages can be found at http://github.com/freifunk-berlin/packages_berlin.
Akira25 marked this conversation as resolved.
Show resolved Hide resolved
Akira25 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we mention "gluon-packages" and "freifunk-potsdam-packages" also. (not shure if ffp-packages made it up to this repo already)



### Build Prerequisites

Expand All @@ -48,6 +56,41 @@ pacman -S base-devel git ncurses lib32-zlib gawk time unzip perl-xml-libxml \
flex wget gettext quilt python2 openssl
```


### Build-System-Structure

Where can I change something? What does these files do?
Akira25 marked this conversation as resolved.
Show resolved Hide resolved

```
:: Settings

config.mk - OpenWrt git repository, revision, select what package lists to build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"OpenWrt git repository, revision" is controlled by modules-file

feeds.conf - OpenWrt feeds.conf that is used for the package feeds during build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"feeds.conf" is converted to modules


configs/ - further configurations
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"further configurations" replace by "snippets of the OpenWrt config-options (compare to call "make menuconfig" in openwrt/)" or something alike

common.config - OpenWrt config-parameters for all targets
$(TARGET).config - target-specific config. Gets added to common.config. Options from target.config
overwrite those from common.config.
common-autobuild.config - gets added on top , if IS_BUILDBOT=yes is defined


patches/ - Patches against the OpenWrt directory
packagelists/ - Package Lists
profiles/ - List of router profiles for each architecture - names come from the Imagebuilder

Akira25 marked this conversation as resolved.
Show resolved Hide resolved
Akira25 marked this conversation as resolved.
Show resolved Hide resolved
:: Building

Makefile - Does all the stuff. Cloning and Updating OpenWrt, patching, running make
assemble_firmware.sh - Script for running the Imagebuilder for a target


:: Images

firmwares/ - Folder where the final images end up

```


### Building all firmwares

To get the source and build the firmware locally use:
Expand All @@ -66,17 +109,18 @@ connection can improve the build time.

You need approximately 10GB of space for the build.

### Directory Layout

You can find the actual firmware images generated by the ImageBuilder (and the ImageBuilder itself)
in `firmwares`. The layout looks like the following:
### Directory Layout of `firmwares/`

You can find the actual firmware images generated by the ImageBuilder (and the
ImageBuilder itself) in `firmwares`. The layout looks like the following:

```
firmwares/
TARGET/
backbone/
manual/
images..
default/
notunnel/
images..
...
OpenWrt-ImageBuilder-....tar.xz
Expand All @@ -94,73 +138,103 @@ firmwares/
*.ipk
```

As you notice there are several different image variants ("backbone", "default", etc.).
These different *packages lists* are defined in `packages/`.
See the "Features" section above for a description of the purpose of each package list.
With the "OpenWrt-Imagebuilder" you can assemble your own image variant with your
*packages lists* without having to compile everything yourself. The "OpenWrt-SDK" is
the fastest way to build your own packages or programs without compiling OpenWrt itself.
The "initrd" directory contains some initrd-images for netboot, which are required on
some boards to initially install OpenWrt.
As you notice there are several different image variants ("manual", "notunnel",
etc.).
+ These different *packages lists* are defined in `packages/`. See the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

packages/ --> packagelists/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You talk here on the "firmware" folder, right? Then this packages folder is just the storage-repo of all packages available.These packages will be included according to the flavors in packagelists/ to the image or used as source then running opkg install on the router.

"Features" section above for a description of the purpose of each package list.
+ With the "OpenWrt-Imagebuilder" you can assemble your own image variant with
your *packages lists* without having to compile everything yourself. The
*OpenWrt-SDK* is the fastest way to build your own packages or programs without
compiling OpenWrt itself.
+ The "initrd" directory contains some initrd-images for netboot, which are
required on some boards to initially install OpenWrt.


### customizing make

`make` will use by default `TARGET` and `PACKAGES_LIST_DEFAULT` defined in
`config.mk`. You can customize this by overriding them:

```
make TARGET=mpc85xx PACKAGES_LIST_DEFAULT=backbone
make TARGET=mpc85xx-generic PACKAGES_LIST_DEFAULT=manual
```
in addition you can build your own image from a prebuilt imagebuilder by something like:

```
make images IB_FILE=<file> TARGET=... PACKAGES_LIST_DEFAULT=...
```

The default target is `ar71xx-generic`. For a complete list of supported targets look in `configs/` for the target-specific configs.
Each of these targets need a matching file in `profiles/` with the profiles (boards) that should be build with the imagebuilder.
The default target is `ar71xx-generic`. For a complete list of supported targets
look in `configs/` for the target-specific configs. Each of these targets need a
matching file in `profiles/` with the profiles (boards) that should be build
with the imagebuilder.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"should be build by the imagebuilder"?

Copy link
Contributor

@pmelange pmelange Jul 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"should be built by" would be the queen's english.


additional options

* IS_BUILDBOT :
* this will be "yes" when running on the buildbot farm and helps to save some disc-space by removing files not required anymore. On manual builds you should not set this to "yes", as you have to rebuild the whole toolchain each time.
* this will be "yes" when running on the buildbot farm and helps to save some
disc-space by removing files not required anymore. On manual builds you
should not set this to "yes", as you have to rebuild the whole toolchain each
time.
* SET_BUILDBOT :
* "env" the Makefile will honor the "IS_BUILDBOT" environment
* "yes" the Makefile will always act as "IS_BUILDBOT" was set to "yes"
* "no" the Makefile will always act as "IS_BUILDBOT" was set to "no" / is unset. This way we can run builds on the buildbot like a local build.
* "no" the Makefile will always act as "IS_BUILDBOT" was set to "no" / is unset.
This way we can run builds on the buildbot like a local build.


### Build a new router model for an existing architecture

Everything that is in the [Table of Hardware](https://openwrt.org/toh/start)
should work at least in theory. Wifi should be ath9k / ath10k. mt76 also appears
to work.

You can use the Imagebuilder to get a list of all supported models for a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably the better place, than the mentioned above, to reference "make info" if we want to reference it at all.

platform. You can also look in the [source](https://git.openwrt.org/?p=openwrt/openwrt.git;a=tree;f=target/linux/ar71xx/image)
for model names.

Just put the name in the `/profiles/<target>.profiles` and it should work.
Akira25 marked this conversation as resolved.
Show resolved Hide resolved

Akira25 marked this conversation as resolved.
Show resolved Hide resolved

### Continuous integration / Buildbot

The firmware is [built
automatically](http://buildbot.berlin.freifunk.net/one_line_per_build) by our [buildbot farm](http://buildbot.berlin.freifunk.net/buildslaves). If you have a bit of CPU+RAM+storage capacity on one of your servers, you can provide a buildbot slave (see [berlin-buildbot](https://github.com/freifunk/berlin-buildbot)).
The firmware is [built automatically](http://buildbot.berlin.freifunk.net/one_line_per_build)
by our [buildbot farm](http://buildbot.berlin.freifunk.net/buildslaves). If you
have a bit of CPU+RAM+storage capacity on one of your servers, you can provide
a buildbot slave (see [berlin-buildbot](https://github.com/freifunk/berlin-buildbot)).

All branches whose name complies to the "X.Y.Z" pattern are built and put into
the "stable" downloads directory: [http://buildbot.berlin.freifunk.net/buildbot/stable/](http://buildbot.berlin.freifunk.net/buildbot/stable/)

All branches whose name complies to the "X.Y.Z" pattern are built and put into the "stable" downloads directory:
[http://buildbot.berlin.freifunk.net/buildbot/stable/](http://buildbot.berlin.freifunk.net/buildbot/stable/)
All branches with names not fitting the "X.Y.Z" pattern are built and put into
the "unstable" directory: [http://buildbot.berlin.freifunk.net/buildbot/unstable/](http://buildbot.berlin.freifunk.net/buildbot/unstable/)
Note that in the directory there is no reference to the branch name; unstable
builds can be identified by build number only.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

referencing the VERSION.txt file? As this is the best place to look up on which branch and revision a build is based on.


All branches with names not fitting the "X.Y.Z" pattern are built and put into the "unstable" directory:
[http://buildbot.berlin.freifunk.net/buildbot/unstable/](http://buildbot.berlin.freifunk.net/buildbot/unstable/)
Note that in the directory there is no reference to the branch name; unstable builds can be identified by build number only.

#### Creating a release

Every release has a [semantic version number](http://semver.org); each major version has its own codename.
We name our releases after important female computer scientists, hackers, etc.
For inspiration please take a look at the related
[ticket](https://github.com/freifunk-berlin/firmware/issues/24).
Every release has a [semantic version number](http://semver.org); each major
version has its own codename. We name our releases after important female
computer scientists, hackers, etc. For inspiration please take a look at the
related [ticket](https://github.com/freifunk-berlin/firmware/issues/24).

For a new release, create a new branch. The branch name must be a semantic version
number. Make sure you change the semantic version number and, for major releases,
the codename in the README and config files (./configs/*)
For a new release, create a new branch. The branch name must be a semantic
version number. Make sure you change the semantic version number and, for major
releases, the codename in the README and config files (`./configs/`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(./configs/) --> in all other places we did not prefix the path by "./"


The buildbot will build the release and place the files in the stable direcotry
once you pushed the new branch to github.


### Patches with "git format-patch"

**Important:** all patches should be pushed upstream!

If a patch is not yet included upstream, it can be placed in the corresponding subdirectory below the`patches`
directory. To create a correct patch-file just use the [`git format-patch`](https://git-scm.com/docs/git-format-patch) command.
If a patch is not yet included upstream, it can be placed in the corresponding
subdirectory below the`patches/` directory. To create a correct patch-file just
use the [`git format-patch`](https://git-scm.com/docs/git-format-patch) command.


#### Create a patch

Expand All @@ -181,21 +255,23 @@ or continue to the relevant feed directory:
cd feeds/luci
```

use the normal `git commit` workflow to apply your changes to the code. When done convert your last commit
into a patch by running:
use the normal `git commit` workflow to apply your changes to the code. When
done convert your last commit into a patch by running:

```bash
git format-patch --start-number <n> HEAD^
```
where `n` is the next free number of the correlating patch-subdirectory. You can use something like `HEAD^^^^`
to create patch-files from you last 4 commmits, or even use a git-rev directly. Feel free to squash multiple
commits into a single one before creating the patch-file or use something like
where `n` is the next free number of the correlating patch-subdirectory. You can
use something like `HEAD^^^^` to create patch-files from you last 4 commmits, or
even use a git-rev directly. Feel free to squash multiple commits into a single
one before creating the patch-file or use something like

```bash
git format-patch --stdout HEAD^^^^ > patches/routing/0008-awesome.patch
```
to create a single file of these 4 commits


#### Modify a patch

To update an existing patch do the same as above:
Expand All @@ -205,31 +281,35 @@ make clean patch
cd openwrt
cd feeds/luci
```
Then just add a new commit with your changes and squash it with the commit relating to the patch-file.
To update the patch-file use the same `git format-patch` sequence as you did when creating the patch
initially.
Then just add a new commit with your changes and squash it with the commit
relating to the patch-file. To update the patch-file use the same
`git format-patch` sequence as you did when creating the patch initially.


#### Delete a patch

To remove a patch-file you have to remove it from the patch-subdirectory and update the build-
environment:
To remove a patch-file you have to remove it from the patch-subdirectory and
update the build- environment:

```bash
git rm patches/openwrt/0010-unrelevant-change.patch
make patch
```


### Submitting patches

#### Freifunk Berlin

Please create a pull request for the project you want to submit a patch.
If you are already member of the Freifunk Berlin team, please delete branches once they have been merged.
Please create a pull request for the project you want to submit a patch. If you
are already member of the Freifunk Berlin team, please delete branches once they
have been merged.


#### OpenWrt

Create a commit in the openwrt directory that contains your change. Use `git
format-patch` to create a patch:
Create a commit in the openwrt directory that contains your change. Use
`git format-patch` to create a patch:

```
git format-patch origin
Expand Down