Apache Sling Aggregator

This module is part of the Apache Sling project.

It provides an XML file that lists all Sling modules, to allow for tools like repo to process multiple repositories at once, along with some utility scripts to help manage our large number of repositories.

To update that XML file to include new Sling modules, run the collect-sling-repos.groovy (with -a, see comments in the script), check the diffs of the default.xml file and push the result if ok.

The list of modules is in a self-explaining format and can also be used in your own scripts if preferred.

The list of repositories on our website is also generated from this data using a Groovy page template.

Note that there are related efforts at SLING-7331 and SLING-7262, we'll need to consolidate all this at some point.

Modules

You can find a list of the Apache Sling modules here. This list is generated from the script generate-aggregator-table.groovy.

Updating Module Badges

We have a simple script to update the badges in GitHub's README.md files. To update a single repository:

./generate-project-badges.groovy [REPO_DIR]

To update all repositories:

repo forall -c '[SLING_DIR]/aggregator/generate-project-badges.groovy .'

Updating the .asf.yaml files

We maintain a set of .asf.yaml files, to set reasonable defaults for all Sling GitHub repositories. To make maintenance simple, we use the scripts/update-asf-yaml.groovy.

The script accepts a list of directories to generate or update the listing file for.

$ repo list --path-only | xargs groovy aggregator/scripts/update-asf-yaml.groovy
$ repo forall -c 'git add .asf.yaml'
$ repo forall -c 'git commit -m "SLING-12345 Updating .asf.yaml" -m "Detailed description"'

Once you are happy with what the commits look like you can push the changes, with a delay. It is recommended to use the provided repo-push-with-delay.sh script from the root of the repo checkout.

$ aggregator/scripts/repo-push-with-delay.sh

Updating the Aggregator List

To update the aggregator list:

groovy generate-aggregator-table.groovy [SLING_DIR]

Prerequisites

1. Use the repo tool to extract all of the repositories in the sling aggregator
2. Ensure you have SSH based access enabled to GitHub
3. Ensure all repository workspaces are in a clean state

Retrieving all Sling modules

This module allows quick checkout of all Sling modules from Git. It requires the local installation of the repo tool.

Repo Tool Installation (all platforms)

$ curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
$ chmod a+x ~/bin/repo

See also the detailed instructions at https://source.android.com/docs/setup/download/source-control-tools#repo.

Repo Tool Installation on Mac with Homebrew

brew install repo

Synchronizing all Git repositories

Initialise the local repo checkout and synchronise all git repositories. The commands below must be run in the sling-aggreator git checkout.

$ repo init --no-clone-bundle -u https://github.com/apache/sling-aggregator.git
$ repo sync --no-clone-bundle -j 16
$ repo forall -c 'git checkout master'

The last command ensures that all repositories are set to use the master branch. For some reason repo checks out the latest revision but does not check out a specific branch.

The output is a flat list of all Sling modules.

The -j 16 flag instructs repo to run 16 parallel checkout jobs and is added for performance reasons only.

Eclipse EGit Support

When trying to add such a repository with EGit in the local working copy is not detected due to the way that repo works with symbolic links. This has been reported in Eclipse Bugzilla Issue #579042.

Be careful with "mass pushes" to repositories

ASF Infra notes that pushing to many Git repositories quickly can cause heavy load on their github push queue.

Please use the provided script to push changes that affect multiple repositories:

$ aggregator/scripts/repo-push-with-delay.sh

Updating the list of modules

That list is found in the default.xml file.

It is used to generate the list of Git Repositories on our website.

Install Groovy with with ASDF

asdf install

Install Groovy on Mac with Homebrew

brew install groovy

To update it:

groovy collect-sling-repos.groovy > default.xml

Check changes with git diff and commit if needed.