Feature: Build Dist Archives via CI and Upload them to OBS

.gitignore

@@ -60,3 +60,7 @@ config.h.in~
 
 # Ignore vim swap files
 .*.swp
+
+
+# CI: OSC temp stuff
+/osc_tmp

.gitlab-ci.yml

@@ -1,5 +1,5 @@
-# Build Icecast
-build:
+# Build Icecast - test
+make_dist:
   variables:
     GIT_SUBMODULE_STRATEGY: recursive
 
@@ -12,8 +12,11 @@ build:
 
   before_script:
   - apk update
+  - cat /etc/os*
   - apk add musl-dev git make gcc automake autoconf libtool
   - apk add curl-dev libogg-dev libvorbis-dev libxslt-dev libxml2-dev
+  # gzip required because busybox gzip does not understand best / zip for make dist
+  - apk add gzip zip tar
   # Required for tests
   #- apk add curl ffmpeg
   # Create user to run tests
@@ -22,7 +25,52 @@ build:
   script: 
     - ./autogen.sh
     - ./configure || cat config.log
-    - make
+    - make dist
+    - ls -la
     # Tests
     #- su -c "./tests/admin-tests.sh" icecast
 
+  artifacts:
+    paths:
+      - icecast-*.tar.gz
+      - icecast-*.zip
+    expire_in: 1 week
+
+upload_dist:
+  tags:
+    - docker
+    - linux
+
+  only:
+    - master
+    - devel
+
+  stage: deploy
+
+  dependencies:
+    - make_dist
+
+  before_script:
+    - ./ci/osc/prepare-osc-tools.sh
+
+  script:
+    - ./ci/osc/handle-osc-upload.sh
+
+upload_dist_release:
+  tags:
+    - docker
+    - linux
+
+  only:
+    - tags
+
+  stage: deploy
+
+  dependencies:
+    - make_dist
+
+  before_script:
+    - ./ci/osc/prepare-osc-tools.sh
+
+  script:
+    - ./ci/osc/handle-osc-upload.sh release

ci/README.md

@@ -0,0 +1,49 @@
+# Icecast CI for automated OBS Uploads
+
+This directory contains scripting triggered by .gitlab-ci.yml to automatically upload releases and nightlies to build.opensuse.org.
+The Pipeline triggers are set so that releases are pushed only on tags and nightlies on master/devel commits into respective OBS projects.
+These are defined in the ci/osc/ for releases and nightlies in their respective config files.
+
+# How to release
+
+To make a new release please call the magic version changer as following from the repo root:
+
+```
+export ICECAST_BETA_VERSION=3
+ci/create-changelog-and-set-versions.sh "$ICECAST_BETA_VERSION" "2.4.99.$ICECAST_BETA_VERSION" "2.5 beta$ICECAST_BETA_VERSION" "25-beta-$ICECAST_BETA_VERSION" "2.5-beta$ICECAST_BETA_VERSION" "_VERSION_ARCHIVE_" "2.4.99.$ICECAST_BETA_VERSION" "now" "Stephan Jauernick <[email protected]>" "Preparing for 2.5 beta$ICECAST_BETA_VERSION aka 2.4.99.$ICECAST_BETA_VERSION" "icecast" "mingw32-icecast" "mingw32-icecast-installer"
+```
+
+Please adapt the "ICECAST_BETA_VERSION", the Author, the Date(now; please enter a valid ISO8601 Date if needed) and the Message as needed.
+
+This script/mechanism will update all the version references and then show you a git status/diff. Please check all changes and commit them as needed.
+
+After tagging and uploading the release will be picked up by gitlabs CI and a release will be pushed to OBS.
+
+# Nightlies
+
+A nightly will be build on each change to master/devel - these will be marked with a git version + build date in the version and a "correct" changelog entry where relevant.
+
+# Required Repo Configuration in Gitlab
+
+- master and devel need to be set to protected with Maintainer only push/merge permissions
+- all tags need to be set to protected with Maintainer only permissions
+- There needs to be a "File" Variable configured as follows:
+
+Name: OSC_RC
+Protected only: yes
+Content:
+```
+[general]
+apiurl = https://api.opensuse.org
+
+[https://api.opensuse.org]
+user = OBS USER
+pass = OBS PASSWORD
+```
+
+The referenced user needs to have Maintainer access to the OBS projects referenced in the build configuration files.
+
+# How to improve
+
+Make version update script more robust... currently it will probably explode when we do a final 2.5 release.
+Rework OBS spec and debian files to modern standards. 

ci/create-changelog-and-set-versions.sh

@@ -0,0 +1,68 @@
+#!/bin/bash -xe
+
+BETA_VERSION=${1:?Missing Beta Version, Use 2}; shift
+SHORT_VERSION=${1:?Missing Short Version, Use 2.4.99.2}; shift
+STRANGE_VERSION=${1:?Missing Strange Version, Use '2.5 beta2'}; shift
+HTML_VERSION=${1:?Missing HTML Version, Use '25-beta-2'}; shift
+WIN32_VERSION=${1:?Missing Win32 Version, Use '2.5-beta2'}; shift
+ARCHIVE_VERSION=${1:?Missing Archive Version, Use '2.4.99.2' for ci or _VERSION_ARCHIVE_ for release}; shift
+CI_VERSION=${1:?Missing CI Version - Use 2.4.99.2+bla}; shift
+DATE=`date --date=${1:?ISO DATE or now} --iso-8601=seconds`; shift
+AUTHOR=${1:?Mail Address}; shift
+TEXT=${1:?Release Text}; shift
+ICECAST_PROJECT=${1:?Icecast OSC Project Name}; shift
+W32_ICECAST_PROJECT=${1:?Icecast W32 OSC Project Name}; shift
+W32_ICECAST_INSTALLER_PROJECT=${1:?Icecast W32 Installer OSC Project Name}; shift
+
+OSC_BASE_DIR=osc_tmp
+
+# upon release we modify the templates - in ci we modiy temporary files
+if [ "$ARCHIVE_VERSION" = "_VERSION_ARCHIVE_" ]; then
+  OSC_BASE_DIR=ci/osc
+fi
+
+pushd $OSC_BASE_DIR
+
+sed -i "1s#^#icecast2 ($CI_VERSION-1) UNRELEASED; urgency=medium\n\n  * $TEXT\n\n --  $AUTHOR  `date --date=$DATE +"%a, %d %b %Y %H:%M:%S %z"`\n\n#"  $ICECAST_PROJECT/debian/changelog
+
+for i in "$ICECAST_PROJECT/$ICECAST_PROJECT.spec" "$W32_ICECAST_INSTALLER_PROJECT/$W32_ICECAST_INSTALLER_PROJECT.spec" "$W32_ICECAST_PROJECT/$W32_ICECAST_PROJECT.spec"; do
+  sed -i "s/_VERSION_ARCHIVE_/$ARCHIVE_VERSION/; s/^Version:\(\s*\)[^\s]*$/Version:\1$CI_VERSION/; s#^%changelog.*\$#\0\n* `date --date=$DATE +"%a %b %d %Y"` $AUTHOR - $CI_VERSION-1\n- $TEXT\n\n#" "$i";
+done
+
+sed -i "s/\(icecast_win32_\).*\(.exe\)/\1$WIN32_VERSION\2/" $W32_ICECAST_INSTALLER_PROJECT/$W32_ICECAST_INSTALLER_PROJECT.spec
+
+popd
+
+# we only update the changelog for releases - until i figure out if we want to run the magic script pre CI
+if [ "$ARCHIVE_VERSION" = "_VERSION_ARCHIVE_" ]; then
+  sed -i "1s#^#`date --date=$DATE +"%Y-%m-%d %H:%M:%S"`  $AUTHOR\n\n        * $TEXT\n\n#" ChangeLog
+fi
+
+sed -i "1s#\[[.0-9]*\]#[$SHORT_VERSION]#" configure.ac
+
+sed -i "s/Icecast .* Documentation/Icecast $STRANGE_VERSION Documentation/; s/icecast-.*-documentation/icecast-$HTML_VERSION-documentation/" doc/index.html
+
+sed -i "s/\(\"DisplayVersion\" \"\).*\(\"\)$/\1$STRANGE_VERSION\2/" win32/icecast.nsis
+sed -i "s/\(OutFile \"icecast_win32_\).*\(.exe\"\)$/\1$WIN32_VERSION\2/" win32/icecast.nsis
+
+sed -i "s/^\(export ICECAST_VERSION=\).*$/\1$SHORT_VERSION/; s/\(export ICECAST_BETA_VERSION=\).*$/\1$BETA_VERSION/" ci/osc/*-config.sh
+
+# we only do this for release builds
+if [ "$ARCHIVE_VERSION" = "_VERSION_ARCHIVE_" ]; then
+  sed -i "s/^\(export RELEASE_AUTHOR=\"\).*\(\"\)$/\1$AUTHOR\2/; s/\(export RELEASE_DATETIME=\).*$/\1$DATE/" ci/osc/release-config.sh
+fi
+
+if [ "$ARCHIVE_VERSION" != "_VERSION_ARCHIVE_" ]; then
+  if ! git diff --quiet; then
+    echo "git detected differences after ci driven create changelog run, this should not happen - please check";
+    git status
+    git --no-pager diff
+    exit 1;
+  else
+    echo "no repo diffs detected, this is good as CI should not change the repo(only temp files)!"
+  fi
+else
+  echo "applied changes to versions, please verify and commit them for a new release"
+  git status
+  git --no-pager diff
+fi

ci/fix-dsc.sh

@@ -0,0 +1,21 @@
+#!/bin/bash -xe
+
+BASE=`dirname $0`
+TARGET=icecast.dsc
+TARBASE=icecast2_${ICECAST_CI_VERSION:?Missing ICECAST_CI_VERSION}
+sed "s/^Version: .*$/Version: $ICECAST_CI_VERSION-1/" $BASE/$TARGET.templ > $TARGET
+
+function helper {
+        HEADER=$1
+        FUNCTION=$2
+
+        echo "$HEADER" >> $TARGET
+        for FILE in $TARBASE.orig.tar.gz $TARBASE-1.debian.tar.gz; do
+                echo -e " `$FUNCTION $FILE | awk '{ print \$1 }'` `stat -c '%s' $FILE` $FILE" >> $TARGET
+        done
+}
+
+
+helper "Checksums-Sha1:" "sha1sum"
+helper "Checksums-Sha256:" "sha256sum"
+helper "Files:" "md5sum"

ci/icecast.dsc.templ

@@ -0,0 +1,12 @@
+Format: 3.0 (quilt)
+Source: icecast2
+Binary: icecast2
+Architecture: any
+Version: 2.4.99.2-1
+Maintainer: Thomas B. Ruecker <thomas@ruecker.fi>
+Uploaders: Thomas B. Ruecker <thomas@ruecker.fi>
+Homepage: http://www.icecast.org/
+Standards-Version: 3.9.5
+Build-Depends: cdbs, autotools-dev, devscripts, libtool, automake1.11, autoconf, debhelper, dh-buildinfo, libogg-dev (>> 1.0.0), libvorbis-dev (>> 1.0.0), libxslt1-dev | libxslt-dev, libxml2-dev, libcurl4-gnutls-dev, libtheora-dev (>= 0.0.0.alpha7), libspeex-dev, libssl-dev, po-debconf, pkg-config, dh-autoreconf, automake
+Package-List:
+ icecast2 deb sound optional arch=any

ci/osc/handle-osc-upload.sh

@@ -0,0 +1,85 @@
+#!/bin/bash -xe
+
+SCRIPT_DIR=`dirname $0`
+SCRIPT_DIR=`realpath $SCRIPT_DIR`
+CONFIG=${1:-nightly}
+
+. $SCRIPT_DIR/$CONFIG-config.sh
+
+: "${OSC_RC:?Variable OSC_RC not set or empty}"
+: "${CI_PIPELINE_URL:?Variable CI_PIPELINE_URL not set or empty}"
+
+pwd
+
+ls -la
+
+rm -rf osc_tmp
+mkdir -p osc_tmp
+cd osc_tmp
+
+export HOME=`pwd`
+export SOURCE=`pwd`/..
+export OSC_CMD="osc-wrapper.py --config=$OSC_RC"
+
+# checkout into a dkirectory named like the project - avoiding having it in a subdir called OBS_BASE
+for i in "$ICECAST_PROJECT" "$W32_ICECAST_INSTALLER_PROJECT" "$W32_ICECAST_PROJECT"; do
+  $OSC_CMD checkout -o "$i" $OBS_BASE "$i"
+  rm -vrf "$i"/*
+done
+
+# no comment needed
+for i in "$ICECAST_PROJECT" "$W32_ICECAST_PROJECT"; do
+  cp $SOURCE/icecast-$ICECAST_VERSION.tar.gz "$i"/icecast2_$ICECAST_CI_VERSION.orig.tar.gz 
+done
+
+# we copy the spec for these projects - for the icecast project the spec is globeed
+for i in "$W32_ICECAST_INSTALLER_PROJECT" "$W32_ICECAST_PROJECT"; do
+  cp -a $SCRIPT_DIR/$i/$i.spec $i/
+done
+
+# this is more complex because we have more files.
+cp -a $SCRIPT_DIR/$ICECAST_PROJECT/icecast* $ICECAST_PROJECT/
+cp -a $SCRIPT_DIR/$ICECAST_PROJECT/debian $ICECAST_PROJECT/
+
+if [ "$DISABLE_CHANGELOG" == "0" ]; then
+  pushd $SOURCE
+    $SCRIPT_DIR/../create-changelog-and-set-versions.sh "$ICECAST_BETA_VERSION" "$ICECAST_VERSION" "2.5 beta$ICECAST_BETA_VERSION" "25-beta-$ICECAST_BETA_VERSION" "2.5-beta$ICECAST_BETA_VERSION" "$ICECAST_VERSION" "$ICECAST_CI_VERSION" "$RELEASE_DATETIME" "$RELEASE_AUTHOR" "CI Build - $CI_PIPELINE_URL" "$ICECAST_PROJECT" "$W32_ICECAST_INSTALLER_PROJECT" "$W32_ICECAST_PROJECT"
+  popd
+else
+  for i in "$ICECAST_PROJECT/$ICECAST_PROJECT.spec" "$W32_ICECAST_INSTALLER_PROJECT/$W32_ICECAST_INSTALLER_PROJECT.spec" "$W32_ICECAST_PROJECT/$W32_ICECAST_PROJECT.spec"; do
+    sed -i "s/_VERSION_ARCHIVE_/$ICECAST_VERSION/;" "$i";
+  done
+fi
+
+tar -C $ICECAST_PROJECT -cvzf $ICECAST_PROJECT/icecast2_$ICECAST_CI_VERSION-1.debian.tar.gz debian/
+
+# remove debian/ so it does not end up in the archive
+rm -rf $ICECAST_PROJECT/debian
+
+# we fix the dsc to have the correct hashsums so dpkg-buildpackage and co do not complain
+
+pushd $ICECAST_PROJECT
+
+$SCRIPT_DIR/../fix-dsc.sh
+
+popd
+
+# we use addremove to detect changes and commit them to the server
+for i in "$ICECAST_PROJECT" "$W32_ICECAST_INSTALLER_PROJECT" "$W32_ICECAST_PROJECT"; do
+  pushd $i
+
+  $OSC_CMD addremove
+  $OSC_CMD diff
+  $OSC_CMD commit -m "Commit via $CI_PIPELINE_URL"
+
+  popd
+done
+
+# we cleanup because the OSC_RC should not remain on disk
+if [ "$NOCLEANUP" != "1" ]; then
+  shred -vzf $OSC_RC .osc_cookiejar
+  cd ..
+  echo > $OSC_RC
+  rm -rf "osc_tmp"
+fi
+

ci/osc/icecast/debian/README.Debian

@@ -0,0 +1,30 @@
+icecast2 for Debian
+-------------------
+
+In the Debian packaging the configuration files have been symlinked from
+the upstream location below /usr/share to /etc. This is needed to
+satisfy FHS (/usr/share are for static content only).
+If running icecast2 in a chroot environment, beware that the symlinks to
+/etc will break. A possible (untestet!) solution might be to manually
+put the configuration files back below /usr/share - and revert the hack
+again before updating the package!
+
+ -- Jonas Smedegaard <[email protected]>  Thu, 20 May 2004 21:04:27 +0200
+
+
+It is recommended to run icecast under a dedicated user account, which only
+has access to write the log files.  The Debian package creates such an
+account, named 'icecast2', and uses it by default, but you are free to
+reconfigure it and remove the account.
+
+Edit /etc/default/icecast2 to change the init-script configuration.
+
+It is possible (but discouraged for security reasons) to bind to a
+privileged port (like standard web port 80). Edit /etc/init.d/icecast2
+to not change userid and instead set the correct userid and group in
+/etc/icecast2/icecast2.xml. Beware that this way you rely on the
+icecast2 binary to properly drop privileges (instead of the much more
+thoroughly audited start-stop-daemon). Thanks to Jürgen A. Erhard
+<[email protected]> for the tip.
+
+ -- Keegan Quinn <[email protected]>