-
Notifications
You must be signed in to change notification settings - Fork 25
Building a Microkernel ISO
The Microkernel used by Razor is actually a variant of the Tiny Core Linux (TCL) kernel. The Microkernel is built using the 'Core' release of TCL, which can be found here. In order to run the tools that are needed by the Microkernel (Facter for discovery and MCollective for management), a number of TCL Extensions need to be added to our Microkernel. It is these extensions that differentiate our Microkernel instance from the basic TCL Core release. Previously, we relied on a procedure that involved booting a VM using the 'standard' TCL Core ISO, installing (and configuring) the extensions that we needed from the command line, and then creating an overlay (basically a snapshot, in the form of a gzipped tarfile) of key parts of the filesystem from that running TCL instance that we could use to 'automatically' build a Microkernel instance from that same TCL ISO using a couple of scripts (and a few other dependencies that were extracted 'by hand' from the Razor-Microkernel project).
This manual process was fairly tedious and somewhat error prone. Not only that, but the process was difficult (if not impossible) to automate. As such, we decided to take a different approach to building the Razor Microkernel ISO. This new approach uses a bash script (which is part of the Razor-Microcore project itself) to create a single gzipped tarfile (which we'll refer to as the 'bundle file' in this document) that contains an 'overlay file' (another gzipped tarfile), as well as the scripts and dependencies are needed to merge that overlay file with the 'standard' TCL Core ISO (which is also included in the bundle file) in order to build a new version of the Razor Microkernel ISO. The components for the overlay file (and the dependencies that it needs) are all either copied over from the Razor-Microkernel project or downloaded from online sources to local directories (if they are not part of the Razor-Microkernel project) before being included in the bundle file. Below, we will outline the procedure that followed by this script, and in a follow-on page we will describe how the gzipped tarfile (the 'bundle') that is built by this new bash script can be used to easily build your own Razor Microkernel ISO.
The script that is used for this process (the 'build-bundle-file.sh' script) can be found in the top-level directory of the Razor-Microkernel project itself. The usage for this script is readily available by adding a '-h' (or '--help') flag to the command line when invoking the script:
test@server:~/Razor-Microkernel$ ./build-bundle-file.sh -h
Usage: ./build-bundle-file.sh OPTIONS
This script builds a gzipped tarfile containing all of the files necessary to
build an instance of the Razor Microkernel ISO.
OPTIONS:
-h, --help print usage for this command
-r, --reuse-prev-dl reuse the downloads rather than downloading again
-b, --builtin-list FILE file containing extensions to install as builtin
-m, --mirror-list FILE file containing extensions to add to TCE mirror
-p, --build-prod-image build a production ISO (no openssh, no passwd)
-d, --build-debug-image build a debug ISO (enable automatic console login)
-t, --tc-passwd specify a password for the tc user
Note; currently, the default is to build a development ISO (which includes the
openssh.tcz extension along with the openssh/openssl configuration file changes
and the passwd changes needed to access the Microkernel image from the command
line or via the console). Also note that only one of the '-p' and '-d' flags
may be specified and the '-t' option may not be used when building a production
ISO (using the '-p' flag).
test@server:~/Razor-Microkernel$
When invoking this script, the '-b' and '-m' flags (or their more verbose equivalents) MUST be provided (if they are not, then an error will be thrown and the command usage will be printed), and the files that are included as arguments to these two flags must be readable. These files should contain a list of builtin extensions and local TCE mirror extensions (respectively) that should be downloaded and included in the overlay file we are creating. The builtin extensions are those that should be installed as part of the boot process for the Microkernel (making them available when tasks like network initialization take place, for example), while the TCE mirror extensions are simply placed into a local TCE mirror in the Microkernel's filesystem (where they can be used as part of the Microkernel Controller initialization process, post-boot). Examples for these two files are included in the 'additional-build-files' subdirectory in the Razor Microkernel project (the files are called 'builtin-extensions.lst' and 'mirror-extensions.lst', respectively). The other arguments to the script are optional, and default values are used if they are not specified (by default, the previous downloads are not re-used and the system builds a development Microkernel, not a production one). A simple example of the usage for this shell script that uses the default lists of extensions provided as a part of the Razor-Microkernel project is as follows:
./build-bundle-file.sh -r -b additional-build-files/builtin-extensions.lst \
-m additional-build-files/mirror-extensions.lst -d -t test1234 2>&1 | tee t.t
The 'build-bundle-file.sh' script must be run from the top-level directory of the Razor-Microkernel project. When it is invoked, this script will first create a directory structure that looks something like the following:
tmp-build-dir/
└── build_dir
Here, 'tmp-build-dir' is the working directory created by the script and the 'tmp-build-dir/build_dir' subdirectory is the directory we will be using to create the final bundle. This 'bundle' file will contain the overlay file itself (in the 'razor-microkernel-files.tar.gz' file that is placed into the 'tmp-build-dir/build_dir/dependencies' directory), along with all of the files and scripts that are needed in order to turn that to turn that overlay file into a working Razor Microkernel instance.
It should be noted that the sub-directories of the 'tmp-build-dir' directory (with the exception of the 'build_dir' subdirectory) will eventually be mapped into the root-level directories in the overlay file that we are building here (eg. the 'tmp-build-dir/usr/local/bin' directory in the local filesystem will correspond to the 'usr/local/bin' directory in our overlay). We should also note that the final directory structure that results from this build process will be left in place at the completion of the bundle build process (so that the downloads that the directory structure contains can be re-used, if necessary, to build another bundle (with additional extensions?).
After creating the initial 'build_dir' subdirectory (and the 'build_dir/dependencies' subdirectory), the 'build-bundle-file.sh' script copies over the scripts that will be needed to build the Razor Microkernel ISO directly from the 'iso-build-files' subdirectory of the Razor-Microkernel project. These scripts are placed directly into the 'tmp-build-dir/build_dir' subdirectory. If we are building a production ISO, the copy of the 'rebuild_iso.sh' script that is made via this process will also be modified (on the fly using 'sed') so that it outputs an ISO file with the string 'prod' in its name (rather than the string 'dev') when it is invoked. The next step in the process is to copy over the modified DHCP client scripts from the Razor-Microkernel project to the corresponding locations in the 'tmp-build-dir'. After that, the script copies over the Razor Microkernel Controller script (and its dependencies) into the appropriate locations in the 'tmp-build-dir/usr/local/bin' and 'tmp-build-dir/usr/local/lib/razor_microcontroller' subdirectories. Finally, the script copies over the list of gems from the 'opt/gems/gem.list' file of the Razor-Microkernel project into the 'tmp-build-dir/opt/gems' subdirectory. With that step, the process of copying files from the Razor Microkernel project into the directories that we will be creating our overlay from is complete and we are ready to start downloading some dependencies.
The download process starts by using the list of gems that was copied over from the Razor-Microkernel project (above) to determine what gems should be downloaded (so that they will be included in the overlay file). The gems in that list are downloaded to the 'tmp-build-dir/opt/gems' directory using the 'gem fetch...' command (Note; this assumes that the local system has RubyGems already installed on it and that the 'gem' command is accessible by the user running this shell script). With the gems in place, the script then downloads two sets of extensions from the TCL Extension Repository (or TCE repository). The first set of extensions downloaded are used to construct a local TCE mirror, and the second set are placed into a subdirectory from which they will automatically be installed during the boot process. The actual extensions that are installed are read from flat files containing the list of extensions to include in the local mirror and a list of extensions to set up as 'built-in extensions' (which appear in the 'additional-build-files/mirror-extensions.lst' and 'additional-build-files/builtin-extensions.lst' files in the Razor-Microkernel project, respectively). Files from the first list will be placed into a mirror subdirectory under the 'tmp-build-dir/tmp/tinycorelinux' subdirectory, while those from the second will be placed into the 'tmp-build-dir/tmp/builtin' subdirectory. In the second case, the list of extensions that is loaded is also used to construct a 'tmp-build-dir/tmp/builtin/onboot.lst' file that will be used by the Microkernel to determine which of the 'built-in' extensions should be loaded on boot.
After the 'bundle initialization' process is complete, the directory structure will actually look a bit more complicated (currently, this is what it looks like after initializing the temporary build directory but before building the bundle):
build-files
└── razor-microkernel-bundle-dev.tar.gz
tmp-build-dir
├── build_dir
│ ├── add_version_to_mk_fs.rb
│ ├── build_initial_directories.sh
│ ├── build_iso_yaml.rb
│ ├── Core-current.iso
│ ├── dependencies
│ │ ├── mcollective-setup-files.tar.gz
│ │ ├── mk-open-vm-tools.tar.gz
│ │ ├── razor-microkernel-overlay.tar.gz
│ │ └── ssh-setup-files.tar.gz
│ └── rebuild_iso.sh
├── etc
│ ├── init.d
│ │ └── dhcp.sh
│ ├── inittab
│ ├── passwd
│ └── shadow
├── mcollective-2.0.0.tgz
├── opt
│ ├── bootsync.sh
│ ├── gems
│ │ └── ...
│ └── rubygems-1.8.24.tgz
├── tmp
│ ├── builtin
│ │ ├── onboot.lst
│ │ └── optional
│ │ └── ...
│ ├── first_checkin.yaml
│ ├── mk_conf.yaml
│ └── tinycorelinux
│ ├── 4.x
│ │ └── x86
│ │ └── tcz
│ │ └── ...
│ └── ...
├── usr
│ ├── local
│ │ ├── bin
│ │ │ ├── lscpu
│ │ │ └── ...
│ │ ├── lib
│ │ │ └── ruby
│ │ │ └── 1.8
│ │ │ └── razor_microkernel
│ │ │ └── ...
│ │ ├── mcollective -> /usr/local/tce.installed/mcollective-2.0.0
│ │ ├── sbin
│ │ │ └── sfdisk
│ │ └── tce.installed
│ │ └── mcollective-2.0.0
│ │ └── ...
│ ├── sbin
│ │ └── dmidecode -> /usr/local/sbin/dmidecode
│ └── share
│ └── udhcpc
│ └── dhcp_mk_config.script
└── util-linux.tcz
With this directory structure in place, building the actual bundle is a very simple process (consisting of one command in the 'build-bundle-file.sh' script). When this process is complete, a single bundle file will be added to the 'build-files' subdirectory of the Razor-Microkernel project (named 'razor-microkernel-bundle-dev.tar.gz' for a development bundle and 'razor-microkernel-bundle-prod.tar.gz' for a production bundle). That bundle file will contain all of the scripts and files that are needed to build a new Microkernel ISO from the current Razor-Microkernel project (and the latest versions of its dependencies). This is what that current contents of a development bundle look like:
-rw-rw-r-- tjmcs/tjmcs 8290304 2012-06-04 22:24 Core-current.iso
-rwxrwxr-x tjmcs/tjmcs 1385 2012-05-02 13:55 add_version_to_mk_fs.rb
-rwxrwxr-x tjmcs/tjmcs 1263 2012-06-19 16:50 build_initial_directories.sh
-rwxrwxr-x tjmcs/tjmcs 2501 2012-05-02 13:25 build_iso_yaml.rb
drwxrwxr-x tjmcs/tjmcs 0 2012-06-21 16:27 dependencies/
-rw-rw-r-- tjmcs/tjmcs 30176283 2012-06-21 16:27 dependencies/razor-microkernel-overlay.tar.gz
-rw-rw-r-- tjmcs/tjmcs 813074 2012-06-08 14:19 dependencies/mk-open-vm-tools.tar.gz
-rw-r--r-- tjmcs/tjmcs 5723 2012-06-07 19:45 dependencies/ssh-setup-files.tar.gz
-rw-r--r-- tjmcs/tjmcs 374 2012-06-06 15:30 dependencies/mcollective-setup-files.tar.gz
-rwxr-xr-x tjmcs/tjmcs 1414 2012-05-02 13:46 rebuild_iso.sh
To use this bundle file, simply copy it over to another directory on this (or another) machine and unpack it
# tar zxvf razor-microkernel-bundle-dev.tar.gz
From that same directory, the sequence of commands to build a new ISO is actually quite simple, first, build the directory structure needed to for the script that constructs the new ISO:
# ./build_initial_directories.sh
and then invoke the script that actually builds the ISO
# ./rebuild_iso.sh 0.9.0.1
Note; the single argument to this second script will create an Razor Microkernel ISO with a version number of 0.9.0.1. You can replace this argument with any version number you would like to use (although we tend to use 4 digit version numbers like the one shown above in current releases of the Razor Microkernel).
Once the second shell script command finishes, you should have a new Razor Microkernel ISO build in the current working directory; an ISO that is suitable for use with Razor and that contains all of the dependencies and extensions included in the bundle build process (above). If, for some reason, you find an issue with the ISO you have built, rebuilding a new ISO is as simple as running the same three shell scripts that are shown, above; perhaps with a new set of built-in and/or additional extensions declared during the initial bundle build process.
There are a few dependencies that must be met for this build process to run successfully. The bundle file build process (described above) relies on the 'unsquashfs' command to extract the 'lscpu' and 'sfdisk' commands from the 'util-linux.tcz' extension that it downloads from the standard Tiny Core Linux Extension Mirror. That same script also makes extensive use of the 'wget' command to download the dependencies that it needs and uses the 'gem' command to 'fetch' the gems that are needed from the 'Ruby Gems' gem repository. Finally, the dynamic downloading of the dependencies and extensions that need to be included in the overlay file that is being constructed require that an active network connection exists (with access to the internet) and that the local Razor-Microkernel project directory be writeable by the current user. If any of these dependencies are not met, the bundle file build process will fail.
The script that builds the Microkernel ISO also has a number of dependencies on system level commands that may or may not exist on all platforms (and that may or may not be accessible for all users on any given platform). It uses the 'chroot' and 'ldconfig' commands to set up the contents of the 'tmp' subdirectory that it uses when building the 'core.gz' file that will be placed into the Microkernel ISO. If these commands are not accessible (or are not usable by the current user), then the resulting ISO may or may not be usable as a boot image by Razor. This script also uses the 'cpio' command to construct the 'core.gz' file and uses the 'advdef' command to further compress that file before placing it into the Microkernel ISO. We have seen issues on some platforms with the flags for these commands not being consistent (or with these commands not being available at all). The build process has been successfully tested and used under recent releases of Ubuntu (the 64-bit server release, but that's probably not important), but failed for some users who were trying to use the same scripts under OS X. The Ubuntu Server instance we have used most extensively uses the GNU cpio command and advancecomp v1.15 (for the 'advdef' command). This build process hasn't much testing on other platforms to date, so your mileage may vary in terms when it comes to using this build process (depending on the platform that you use).
By following the procedure outlined, above, we were able to create 'bundle file' containing an 'overlay file' that can be used to add a number of extensions to the default TCL Core distribution. We were then able to use that 'bundle file' (the 'overlay file', additional dependencies, and scripts that it contains) to build a new Microkernel ISO from the standard TCL Core distribution ISO.
Below we provide a listing of the current version of the 'build-bundle-file.sh' script and a nearly complete 'post-bundle-build' directory structure for reference (we say nearly complete because we're only showing the 'tcz' files for the extensions, not the associated md5, list, info, and dep files, and we've cut out all of the files and subdirectories under the '/usr/local/tce.installed/mcollective-2.0.0' directory, but the rest of the structure is intact).
First, the shell script listing:
#!/bin/sh
#
# Used to build the bundle file needed to build a new version of the
# Razor Microkernel ISO (from the contents of the Razor Microkernel
# project and it's dependencies. The file built by this script can
# be copied over to another directory (on another machine?) and unpacked.
# Once it has been unpacked, running the 'build_initial_directories.sh'
# script in that directory, followed by the 'rebuild_iso.sh' script,
# will result in a new ISO built from the current state of the this
# (Razor-Microkernel) project.
#
# Note: the bundle file does not creaate a subdirectory, so a new, clean
# directory should be used when unpacking the bundle file to build a
# new version of the Microkernel ISO.
# define a function we can use to print out the usage for this script
usage()
{
cat << EOF
Usage: $0 OPTIONS
This script builds a gzipped tarfile containing all of the files necessary to
build an instance of the Razor Microkernel ISO.
OPTIONS:
-h, --help print usage for this command
-r, --reuse-prev-dl reuse the downloads rather than downloading again
-b, --builtin-list FILE file containing extensions to install as builtin
-m, --mirror-list FILE file containing extensions to add to TCE mirror
-p, --build-prod-image build a production ISO (no openssh, no passwd)
-d, --build-debug-image build a debug ISO (enable automatic console login)
-t, --tc-passwd specify a password for the tc user
Note; currently, the default is to build a development ISO (which includes the
openssh.tcz extension along with the openssh/openssl configuration file changes
and the passwd changes needed to access the Microkernel image from the command
line or via the console). Also note that only one of the '-p' and '-d' flags
may be specified and the '-t' option may not be used when building a production
ISO (using the '-p' flag).
EOF
}
# initialize a few variables to hold the options passed in by the user
BUILTIN_LIST=
MIRROR_LIST=
TC_PASSWD=
RE_USE_PREV_DL='no'
BUILD_PROD_ISO='no'
BUILD_DEBUG_ISO='no'
# options may be followed by one colon to indicate they have a required argument
if ! options=$(getopt -o hrb:m:pdt: -l help,reuse-prev-dl,builtin-list:,mirror-list:,build-prod-image,build-debug-image,tc-passwd: -- "$@")
then
usage
# something went wrong, getopt will put out an error message for us
exit 1
fi
set -- $options
while [ $# -gt 0 ]
do
case $1 in
-r|--reuse-prev-dl) RE_USE_PREV_DL='yes';;
-b|--builtin-list) BUILTIN_LIST=`echo $2 | tr -d "'"`; shift;;
-m|--mirror-list) MIRROR_LIST=`echo $2 | tr -d "'"`; shift;;
-p|--build-prod-image) BUILD_PROD_ISO='yes';;
-d|--build-debug-image) BUILD_DEBUG_ISO='yes';;
-t|--tc-passwd) TC_PASSWD=`echo $2 | tr -d "'"`; shift;;
-h|--help) usage; exit 0;;
(--) shift; break;;
(-*) echo "$0: error - unrecognized option $1" 1>&2; usage; exit 1;;
esac
shift
done
if [ -z $BUILTIN_LIST ] || [ -z $MIRROR_LIST ]; then
echo "\nError (Missing Argument); the 'builtin-list' and 'mirror-list' must both be specified"
usage
exit 1
elif [ ! -r $BUILTIN_LIST ] || [ ! -r $MIRROR_LIST ]; then
echo "\nError; the 'builtin-list' and 'mirror-list' values must both be readable files"
usage
exit 1
elif [ $BUILD_DEBUG_ISO = 'yes' ] && [ $BUILD_PROD_ISO = 'yes' ]; then
echo "\nError; Only one of the '-d' and '-p' options should be specified"
echo " (ISO cannot be both a debug and production ISO)"
usage
exit 1
elif [ ! -z $TC_PASSWD ] && [ $BUILD_PROD_ISO = 'yes' ]; then
echo "\nError; Only one of the '-t' and '-p' options should be specified"
echo " (Cannot specify a 'tc' password to use for a production ISO)"
usage
exit 1
fi
# if not, then make sure we're starting with a clean (i.e. empty) build directory
if [ $RE_USE_PREV_DL = 'no' ]
then
if [ ! -d tmp-build-dir ]; then
# make a directory we can use to build our gzipped tarfile
mkdir tmp-build-dir
else
# directory exists, so remove the contents
rm -rf tmp-build-dir/*
fi
fi
# initialize a couple of variables that we'll use later
TOP_DIR=`pwd`
TCL_MIRROR_URI='http://distro.ibiblio.org/tinycorelinux/4.x/x86/tcz'
TCL_ISO_URL='http://distro.ibiblio.org/tinycorelinux/4.x/x86/release/Core-current.iso'
RUBY_GEMS_URL='http://production.cf.rubygems.org/rubygems/rubygems-1.8.24.tgz'
#MCOLLECTIVE_URL='http://puppetlabs.com/downloads/mcollective/mcollective-1.2.1.tgz'
MCOLLECTIVE_URL='http://puppetlabs.com/downloads/mcollective/mcollective-2.0.0.tgz'
#OPEN_VM_TOOLS_URL='https://github.com/downloads/lynxbat/Razor-Microkernel/mk-open-vm-tools.tar.gz'
OPEN_VM_TOOLS_URL='https://github.com/downloads/puppetlabs/Razor/mk-open-vm-tools.tar.gz'
# create a folder to hold the gzipped tarfile that will contain all of
# dependencies
mkdir -p tmp-build-dir/build_dir/dependencies
# copy over the scripts that are needed to actually build the ISO into
# the build_dir (from there, they will be included into a single
# gzipped tarfile that can be unpacked and will contain almost all of
# the files/tools needed to build the Microkernel ISO)
cp -p iso-build-files/* tmp-build-dir/build_dir
if [ $BUILD_PROD_ISO = 'yes' ]; then
sed -i 's/ISO_NAME=rz_mk_dev-image/ISO_NAME=rz_mk_prod-image/' tmp-build-dir/build_dir/rebuild_iso.sh
elif [ $BUILD_DEBUG_ISO = 'yes' ]; then
sed -i 's/ISO_NAME=rz_mk_dev-image/ISO_NAME=rz_mk_debug-image/' tmp-build-dir/build_dir/rebuild_iso.sh
fi
# create a copy of the modifications to the DHCP client configuration that
# are needed for the Razor Microkernel Controller to find the appropriate
# Razor server for it's first checkin
mkdir -p tmp-build-dir/etc/init.d
cp -p etc/init.d/dhcp.sh tmp-build-dir/etc/init.d
mkdir -p tmp-build-dir/usr/share/udhcpc
cp -p usr/share/udhcpc/dhcp_mk_config.script tmp-build-dir/usr/share/udhcpc
# create copies of the files from this project that will be placed
# into the /usr/local/bin directory in the Razor Microkernel ISO
mkdir -p tmp-build-dir/usr/local/bin
cp -p rz_mk_*.rb tmp-build-dir/usr/local/bin
# create copies of the files from this project that will be placed
# into the /usr/local/lib/ruby/1.8/razor_microkernel directory in the Razor
# Microkernel ISO
mkdir -p tmp-build-dir/usr/local/lib/ruby/1.8/razor_microkernel
cp -p razor_microkernel/*.rb tmp-build-dir/usr/local/lib/ruby/1.8/razor_microkernel
# create copies of the MCollective agents from this project (will be placed
# into the /usr/local/tce.installed/$mcoll_dir/plugins/mcollective/agent
# directory in the Razor Microkernel ISO
file=`echo $MCOLLECTIVE_URL | awk -F/ '{print $NF}'`
mcoll_dir=`echo $file | cut -d'.' -f-3`
mkdir -p tmp-build-dir/usr/local/tce.installed/$mcoll_dir/plugins/mcollective/agent
cp -p configuration-agent/configuration.rb facter-agent/facteragent.rb \
tmp-build-dir/usr/local/tce.installed/$mcoll_dir/plugins/mcollective/agent
# create a copy of the files from this project that will be placed into the
# /opt directory in the Razor Microkernel ISO; as part of this process will
# download the latest version of the gems in the 'gem.list' file into the
# appropriate directory to use in the build process (rather than including
# fixed versions of those gems as part of the Razor-Microkernel project)
mkdir -p tmp-build-dir/opt/gems
cp -p opt/bootsync.sh tmp-build-dir/opt
cp -p opt/gems/gem.list tmp-build-dir/opt/gems
cd tmp-build-dir/opt/gems
for file in `cat gem.list`; do
if [ $RE_USE_PREV_DL = 'no' ] || [ ! -f $file*.gem ]
then
gem fetch $file
fi
done
cd $TOP_DIR
# create a copy of the local TCL Extension mirror that we will be running within
# our Microkernel instances
mkdir -p tmp-build-dir/tmp/tinycorelinux/4.x/x86/tcz
cp -p tmp/tinycorelinux/*.yaml tmp-build-dir/tmp/tinycorelinux
for file in `cat $MIRROR_LIST`; do
if [ $RE_USE_PREV_DL = 'no' ] || [ ! -f tmp-build-dir/tmp/tinycorelinux/4.x/x86/tcz/$file ]
then
wget -P tmp-build-dir/tmp/tinycorelinux/4.x/x86/tcz $TCL_MIRROR_URI/$file
wget -P tmp-build-dir/tmp/tinycorelinux/4.x/x86/tcz -q $TCL_MIRROR_URI/$file.md5.txt
wget -P tmp-build-dir/tmp/tinycorelinux/4.x/x86/tcz -q $TCL_MIRROR_URI/$file.info
wget -P tmp-build-dir/tmp/tinycorelinux/4.x/x86/tcz -q $TCL_MIRROR_URI/$file.list
wget -P tmp-build-dir/tmp/tinycorelinux/4.x/x86/tcz -q $TCL_MIRROR_URI/$file.dep
fi
done
# download a set of extensions that will be installed during the Microkernel
# boot process. These files will be placed into the /tmp/builtin directory in
# the Microkernel ISO. The list of files downloaded (and loaded at boot) are
# assumed to be contained in the file specified by the BUILTIN_LIST parameter
echo `pwd`
mkdir -p tmp-build-dir/tmp/builtin/optional
rm tmp-build-dir/tmp/builtin/onboot.lst 2> /dev/null
for file in `cat $BUILTIN_LIST`; do
if [ $BUILD_PROD_ISO = 'no' ] || [ ! $file = 'openssh.tcz' ]; then
if [ $RE_USE_PREV_DL = 'no' ] || [ ! -f tmp-build-dir/tmp/builtin/optional/$file ]
then
wget -P tmp-build-dir/tmp/builtin/optional $TCL_MIRROR_URI/$file
wget -P tmp-build-dir/tmp/builtin/optional -q $TCL_MIRROR_URI/$file.md5.txt
wget -P tmp-build-dir/tmp/builtin/optional -q $TCL_MIRROR_URI/$file.dep
fi
echo $file >> tmp-build-dir/tmp/builtin/onboot.lst
elif [ $BUILD_PROD_ISO = 'yes' ] && [ -f tmp-build-dir/tmp/builtin/optional/$file ]
then
rm tmp-build-dir/tmp/builtin/optional/$file
rm tmp-build-dir/tmp/builtin/optional/$file.md5.txt 2> /dev/null
rm tmp-build-dir/tmp/builtin/optional/$file.dep 2> /dev/null
fi
done
# download the ruby-gems distribution (will be installed during the boot
# process prior to starting the Microkernel initialization process)
file=`echo $RUBY_GEMS_URL | awk -F/ '{print $NF}'`
if [ $RE_USE_PREV_DL = 'no' ] || [ ! -f tmp-build-dir/opt/$file ]
then
wget -P tmp-build-dir/opt $RUBY_GEMS_URL
fi
# copy over a couple of initial configuration files that will be included in the
# /tmp and /etc directories of the Microkernel instance (the first two control the
# initial behavior of the Razor Microkernel Controller, the third disables automatic
# login of the tc user when the Microkernel finishes booting)
cp -p tmp/first_checkin.yaml tmp/mk_conf.yaml tmp-build-dir/tmp
cp -p etc/inittab tmp-build-dir/etc
# check to see if we're building a "Debug ISO"; if so, use sed to modify the inittab
# file we just copied over so that re-enables autologin
if [ $BUILD_DEBUG_ISO = 'yes' ]; then
AUTO_LOGIN_STR='-nl /sbin/autologin'
OLD_INITTAB_TTY1_PAT='^\(tty1.*\)\(38400 tty1\)$'
sed -i "s/$OLD_INITTAB_TTY1_PAT/\1$(echo $AUTO_LOGIN_STR | sed -e 's/\\\\/\\\\\\\\/g' -e 's/\\//\\\\\\//g' -e 's/&/\\\\\\&/g') \2/" tmp-build-dir/etc/inittab
fi
# get a copy of the current Tiny Core Linux "Core" ISO
file=`echo $TCL_ISO_URL | awk -F/ '{print $NF}'`
if [ $RE_USE_PREV_DL = 'no' ] || [ ! -f tmp-build-dir/build_dir/$file ]
then
wget -P tmp-build-dir/build_dir $TCL_ISO_URL
fi
# download the MCollective, unpack it in the appropriate location, and
# add a couple of soft links
file=`echo $MCOLLECTIVE_URL | awk -F/ '{print $NF}'`
mcoll_dir=`echo $file | cut -d'.' -f-3`
if [ $RE_USE_PREV_DL = 'no' ] || [ ! -f tmp-build-dir/$file ]
then
wget -P tmp-build-dir $MCOLLECTIVE_URL
fi
cd tmp-build-dir/usr/local/tce.installed
tar zxvf $TOP_DIR/tmp-build-dir/$file
cd $TOP_DIR/tmp-build-dir
rm usr/local/mcollective usr/local/bin/mcollectived 2> /dev/null
ln -s /usr/local/tce.installed/$mcoll_dir usr/local/mcollective
ln -s /usr/local/mcollective/bin/mcollectived usr/local/bin/mcollectived
cd $TOP_DIR
# add a soft-link in what will become the /usr/local/sbin directory in the
# Microkernel ISO (this fixes an issue with where Facter expects to find
# the 'dmidecode' executable)
mkdir -p tmp-build-dir/usr/sbin
rm tmp-build-dir/usr/sbin 2> /dev/null
ln -s /usr/local/sbin/dmidecode tmp-build-dir/usr/sbin 2> /dev/null
# copy over a few additional dependencies (currently, this includes the
# following files:
# 1. ssh-setup-files.tar.gz -> contains the setup files needed for the
# SSH/SSL (used for development access to the Microkernel); if
# the '--build-prod-image' flag is set, then this file will be skipped
# 2. mcollective-setup-files.tar.gz -> contains the setup files needed for
# running the mcollective daemon
# 3. mk-open-vm-tools.tar.gz -> contains the files needed for the
# 'open_vm_tools.tcz' extension
# 4. the etc/passwd and etc/shadow files from the Razor-Microkernel project
# (note; if this is a production system then the etc/shadow-nologin
# file will be copied over instead of the etc/shadow file (to block
# access to the Microkernel from the console)
cp -p additional-build-files/*.gz tmp-build-dir/build_dir/dependencies
file=`echo $OPEN_VM_TOOLS_URL | awk -F/ '{print $NF}'`
if [ $RE_USE_PREV_DL = 'no' ] || [ ! -f tmp-build-dir/build_dir/dependencies/$file ]
then
wget -P tmp-build-dir/build_dir/dependencies $OPEN_VM_TOOLS_URL
fi
# Copy over the etc/passwd file to the tmp-build-dir/etc directory.
# If we're building a production system, development system, also copy over the
# etc/shadow file to the same directory. If it's a production system we're
# building the ISO for, then copy over the etc/shadow-nologin file instead
# (and remove the SSH setup files from the files we just copied over to the
# dependencies directory)
cp -p etc/passwd tmp-build-dir/etc
if [ $BUILD_PROD_ISO = 'no' ]; then
cp -p etc/shadow tmp-build-dir/etc
# if a password for the tc user was passed in (using the -t or --tc-passwd flag)
# then use it to replace the default password for the tc user in the shadow
# password file we're burning into the ISO here (requires that openssl be installed
# locally for this to work)
if [ ! -z $TC_PASSWD ]; then
NEW_PWD_ENTRY=`echo $TC_PASSWD | openssl passwd -1 -stdin`
# use sed to replace the default password with the new one generated (above)
# (but remember, need to escape the replacement string for use with sed first,
# which is what the "$(echo ... | sed -e ...)" part of this command does; it
# escapes any '\', '/', and '&' characters in the $NEW_PWD_ENTRY string so that
# they will be passed as literals during replacement instead of being used as
# part of the surrounding sed command)
sed -i "s/^\(tc:\)[^\:]*\(.*\)/\1$(echo $NEW_PWD_ENTRY | sed -e 's/\\\\/\\\\\\\\/g' -e 's/\\//\\\\\\//g' -e 's/&/\\\\\\&/g')\2/" tmp-build-dir/etc/shadow
fi
else
cp -p etc/shadow-nologin tmp-build-dir/etc/shadow
rm tmp-build-dir/build_dir/dependencies/ssh-setup-files.tar.gz
fi
# get the latest util-linux.tcz, then extract the two executables that
# we need from that file (using the unsquashfs command)
file='util-linux.tcz'
if [ $RE_USE_PREV_DL = 'no' ] || [ ! -f tmp-build-dir/$file ]
then
wget -P tmp-build-dir $TCL_MIRROR_URI/$file
fi
unsquashfs -f -d tmp-build-dir tmp-build-dir/util-linux.tcz `cat additional-build-files/util-linux-exec.lst`
# create a gzipped tarfile containing all of the files from the Razor-Microkernel
# project that we just copied over, along with the files that were downloaded from
# the network for the gems and TCL extensions; place this gzipped tarfile into
# a dependencies subdirectory of the build_dir
cd tmp-build-dir
tar zcvf build_dir/dependencies/razor-microkernel-overlay.tar.gz usr etc opt tmp
# and create a gzipped tarfile containing the dependencies folder and the set
# of scripts that are used to build the ISO (so that all the user has to do is
# copy over this one file to a directory somewhere and unpack it and they will
# be ready to build the ISO
bundle_out_file_name='razor-microkernel-bundle-dev.tar.gz'
if [ $BUILD_PROD_ISO = 'yes' ]; then
bundle_out_file_name='razor-microkernel-bundle-prod.tar.gz'
elif [ $BUILD_DEBUG_ISO = 'yes' ]; then
bundle_out_file_name='razor-microkernel-bundle-debug.tar.gz'
fi
cd build_dir
tar zcvf $TOP_DIR/build-files/$bundle_out_file_name *
cd $TOP_DIRFollowed by the post-build directory structure:
tmp-build-dir/
├── build_dir
│ ├── add_version_to_mk_fs.rb
│ ├── build_initial_directories.sh
│ ├── build_iso_yaml.rb
│ ├── Core-current.iso
│ ├── dependencies
│ │ ├── mcollective-setup-files.tar.gz
│ │ ├── mk-open-vm-tools.tar.gz
│ │ ├── razor-microkernel-overlay.tar.gz
│ │ └── ssh-setup-files.tar.gz
│ └── rebuild_iso.sh
├── etc
│ ├── init.d
│ │ └── dhcp.sh
│ ├── inittab
│ ├── passwd
│ └── shadow
├── mcollective-2.0.0.tgz
├── opt
│ ├── bootsync.sh
│ ├── gems
│ │ ├── daemons-1.1.8.gem
│ │ ├── facter-1.6.9.gem
│ │ ├── gem.list
│ │ ├── json_pure-1.7.3.gem
│ │ └── stomp-1.2.2.gem
│ └── rubygems-1.8.24.tgz
├── tmp
│ ├── builtin
│ │ ├── onboot.lst
│ │ └── optional
│ │ ├── bash.tcz
│ │ ├── dmidecode.tcz
│ │ ├── firmware-bnx2.tcz
│ │ ├── gcc_libs.tcz
│ │ ├── libssl-0.9.8.tcz
│ │ ├── lshw.tcz
│ │ ├── openssh.tcz
│ │ ├── openssl-1.0.0.tcz
│ │ ├── ruby.tcz
│ │ └── scsi-3.0.21-tinycore.tcz
│ ├── first_checkin.yaml
│ ├── mk_conf.yaml
│ └── tinycorelinux
│ ├── 4.x
│ │ └── x86
│ │ └── tcz
│ │ ├── fuse.tcz
│ │ ├── glib2-dev.tcz
│ │ ├── glib2.tcz
│ │ ├── libdnet.tcz
│ │ ├── libffi-dev.tcz
│ │ ├── libffi.tcz
│ │ ├── ncurses-common.tcz
│ │ ├── ncurses.tcz
│ │ ├── perl5.tcz
│ │ └── procps.tcz
│ ├── kmod-install-list.yaml
│ └── tce-install-list.yaml
├── usr
│ ├── local
│ │ ├── bin
│ │ │ ├── lscpu
│ │ │ ├── mcollectived -> /usr/local/mcollective/bin/mcollectived
│ │ │ ├── rz_mk_controller.rb
│ │ │ ├── rz_mk_control_server.rb
│ │ │ ├── rz_mk_init.rb
│ │ │ ├── rz_mk_tce_mirror.rb
│ │ │ └── rz_mk_web_server.rb
│ │ ├── lib
│ │ │ └── ruby
│ │ │ └── 1.8
│ │ │ └── razor_microkernel
│ │ │ ├── logging.rb
│ │ │ ├── rz_host_utils.rb
│ │ │ ├── rz_mk_configuration_manager.rb
│ │ │ ├── rz_mk_fact_manager.rb
│ │ │ ├── rz_mk_gem_controller.rb
│ │ │ ├── rz_mk_hardware_facter.rb
│ │ │ ├── rz_mk_kernel_module_manager.rb
│ │ │ ├── rz_mk_registration_manager.rb
│ │ │ └── rz_network_utils.rb
│ │ ├── mcollective -> /usr/local/tce.installed/mcollective-2.0.0
│ │ ├── sbin
│ │ │ └── sfdisk
│ │ └── tce.installed
│ │ └── mcollective-2.0.0
│ │ └── ...
│ ├── sbin
│ │ └── dmidecode -> /usr/local/sbin/dmidecode
│ └── share
│ └── udhcpc
│ └── dhcp_mk_config.script
└── util-linux.tcz