-
-
Notifications
You must be signed in to change notification settings - Fork 319
AllInSConstruct
This aims to be a complete, usable, and fairly non-trivial SCons example for C/C++.
I set out with the goal of making a simple to use SCons-based build set up that would automatically build Release and Debug variants into separate directories. Additionally I sometimes like to have my build consolidate all the libs, executables, and/or headers into given directories after building. For instance creating, a source_tree/Lib dir and a source_tree/Include directory for the libs and public header files for utility library projects.
And when you build multiple variations of a lib, some Debug some Release, you often want to give them different names, like 'foo.lib' vs 'foo_dbg.lib' (or 'libfoo.a' vs 'libfoo_dbg.a'). The scripts below handle all that, while providing a few other niceties.
This is still a work in progress, but already it does a lot of things I find useful.
Naturally, you want your SConscript files to be as simple as possible and basically have them 'include' all your standard rules for building. I considered doing that via the Python execfile() function, and storing the default rules in a separate file at the build root, but in the end it seemed like I might as well just Export/Import a build object with all the smarts in methods. Then the global build object can just be defined in the main SConstruct. Originally I thought it would be good to put all the source and target data etc into a big dictionary structure in the SConscript files and pass that to my processing methods, but upon consideration it became clear that what I was really accomplishing with all that was decorating, or 'wrapping', the default Program() and Library() builder methods. The WrapperFunctions Wiki page describes exactly how to do this in a much more straightforward way. Part of the benefit of a wrappers approach is that to the SConscript writer, the syntax looks pretty much like the basic SConscript examples in the SCons User's Guide.
So basically the result is that the SConscript just calls some things that look pretty similar to Program() and Library() calls (specifically, I've named them dProgram() and dLibrary()), but all the extra things happen that I described above. For example:
Import('env')
env.dLibrary(target='foo',source='foo.c *.cpp',CPPPATH=['#Include'])Complete examples of the SConstruct files are included below.
So what's good about it:
-
The SConscripts are relatively simple 1. Easy to write initially 1. Easier to edit by non-SCons experts
-
All the logic is in the SConstruct, so only need to modify SConstruct to change whole build behavior.
-
Variant build products are put in separate sub-directories (build/Debug, build/Release) 1. Using the convention of
debug=1to specify a debug build 1. Just two variants right now,DebugandRelease -
Each sub-SConscript can define multiple libs and exes to build and install, and also specify headers to install. 1. Build products and sources are described using same syntax as the built-in builders, like
Program()1. New functions includedProgram(),dLibrary(),dSubdirs(), anddHeaders(). The latter installs headers. -
Lists of source files can include glob patterns 1. Glob patterns search the source dir not the build dir. The latter is SCons' not-very-useful default. 1. Glob patterns relative to the root using the '#' character (e.g.
#subdir/*.cpp) also work properly. -
Optional install step for libs apps and headers. 1. Not a system install, but copying elsewhere in the build tree 1. E.g., allows copying all libs to a #lib directory 1. E.g., allows copying all exes to a #bin directory 1. E.g., allows copying all include files to a #include directory
-
Optional decoration of lib names based on build variant. E.g. tack on a
'_dbg'to the end of library names in the debug variant. What needs work: -
The mechanism for linking with external libraries. (ParseConfig?)
-
Proper settings of compiler flags for more compilers.
-
External specification of site-specific info like standard include dirs.
-
Overall better separation of config data and instructions in the SConstruct file. (Currently there are many things you have to delve into the guts of the SConstruct to change, like the names of the install directories for libs, programs, and headers.) Other possible negatives:
-
Too C/C++ specific.
-
Recent Python required (I use a fair number of list comprehensions. And being pretty new to Python, I may be using other 'new' Python stuff without realizing it, because it's all new to me.)
Here are the files.
An example SConscript for a dir with some libraries:
Import('env')
env.dLibrary(target='viewer',
libsrc = '''
viewer.cpp
#camera/camdisplay.cpp
#math/mat16fv.cpp
#timer/stopwatch.cpp
''',
CPPPATH = Split('#camera #math #math_objs #timer')
)
env.dHeaders('*.hpp #camera/*.hpp trackball.h')There's only one lib there, but you can of course make multiple dLibrary calls. Also note that not all the sources for this lib are located in the directory with this SConscript. That's fine. The dHeaders call is used to install the headers listed in the '#Include' directory of the tree. Note that it can take a single string that include a combination of globs, build relative globs, and non-globs. Any non-list argument is automatically Split() and anything that looks like a glob is globbed on (and globbed properly, in the source directory, not in the build directory as is the default SCons behavior)
An example SConscript for a dir with several binaries:
Import('env')
for appname in Split('''
test_basic
test_basic_oo
test_keymouse
test_menu
'''):
env.dProgram(target=appname, source=[appname + '.cpp'],
CPPPATH=Split('#viewer #camera #math'),
LIBPATH=['#lib'],
LIBS=['viewer'])The above example uses a loop, but you could certainly also just write each one, one-by-one, if you're afraid of Python and find that more intuitive.
Finally, the top-level Root SConscript just calls subdirs:
Import('env')
env.dSubdirs('subdir1 subdir2 librarysrc tests')And now the main SConstruct file:
# The goal here is to be able to build libs and programs using just
# the list of sources.
# I wish to 'wrap' or 'decorate' the base scons behavior in
# a few ways:
# * default variant builds ('Debug'/'Release')
# * automatic build dirs (which are variant-sensitive)
# - with work-arounds to make *everything* go in the build dir
# * automatic no-fuss globs (that *also* work properly for build dirs)
# * default platform-specific build env vars (also variant-sensitive)
# (i.e. some extra compiler specific CCFLAGS and such)
# * automatic 'installation' of libs, program, and headers into specific
# directories (e.g. '#lib', '#bin', and '#include')
import sys
############################################
# Site-specific setup
# Hmm... this should go somewhere else. SCustomize??
# This is my own site-specific win32 setup...
if sys.platform == 'win32':
stdinc = [r'c:\usr\include']
stdlibinc = [r'c:\usr\lib']
else:
stdinc = []
stdlibinc = []
############################################
# Generic boiler plate
#-------------------------------------------
import os
import os.path
opts = Options('SCustomize')
opts.Add('debug', 'Build with debugging symbols', 0)
opts.Add('CC', 'Set C compiler')
env = Environment(options=opts)
Help(opts.GenerateHelpText(env))
debug = env.get('debug',0)
build_base = 'build'
if debug:
env.Append(CPPDEFINES = ['DEBUG', '_DEBUG'])
variant = 'Debug'
else:
env.Append(CPPDEFINES = ['NDEBUG'])
variant = 'Release'
############################################
# PLATFORM SPECIFIC CONFIGS
############################################
#-------------- win32 MSVC -----------------
if env['CC'] == 'cl':
def freeMSVCHack(env, vclibs):
# SCons automatically finds full versions of msvc via the registry, so
# if it can't find 'cl', it may be because we're trying to use the
# free version
def isMicrosoftSDKDir(dir):
return os.path.exists(os.path.join(dir, 'Include', 'Windows.h')) and os.path.exists(os.path.join(dir, 'Lib', 'WinMM.lib'))
def findMicrosoftSDK():
import SCons.Platform.win32
import SCons.Util
import re
if not SCons.Util.can_read_reg:
return None
HLM = SCons.Util.HKEY_LOCAL_MACHINE
K = r'Software\Microsoft\.NETFramework\AssemblyFolders\PSDK Assemblies'
try:
k = SCons.Util.RegOpenKeyEx(HLM, K)
p = SCons.Util.RegQueryValueEx(k,'')[0]
# this should have \include at the end, so chop that off
p = re.sub(r'(?i)\\+Include\\*$','',p)
if isMicrosoftSDKDir(p): return p
except SCons.Util.RegError:
pass
K = r'SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs'
try:
k = SCons.Util.RegOpenKeyEx(HLM, K)
i=0
while 1:
p = SCons.Util.RegEnumKey(k,i)
i+=1
subk = SCons.Util.RegOpenKeyEx(k, p)
try:
p = SCons.Util.RegQueryValueEx(subk,'Install Dir')[0]
# trim trailing backslashes
p = re.sub(r'\\*$','',p)
if isMicrosoftSDKDir(p): return p
except SCons.Util.RegError:
pass
except SCons.Util.RegError:
pass
return None
# End of local defs. Actual freeMSVCHack begins here
if not env['MSVS'].get('VCINSTALLDIR'):
if os.environ.get('VCToolkitInstallDir'):
vcdir=os.environ['VCToolkitInstallDir']
env.PrependENVPath('INCLUDE', os.path.join(vcdir, 'Include'))
env.PrependENVPath('LIB', os.path.join(vcdir, 'Lib'))
env.PrependENVPath('PATH', os.path.join(vcdir, 'Bin'))
env['MSVS']['VERSION'] = '7.1'
env['MSVS']['VERSIONS'] = ['7.1']
if not env['MSVS'].get('PLATFORMSDKDIR'):
sdkdir = findMicrosoftSDK()
if sdkdir:
env.PrependENVPath('INCLUDE', os.path.join(sdkdir, 'Include'))
env.PrependENVPath('LIB', os.path.join(sdkdir, 'Lib'))
env.PrependENVPath('PATH', os.path.join(sdkdir, 'Bin'))
env['MSVS']['PLATFORMSDKDIR']=sdkdir
# FREE MSVC7 only allows
# /ML(libc) /MT(libcmt) or /MLd(libcd)
# Full IDE versions also have
# /MD(msvcrtd) /MTd(libcmtd) and /MDd(msvcrtd)
# So if you want to debug with the freever, the only option is
# the single-threaded lib, /MLd
vclibs['Debug']='/MLd'
vclibs['Release']='/MT'
# MSVC SETUP
# MDd is for multithreaded debug dll CRT (msvcrtd)
# MD is for multithreaded dll CRT (msvcrt)
# These are just my preferences
vclibs = {'Debug':'/MDd','Release':'/MD'}
freeMSVCHack(env, vclibs)
env.Append(CCFLAGS=[vclibs[variant]])
if debug:
env.Append(CCFLAGS=Split('/Zi /Fd${TARGET}.pdb'))
env.Append(LINKFLAGS = ['/DEBUG'])
# env.Clean('.', '${TARGET}.pdb')
# Need to clean .pdbs somehow! The above line doesn't work!
else:
env.Append(CCFLAGS=Split('/Og /Ot /Ob1 /Op /G6'))
env.Append(CCFLAGS=Split('/EHsc /J /W3 /Gd'))
env.Append(CPPDEFINES=Split('WIN32 _WINDOWS'))
#-------------- gcc-like (default) ---------
else: # generic posix-like
if debug:
env.Append(CPPFLAGS = ['-g'])
else:
env.Append(CPPFLAGS = ['-O3'])
#-------------------------------------------
# Put all the little .sconsign files into one big file.
# (Does this slow down parallel builds?)
# Need to create the build dir before we put the signatures db in there
fullbuildpath = Dir(build_base).abspath
if not os.path.exists(fullbuildpath): os.makedirs(fullbuildpath)
import dbhash
env.SConsignFile(os.path.join(build_base, 'sconsignatures'), dbhash)
# Make a singleton global object for keeping track of all the extra data
# and methods that are being added
class Globals:
def __init__(self):
self.env = env
self.stdinc = stdinc
self.stdlibinc = stdlibinc
self.variant = variant
self.build_base = os.path.join(build_base, variant)
self.libname_decorators = { 'Debug' : '_dbg' }
self.appname_decorators = { 'Debug' : '_dbg' }
self.incinstdir = '#include'
self.libinstdir = '#lib'
self.appinstdir = '#bin'
self.objcache = {}
def Glob(self, pat):
## GLOB IN THE REAL SOURCE DIRECTORY (NOT BUILD DIR)
import glob
prevdir = os.getcwd();
if pat[0] != '#':
os.chdir(self.env.Dir('.').srcnode().abspath)
ret = glob.glob(pat)
else:
pat = pat[1:]
base = os.path.dirname(pat)
searchdir = self.env.Dir('#').srcnode().abspath
os.chdir(searchdir)
ret = ['#'+x for x in glob.glob(pat)]
os.chdir(prevdir)
return ret
def GlobExpand(self, list):
## look for pattern-like things and glob on those
ret = []
for item in list:
if item.find('*') or item.find('?') or item.find('['):
ret += self.Glob(item)
else:
ret += [item]
return ret
def IsALocalLib(self, lib):
# This is rather heuristic determining if a lib is local or not
return lib.find('/') or lib.find(os.sep) or lib[0]=='#'
def MyHeaderMethod(self, env, source, **dict):
if type(source)==type(''): source = Split(source)
source = self.GlobExpand(source)
nodes = []
if hasattr(self,'incinstdir') and self.incinstdir:
for i in source:
nodes.append( env.Install(self.incinstdir, i) )
return nodes
def MyLibraryMethod(self, env, **dict):
relincs = dict.get('CPPPATH',[])
dict['CPPPATH'] = ['.'] + relincs + self.stdinc
# These shenanigans are necessary to get SCons to build non-local
# sources in the VariantDir instead of their own local directories
target = dict.pop('target')
source = dict.pop('source')
if type(source)==type(''): source = Split(source)
allsrc = []
for x in source:
objbase = os.path.basename(x)
if self.objcache.get(objbase):
#print 'Reusing node', objbase
#NOTE: We should check that defines etc are all the same!!
allsrc += self.objcache[objbase]
else:
onode = env.SharedObject(
os.path.splitext(objbase)[0], x,
**dict)
allsrc += onode
self.objcache[objbase]=onode
targpath = '#' + os.path.join(self.build_dir, target)
# decorate libname with e.g. '_dbg'
if hasattr(self,'libname_decorators'):
targpath += self.libname_decorators.get(self.variant,'')
dict['source'] = allsrc
dict['target'] = targpath
node = env.Library(**dict)
if hasattr(self,'libinstdir') and self.libinstdir:
env.Install(self.libinstdir, node)
return node
def MyProgramMethod(self, env, **dict):
# Enhance CPPPATH,LIBPATH
dict['CPPPATH'] = ['.'] + dict.get('CPPPATH',[]) + self.stdinc
dict['LIBPATH'] = dict.get('LIBPATH',[]) + self.stdlibinc
# These shenanigans are necessary to get SCons to build non-local
# sources in the VariantDir instead of their own local directories
target = dict.pop('target')
source = dict.pop('source',[])
if type(source)==type(''): source = Split(source)
allsrc = []
for x in source:
objbase = os.path.basename(x)
if self.objcache.get(objbase):
print 'Reusing node', objbase
allsrc += self.objcache[objbase]
else:
onode = self.env.SharedObject(
os.path.splitext(objbase)[0], x,
**dict)
allsrc += onode
self.objcache[objbase]=onode
targpath = '#' + os.path.join(self.build_dir, target)
# decorate app name with e.g. '_dbg'
if hasattr(self,'appname_decorators'):
deco = self.appname_decorators.get(self.variant,'')
targpath += deco
# decorate local lib names with e.g. '_dbg'
if hasattr(self,'libname_decorators'):
deco = self.libname_decorators.get(self.variant,'')
# decorate source lib names with e.g. '_dbg'
LIBS = []
for l in dict.pop('LIBS',[]):
if self.IsALocalLib(l):
#print "Decorating lib", l, '->', l+deco
LIBS += [l+deco]
else:
LIBS += [l]
dict['target']=targpath
dict['source']=allsrc
dict['LIBS']=LIBS
node = env.Program(**dict)
if hasattr(self,'appinstdir') and self.appinstdir:
self.env.Install(self.appinstdir, node)
return node
def MySubdirsMethod(self, env, subdirs, **dict):
# Build sub-directories
if type(subdirs)==type(''): subdirs=Split(subdirs)
for d in subdirs:
savedir = self.build_dir
self.build_dir = os.path.join(self.build_dir, d)
env.SConscript(os.path.join(d, 'SConscript'),
exports=['G','env'])
self.build_dir = savedir
G = Globals()
# Wrap the methods of G into method objects, and then add them
# as methods to Environment.
# See http://www.scons.org/cgi-bin/wiki/WrapperFunctions
def MyLibraryMethod(env, **dict):
G.MyLibraryMethod(env, **dict)
def MyHeaderMethod(env, source, **dict):
G.MyHeaderMethod(env, source, **dict)
def MyProgramMethod(env, **dict):
G.MyProgramMethod(env, **dict)
def MySubdirsMethod(env, subdirs, **dict):
G.MySubdirsMethod(env, subdirs, **dict)
from SCons.Script.SConscript import SConsEnvironment # just do this once
SConsEnvironment.dLibrary = MyLibraryMethod
SConsEnvironment.dProgram = MyProgramMethod
SConsEnvironment.dHeaders = MyHeaderMethod
SConsEnvironment.dSubdirs = MySubdirsMethod
G.build_dir = G.build_base
# Call the SConscript in the top-level directory
env.SConscript('SConscript', variant_dir=G.build_dir, exports=['env','G'])Note that a big chunk of that code has to do with trying to find out if the free VC++ toolkit is installed. SCons doesn't currently (ver 0.96.90) detect this even if it is the only compiler available. I just happened to be using the free toolkit while I waited for my actual copy of Visual Studio to arrive, when I started playing with SCons. Consequently I spent a lot of my first day or so with SCons struggling to figure out why on earth the simplest compile with VC required so much code to get the environment right. Well since I struggled through it, I'm including the results of that struggle here in the form of the freeMSVCHack() function above. Hopefully something similar will be built into future releases of SCons.
That's it for now. This is still a work in progress as of 2/18/05, JST. I'll keep updating this page as I make progress. Feedback from the experts is much appreciated.
One last thing, why did I name all my functions like dProgram or dLibrary? What's with the d?
The answer is just I wanted to use names similar to the existing builders I was wrapping, and d seemed like a decent prefix. Maybe it stands for different or decorated or decent. Or maybe dorky :-)
Bill Baxter
Thanks to Steven Knight, Dobes Vandermeer, Gary Oberbrunner and the other folks on the SCons users mailing list for helpful suggestions, pointers, and advice. And of course a big thanks to Steven Knight and the other developers of SCons.