-*- org -*-
This is an Emacs org-mode file. (Tags are GitHub issue numbers. See https://github.com/zrajm/fix/issues/)
–config=FILE, $FIX_CONFIG
Set config file (Default: “$FIX_WORK_TREE/.fixrc”)
–fix-dir=DIR, $FIX_DIR, core.fixDir
Set build metadata dir to DIR. (Default: “$FIX_WORK_TREE/.fix”)
–work-tree=DIR, $FIX_WORK_TREE, core.workTree
Will set the worktree of the build project. If worktree is not explicitly given, it will be the first parent directory that contains a file named ‘.fixrc’.
The value of this option is implicitly prepended to all build arguments. So that if ‘targettree = build’, then ‘fix test.txt’ will be equivalent to ‘fix build/test.txt’. (If this value is not set, then the current worktree path is prepended instead.)
Included in milestone for version 1.0.
When a buildscript runs, its working directory will be in the source dir, so that the following will work, regardless of whether you use a separate source dir or not. (If your buildscripts are running commands that are particular to your project, you can add their directory to your $PATH in ‘.fixrc’, which is read before each build.)
markdown2html <index.md
(This will produce HTML on standard output.) Or:
markdown2html <index.md >$1
(Which writes its output directly to the temporary output file.)
Notice that you can have subdirs, too. Let’s imagine you have a project that looks like this:
work/ \_ script/ \_ default.html.fix \_ source/ \_ guide/ \_ index.md \_ target/
If you now build ‘target/guide/index.html’, fix will first look for a matching buildscript, in this order:
script/guide/index.html.fix script/guide/default.html.fix script/guide/default.fix script/default.html.fix <– Found! script/default.fix
Fix will then change the current directory to the source dir of the target to build (in this case ‘source/guide’) And thereafter the buildscript is started and two arguments are passed to it:
$1 - (full path to) temporary output file $2 - filename (without path) of target file buing built
Included in milestone for version 1.0.
If ‘abc/TARGET.txt’ was generated using buildscript ‘default.fix’, it needs rebuilding if any of the higher priority buildscripts are ever created.
abc/TARGET.txt abc/default.txt.fix abc/default.fix default.txt.fix default.fix
- get all command line options (one pass)
- invoke special options: –help, –man, –config
- set options to builtin defaults
- override options with read config file options
- override options with command line options
This allows for splitting tests into different batches, which is nice when trying to figure out what has gone wrong somewhere.
Putting all tests in `t/` etc.
The fix program should be usable as-is, but all the extras (e.g. tests, code coverage and such) that require extra infrastructure should be buildable using standard (modern) CPAN methods.
http://search.cpan.org/~leont/Module-Build-0.4205/lib/Module/Build.pm
Should not display the ‘(previously built)’ message (since we’ll have to rebuild it anyway). It currently looks like this:
> fix lolo fix lolo (previously built) fix lolo
Currently a rebuild cannot be made if the fixdir has been deleted, but a previously built target still exists. It currently looks like this:
> ./fix lolo fix lolo > rm ./fix > ./fix lolo fix lolo fix lolo: Target externally modified, aborting
Target should here be rebuilt, and if the newly generated target looks the same as the previous one the build should be registered as successful (with the old file kept, and the tempfile deleted). Here’s some pseudo code:
build tempfile if tempfile != old_target if not –force error ‘target externally modified’ return FAILURE endif rename tempfile -> old_target else delete tempfile endif register successful build return SUCCESS
A tempfile should not be overwritten, if it already exists, and wasn’t produced by fix itself.
If a file has unchanged dependencies (and dependencies of dependencies) then there is no need to try to build it again, since we know build will fail this time around too…
Abort should not exit brutally (e.g. causing stats to be broken except for the file causing the abort), but recurse gracefully back again. build_target() and update_targets() should maybe have three return values?
1 = I did my stuff, and there were changes 0 = I did my stuff, nothing was changed -1 = Something went wrong, KILL THIS BUILD NOW!
Means we could ditch the runstate as well (since its currently only used for this).
Relative to the user’s PWD, not work_tree, or anything else.
Right now the output is:
fix lolo (previously built) fix lolo fix Cannot run buildscript ‘lolo.fix’: Permission denied
(And similarly when the shebang is invalid.) The two first lines should (probably) not be present in the output.
Use buildscript with ‘kill -SIGNAL $$’ in to test this.
Not relative to worktree (as it is now).
debug(), info(), msg(), quit() and friends should allow sprintf-like substitution, where first argument is a string with optional %s in, and the following are filenames. These filenames should be automatically rewritten to be relative to $Opt{pwd} before output. Example:
debug(“Building target: %s”, $target);
A ‘default.fix’ buildscript executing the non-existing command “foo” will give the following somewhat obscure message.
/home/zrajm/fix/script/default.fix: 1: \ /home/zrajm/fix/script/default.fix: foo: not found
It’s not exactly clear what’s going on here, and the error message should at least provide a clear context, to assist the user in remedying the error.
If scripttree changes between builds, the build metadata does not update properly. (There’s a NOTA BENE in the docs about this.)
delete() should, after having removed a file, delete the directory it resided in if that dir became empty (all the way up to the storage dir).
As in git, all statefiles should be chmodded ‘-w’ to prevent accidental deletion.
(More like git’s config format.) – This is needed for ‘core.execPath’, which should be allowed to occur multiple times.
Currently only one value of ‘core.execPath’ can be specified in the config file (though this isn’t all that damaging, since one may give several paths separated by ‘:’).
(More like git’s config format.)
(More like git’s config format.)
File functions should not exist in multiple incarnations. The following should be combined:
- _read_file() and read_file()
- _write_file() and write_file()
All file and filename manipulation should be gathered in one module, and stuff should be imported from there as needed.
As it is now checksum dependencies seem to be working but is sometimes oddly inefficient.
Which checksums should be passed in initially?
Test “Rebuild from ‘a.txt’” (of the checksum dependency tests) require two invocations of ‘fix’ – this seems wasteful, since only one target needs to be rebuilt.
Fork and run parallel ‘fix’ for each argument given.
I want to be able to write ‘fix all’ anywhere in the worktree hierarchy, and have it have the same effect. How about creating aliases with a section like this in ‘.fixrc’?
[alias] all = ./all clean = –clean
These would all execute from the worktree root (regardless of where you are in the worktree at the time of execution), and whatever is to the right of the equal sign would be passed to fix as arguments.
Alias names would just be an alphanumeric word, so build a file with the same name, one would use an explicit relative path (e.g. ‘fix ./help’).
Apenwarr does not produce an output file if the file written is zero bytes in length, but this is a bit of an odd special case which fix does not have to care about if there is alias capability.
There should be an alias to execute on plain ‘fix’ (without arguments). How should this be indicated in the config? By having an alias called ‘default’, maybe? (Since this is what the default rules are called, it is not very likely that a user would name a target ‘default’.)
Write each buildscripts STDERR to file, and, if build fails, output content of the failing scripts STDERR, before terminating.
State storage should use same file format as the ‘.fixrc’ file, i.e. something as close as possible to the .ini format.
checksum = b282f06fc2cb678ccf84e3ae8b1317be81252391 deps = ‘ab12.fix’ deps = ‘ab’, ‘12’ type = target
Should load into a data structure looking like this:
{ ‘checksum’ => ‘b282f06fc2cb678ccf84e3ae8b1317be81252391’, ‘deps’ => [ [ ‘ab12.fix’ ], [ ‘ab’, ‘12’ ] ], ‘type’ => ‘target’ }
Described in POD (uncomment when implemented). Alternative option names ‘–all’, ‘–rebuild’ or ‘–everything’.
Building should be so good and dependable that this option is never, ever needed, nevertheless, it should exist for emergencies.
Described in POD (uncomment when implemented).
Should be used in buildscript to indicate that it should always be considered dirty, and needs to be rebuilt every time. This would be used for a target that use outside information that is expected to always change (e.g. running processes or something like that).
Best way to use this flag would be to make a special buildscript, using ‘fix –dirty’, that capture this information in a file (doing a minimal amount of work) and then have other buildscripts depend upon this file. That way, if this always-dirty buildscript generate an output identical to the previous output a minimal amount of work would have to be done.
Described in POD (uncomment when implemented). Could also be called ‘–overwrite’.
Described in POD (uncomment when implemented).
Should be able to output a general dependency graph, as well a specific graph of a specific run (displaying which measures taken, and which targets were rebuild). – Visualizing what parts of a dependency tree has been checked, and in which way, should help in optimizing the main loop. I wants this! :)
List which target(s) will be rebuilt in the next rebuild (without actually rebuilding anything).
Option name ‘–watch’ or ‘–auto’ maybe? Would use inotifywait to look for changes and rebuild, whenever something is written.
Test that relative dirs are interpreted as relative to:
- –exec-path – user current dir
- $FIX_EXEC_PATH – user current dir
- core.execPath – worktree directory
Currently the name of the interrupted subtest is never outputted, which makes it hard to see where the error occurred.
“Bad ‘#!’ interpreter in buildscript”.
Both with files as args, and ‘delete(“.”)’.
Add tests checking that source deps and failed source deps are written to the runstate when ‘fix –stats –source’ is run.
Test that relative dirs are interpreted as relative to:
- –exec-path – user current dir
- $FIX_EXEC_PATH – user current dir
- core.execPath – worktree directory
100% code coverage + negative tests
Each test should do less. All state should be created by the test setup code (so tests can be run in random order), and test only one specific thing.
Something similar to Path::Class.
An object oriented module where each dir- and filename is an object that stringifies to the platform-specific filename. This object should have methods for manipulating the filename (cleaning it up, do up down directories etc.).
More about Path::Class:
MacPorts man (“man, version 1.6c” according to “man –version”) does not have the ‘-l’ option to read from standard input (or, as far as I can determine from the manpage, any other way of accepting a manpage on stdin).
So should either use plain text + less, or write manpage to tempfile and display that.
An option for fix to output all available targets, which then could be used for completion?
The tricky part would be to list targets built with ‘default’ rules; but maybe this could be done by looking at source files + available ‘default’ buildscripts.
There seem to be a little bit of hassle to get it to work, and keep itself automatically up-to-date when pulling and cloning… :/
Reference: Including a GitHub Wiki in a Repositor
Some cute and fix-y animal? (Though a fox would be somewhat cliché.)
Some animal that builds stuff? A beaver? An ant? (Termites are kinda ugly, they seem to fit the bill otherwise, though.) Beaver with a chainsaw? (Kinda implies brute force, rather than intelligence/finesse, though.)
The “Bower” project have a very nifty GitHub page. I’d like something equally (graphically) inspiring.
Simplify and standardize all code.
https://github.com/ndmitchell/build-shootout
The below idea does not work well with stat fingerprints, but maybe it can be modified to do so?
What if, instead of using the source/target filename as the file/lookup name in the keepstore, one used the sha1sum? And likewise with dependencies. (This would make it very fast to check if deps need to be rebuilt – a single stat is all that is needed to answer whether a dependency exists and is up-to-date.)
name = abc/target.txt deps = fad4699ef2f5c08e2699618c50310fb39b742a27 \ 2c0f8562db8543857de01b0cabc88b978ebfb63d type = target
When building, full keepstate should be written upon completion of compile. And thereafter the old keepstate file should be removed. (This would mean that we can skip writing keepstate when one already exist – if checksum is the same, then everything else about it is the same too. We still have to write to tempfile, in order to get update atomic, though.)
Currently stat fingerprints are only used in the runstate and keepstate. Could it accelerate stuff more generally if we used it when examining targets and dependencies?
Test case, “Rerun after change of subtarget ‘2’” has invocations of fix in a “backwards” order:
fix ab fix 1 2 fix ab 12
Is this really as it should be?
Right now fix doesn’t care which dir it is in (except when executing buildscripts). But maybe we should do all dependency scanning while standing in the root dir of the work tree, then cd’ing for each buildscript we execute (cd’ing back to worktree root on recursive invocation). To much overhead?
The benefit is that we can use paths relative to the work tree dir internally, and do not have to rewrite/convert paths name before writing them to state storage.
Fix could delete dead targets. (Looking at their checksum to see whether or not they have changed. If no change has been made, then fix generated and “own” the target, and should be free to delete it.) Deleting of modified targets would be acceptable using ‘–force’.
From ”Four Interesting Build Tools” (by Neil Mitchell, February 2012):
Tup uses this: If a rule to build ‘foo’ is deleted from the rule list, then Tup automatically deletes the file ‘foo’. The problem of dead build results is serious, resulting in builds succeeding that should have failed, and that will fail as soon as a clean build is performed (to reduce this risk, we suggest an overnight build which starts from scratch). However, it is often useful to have build modes which generate skeleton files which are then modified by the user – deleting these files would be most unwelcome. It would be easy to add support for deleting dead build results to Shake, but we choose not to.
Piping the test output through the following “one-liner” makes output prettier. This should be built into the test script.
| perl -e ‘BEGIN{%x=(qw/not 31 ok 32/,”“,0)};$r=m/^(not | ok | )/;print”\e[$x{$&}m$_\e[m”’ |
Maybe use __DATA__ to store test script separately, and then pipe it through Perl to use it? That way the parser won’t have to process the test, unless they’re actually used.
A HTML file requires that a CSS file exists, but the HTML does not need to be recompiled when the CSS changes.
Seems excessive. Maybe only –debug should trigger this?
Is this really motivated? – If so, need to replace ‘mkpath’ and ‘rmtree’ with own functions.
A buildscript should be able to produce a directory, with content, or a symlink, and this should be handled properly (with changes properly detected etc) just like for normal files. (This would require checksumming the link name, and some representation of the dictionary content.)
Keeping old ‘fix’ around under the name ‘fix-old’ until all interesting stuff has been grabbed from it. 253 - One or more targets were externally modified 254 - Command line parse errors 255 - Low level error (failed to rename file etc.)Aborting on command line parse errors, should probably return 254(?) as this is quite an unlikely exit status from a buildscript. Also update ‘EXIT STATUS’ section in docs.
Described in POD (uncomment when implemented). Described in POD (uncomment when implemented). Runstate is the state that is cleared when starting B<fix> (it is used for communication between the mother process and her children) Each child should load and increment a counter, so that we, after a complete build run, may see inspect it to see how many times fix was executed in order to build all dependencies. All state written should be relative to work_tree. I.e. one should be able to run ‘fix test’, as well as ‘fix ../test’ or ‘fix HELLO/test’ (where all those are referring to the same target ‘test’) with the exact same result. Fix looks for ‘.fixrc’ to know where the base of its work tree is. If no work tree is found, use is advised to ‘touch .fixrc’ to fix this.Invoking fix when $PWD is not inside a fix worktree should result in an error (this is what git does, regardless of whether files on the command line are inside a work tree or not). Git gives the following error when you’re outside the work tree:
fatal: Not a git repository (or any of the parent directories): .git
Change to the dir containing the buildscript before running it. Git says the following when you try to do this: fatal: ‘..’ is outside repository (It’s too easy to forget to run one of them before committing.) Should pass (same as Gup, but with both filenames as relative paths):$1 - path to a temporary output file $2 - path to the target
If buildscript is in same directory as target, both $1 and $2 will be a simple filenames (with no path components).
If buildscript is in a parent dir of the target, then that dirname will be included in $1 and $2.
E.g. if you’re building ‘dir/target’, and the buildscript invoked is ‘default.fix’ (one directory down from the target). Then $1 = ‘dir/target-fixing’, and $2 = ‘dir/target’. (This is true even if you run fix from inside ‘dir’, using ‘fix target’; or even if you’re inside ‘dir/subdir’ and run it using ‘fix ../target’.)
NOTA BENE: When implemented, check all occurrences of $1, $2 and $3 in POD and make sure they’re updated to reflect what is actually passed to the buildscripts.
If TARGET.fix isn’t found, look for other possible buildscripts (in current directory and beyond). Look for these:TARGET.A.B.fix default.A.B.fix default.A.fix default.fix ../default.A.B.fix ../default.A.fix ../default.fix ../../default.A.B.fix ../../default.A.fix ../../default.fix
Stop the search at the root of the work tree (= git-like behavior).
All testing is now done in run_fix_test(). Desired effects are all expressed as args, severely reducing test sizes and making tests more readable. Which convert a pathname to a relative path. (They’re currently relative to the current dir.) Some tests will be affected by this. Relevant tests have the comment: ‘FIXME: When stats are relative to $FIX_WORK_TREE.’Dependency should not only indicate file, but the content (i.e. checksum) of that file. Let’s say you have two targets, like this:
a -> b
Initially you run ‘fix a’, to build both of them. Then you make a couple of changes to the buildfile for ‘b’ (that cause a change in ‘b’), and rebuild that with ‘fix b’.
If only naive dependencies are implemented, the keepstate is now in an inconsistent state. When ‘a’ was first built, it was registered as up-to-date, and when we modified and rebuilt ‘b’ it, too, was registered as up-to-date. But the current state of ‘a’ include the old version of ‘b’! Running ‘fix a’ at this point will fix nothing since keepstate storage says everything is ok (even though ‘a’ should be updated to include the change in ‘b’)!
Because of this each dependency need to include a checksum.
Added compatibility subtest() function for older versions of Test::More (i.e. for my desktop machine).The compatibility function, is created whenever subtest() cannot be found after loading Test::More, is just a simplistic emulation of the actual subtest(). It performs the tests (without them being ‘sub’ in any way) and thereafter outputs the name argument. It looks like this:
BEGIN { *subtest = sub { my ($name, $code) = @_; $code->(); note $name; } unless exists &subtest; }
Fix will use a semantic versioning scheme from version 1.0.0 and thereafter.- major (1st number) – Breaking changes.
- minor (2nd number) – New, but backward-compatible changes.
- patch (3rd number) – Smaller changes.
Whenever a major or minor version number changes, it will be added as a git ‘tag’. Patch number will simply be the number of changes made to the ‘fix’ file since last tag.
This git commands lists the number of commits since the last tag, i.e. the number of lines should equal the <patch> number.
git log –follow –oneline “$(git tag|tail -n1)”.. fix
Base64 checksums are 27 characters long (hex are 40). Shorter checksums makes for smaller files and easier comparison of diffs in test output. Errors in options are reported separately (i.e. if options parsing fails fix die()s without don’t try to check any of the other stuff). State store should have a format version number, and fix should refuse to read the files in ‘.fix’ if it will not be able to make sense of them anyway. Inline documentation should only contain stuff that you need to know when using fix + links to the GitHub repo and its wiki. Info on background, inspiration etc. should reside exclusively on the wiki. Included in milestone for version 1.0.This should be a .ini type config file (just like git’s). It will always be read before any buildscript is invoked, and can be used to set up paths, aliases etc. Here’s an example (adapted from git(1)):
#
#
; Source, script and target directories. [core] ; Recommended defaults. scriptdir = fix targetdir = build addpath = bin
[alias] all = ./all clean = –clean
Included in milestone for version 1.0.And the corresponding ‘core.addPath’ config file variable.
Quoting should me more like described in git-config(1) (under “CONFIGURATION FILE > Syntax”).Only allow the following escape sequences (beside " and \): \n for newline character (NL), \t for horizontal tabulation (HT, TAB) and \b for backspace (BS). No other char escape sequence, nor octal char sequences are valid.
‘–exec-path’ is the name of git’s option with analogous function. Environment variable $FIX_EXEC_PATH should also work. All buildscripts should have a ‘#!’ line at the beginning, and have execute permissions (–x–x–x) turned on.A solution to the portability (and a bunch of other) problems
The above post on the Google Group for redo, suggests that all .do files should be “opaque executables”. (Arguing that this would be a killer app for redo, allowing everyone to write build scripts in their own language.)
Pollard’s critique of the #! processing in apenwarr redo:
This is an unfortunate byproduct of its assuming that “do” files are Bourne shell scripts and only requiring #! when they are not. It passes all “do” scripts directly to the Bourne shell, and thus has to check for #! and do its own #! processing duplicating, not necessarily exactly or correctly, what the kernel does. In particular, the #! implementation in Pennarun redo as of 2012 has several of the well-known and widely documented #! security holes that have long since been closed in actual operating system kernel implementations of the same mechanism, including the script filename race condition. It also has an incorrect magic number check and an additional PATH security hole caused by how it attempts to run the Bourne shell.
A better approach would have been to use the Unix philosophy as it stood: a program to be run is just run, with execve(), as-is; “do” scripts are expected to all have #! to identify their particular interpreters; the system relies on the operating system kernel’s processing of #! and doesn’t roll its own one badly; and there is no hardwiring of one particular shell.
MacPorts man (“man, version 1.6c” according to “man –version”) does not support reading of manpage from standard in (the ‘-l’ option on GNU(?) man).To be cross platform ‘–man’ now displays its output as a plain text file (piped to ‘less’ instead of ‘man’), which forgoes some of the more fancy-looking stuff, but works.
Now use the (more) os-agnostic shebang/magic number:#!/usr/bin/env perl
For this to work $PATH must be set. If not, env will not be able to find the perl interpreter. – There was a problem when implementing this, wherein some of the ‘execPath’ tests cleared the $PATH variable which caused fix to fail with the above shebang, but work just fine with either `#!/usr/bin/perl` or `#!/opt/local/bin/perl`. Confusing as hell!