Revisit automated build process

[taylortom]

There are a number of tickets related to adding functionality to the build process:

• Create a JSLint file for checking compliance with coding standards #201
• Grunt Tracking ID are not generated on build #208
• Enable source mapping in LESS files during grunt dev #280
• Tracking ID's only work in en folder #370

I also think we need to tackle the use-case of allowing output to multiple builds (this ties in with #370, so I think we can kill two birds with one stone there).

Kineo have a vastly modified version here: https://github.com/cgkineo/adapt-grunt, which has possible solutions we can use for a few of these issues.

Alternatively, @oliverfoster has also been looking into using gulp to speed up performance, as the current grunt process can be very time consuming depending on course size.

[EDIT] New related issue added: #503

[EDIT 2] Look into adding jshint to help with code conformity

[oliverfoster]

I'm currently in the prototype stages of the gulp build process.

I think I've got the requirements right. At the moment I'm concentrating on improvements to assets management (change management between build and source to stop continually moving/copying large resources), linting (js, handlebars, less and json) and sourcemaps (for js, handlebars, less and json)

I'll make a brief and list my requirements if/when I think we could use it in production. There are a few small issues at the moment, such as the non-standard way Adapt utilizes requirejs.

[brian-learningpool]

@oliverfoster, I know it's probably early days but I'm curious as to how much of a performance increase you have noticed using Gulp? This would be very useful for the authoring tool.

[olivermartinfoster]

The performance is quite compatible when looked at from a gulp vs grunt perspective as they are both relatively light frameworks running in the same environment. Disk r/w speeds and processing performance are practically identical.

The difference comes when I've looked a the process/task abstractions. Gulp being a array of files streaming through processing pipes means that it reads from the disk once at the start of each task and write once at the end of each task. This means that the number of i/o operatings per process comes down considerably from grunt.

Because each task is a chain of processes using data stores in volatile memory I can do a lot of processing on each file group at good speed before completing and writing. This invariably means that the entire logic for turning file group X into file and sourcemap Y is also a lot easier to understand.

There are also other things that can take advantage of this in-memory processing, like chaining many processes together to form complex maintainance operations I.e.:

fetch destination files, fetch source files, compare and remove missings files from the destination which are not present in the source and then update changed items from source to destination.

This works beautifully when dealing with large video assets as only the exact changes are acted upon saving effort tidying the build folder and I/o utilisation moving files that haven't been changed or added.

Catching errors in the build process is also easier so that I don't have to restart the build utility when writing code anywhere near as often.

Linting javascript, handlebars, less and json is easier.

Generating sourcemaps for less is easy as the entire set of processes we use with grunt collection, collating, building, source mapping and outputting are done in one task. No intermediate files.

I'm sure there are many other things I've missed, by it is the end of the day.

I hope that helps?

[olivermartinfoster]

Please excus grammatical and spelling errors, I'm not good with touchscreens.

[olivermartinfoster]

Short and skinny is: I really like it and it has very real potential for reducing the production costs of the development teams by providing faster, more dependable and vastly more informative debugging environments.

[olivermartinfoster]

As you said though, early days, it'd be nice to collaborate with someone on it. And get some more hands together. I understand though that it's not really the time for new things at the moment, so it may have to wait some.

[danielghost]

Not sure when it will be released but the roadmap for Grunt suggests that 0.5 will sort some of the speed issues. See https://github.com/gruntjs/grunt-docs/blob/master/Roadmap.md.

Also think it has a larger community currently so you may find more plugins that you can use. This may change in the future though.

In addition to performance, what is easier to setup new tasks?

[oliverfoster]

It entirely depends on what you're doing. For single tasks, grunt by a short stretch as most of it is json + some dependency amends and perhaps a new JavaScript task. For multitask processes, gulp probably/definitely wins by a good margin.

[oliverfoster]

For example, to watch and to build less with compiling, linting, sourcemapping and minifying would take two functions (watch and build), plus a line or two to give the functions a task name. Nice and neat. + the dependency's in the package json and the require lines at the top of the script.

[oliverfoster]

It's definitely aparant that grunt currently has the market share. There are a few new ones on the horizon though. All with different abstractions around the issue. I hear brunch.io should be coming along soon, they've gone the grunt route of config files, but they are waaaay more precise.

[olivermartinfoster]

https://github.com/olivermartinfoster/buildsystem/blob/master/gulpfile.js#L640-L719 < is an example of outputting less from an app folder + a core folder + each one of multiple config folders (equivalent to having multiple course folders) into relevant output directories per config folder. with change caching for watches (to futher reduce disk i/o and processing time), source map generation (with content inclusion), less compiling with compilation error output, file concatination, output minification, browser source hierarchy shifting (to control how sources look in the debugger), output writing and stream queuing, all in 79 lines (including whitespace). it could be rewritten in a much better way, but it gives an example of how something so small can solve many of our production issues in adapt.

[oliverfoster]

The only way I can manage LESS sourcemapping into Adapt is by smashing a bit of grunt and a bit of gulp together in the Gruntfile.js. It is however, remarkably effective:

grunt.registerTask('less-sourcemaps', 'Aligning sourcemaps', function() {
        var _ = require('underscore');

        var gulp = require("gulp");
        var concat = require("gulp-concat");
        var less = require("gulp-less");
        var sourcemaps = require("gulp-sourcemaps");

        return gulp
            .src([
                    'src/core/less/*.less',
                    'src/menu/**/*.less',
                    'src/components/**/*.less',
                    'src/extensions/**/*.less',
                    'src/theme/**/*.less'
                ], { base: "." })

                .pipe(sourcemaps.init())

                .pipe(concat( "adapt.css" ))

                .pipe(
                    less()
                )

                .on('error', function (error) {
                    gutil.log(gutil.colors.red(error.message))
                    // Notify on error. Uses node-notifier
                    notifier.notify({
                        title: 'Less compilation error',
                        message: error.message
                    })
                })

                .pipe(sourcemaps.write("./") )


                .pipe(gulp.dest("build/adapt/css")).on("end", function () {
                     //redo source map paths so that they end up in the right place in the browser

                    var mapFile = grunt.file.readJSON("build/adapt/css/adapt.css.map");
                    delete mapFile.sourceRoot;
                    _.each(mapFile.sources, function(item, index) { 
                        mapFile.sources[index]  = item.replace(/\\/g, "/").replace(/src\//g, "" );
                    });
                    grunt.file.write("build/adapt/css/adapt.css.map", JSON.stringify(mapFile, null,4));

                });

    });
...
grunt.registerTask('dev', ['jsonlint', 'copy', 'handlebars', 'bower', 'requirejs-bundle', 'requirejs:dev', 'create-json-config', 'less-sourcemaps', 'watch']);
...

Using these devDependencies in the package.json:

    "gulp": "^3.8.10",
    "gulp-concat": "^2.4.3",
    "gulp-less": "^2.0.1",
    "gulp-sourcemaps": "^1.3.0",

What do we think? PR or no PR?

[oliverfoster]

@darylhedley @brian-learningpool @danielghost

[oliverfoster]

Next question is: how do we cleanup the adapt.css.map file from the build folder on non-dev builds? Should each task be responsible for its house-keeping? Should we delete the build folder at each build? Or should we have a separate 'clean' task?

[oliverfoster]

I'm working on this, gulp has too many nested dependencies. Going for a different solution: Separating the build tool from the framework and using more native node plugins (rather than plugins than need to be compiled). Moving each task into something more readable and editable to open the tool up to more people in the community. I'll keep everyone updated.

[taylortom]

Gulp has been shelved as an alternative.