Aborting jobs

[olekristensen]

Hi there

First and foremost 🙏 thanks for this project. We're a small team at DR (the Danish National Broadcaster) producing breakers for flow tv with scripted After Effects renders, a browser-based frontend etc.

We've been following this project for a while, and decided to integrate nexrender into our node-based render server to replace our own legacy code. It took two afternoons, and our jsx script library just worked. Not needing to open the GUI to build a comp and save a temporary AEP, saves us up to 2 minutes on the renders!

There's just one thing that I would love to discuss: Aborting jobs. This is relevant for us to because we let the users ask for previews, and we want them to be able to let one preview job replace another in progress (say after a couple of edits) aborting the latter.

Since AE 2022 we can see that the aerendercore processes detatch completely from the aerender process. And killing the instance we get from nexrender, leaves the job running.

In our case we maintain our own render queue, and when deleting a running job from the queue the server with the job notices that it's no longer in the table and aborts it.

This is what we do, and it is a bit messy and can lead to some interesting errors:

// Save the instance from nexrender on our renderObject.
onInstanceSpawn : (instance, job, settings) => {
	me.instance = instance;
}

// Run periodically to check if this job is still in the queue from the server
// and still assigned to this machine.
// Otherwise abort it
this.checkRenderJobs = function (allJobsFromServer) {
	var jobOk = false;
	for (var ii = 0; ii < allJobsFromServer.length; ii++) {
		if (allJobsFromServer[ii].id == this.renderObject.id) {
			jobOk = true;
			if (allJobsFromServer[ii].machine != os.hostname()) {
				jobOk = false;
				logger.warn(`Render job has incorrect hostname: ${this.renderObject.id}`);
			}
			break;
		}
	}

	if (!jobOk) {
		this.abortJob();
	}
};

// Try to clean up the running instance when a job needs to be aborted.
this.abortJob = function () {
	logger.warn(`Aborting job: ${this.renderObject.id}`);
	this.finished = true;
	this.error = true;
	if(this.instance){
		this.instance.kill('SIGINT');
	}
	killLingeringAEProcesses();
};

// Kill all the ae processes that are still left behind after aborted jobs
// if we're not supposed to have any active renders on this server.
var killLingeringAEProcesses = function () {
	if(numActiveRenderJobs() == 0){
		exec("killall aerender");
		exec("killall aerendercore");
		exec("killall AdobeCRDaemon");
	}
};

The question is, is there more people with an interest in pursuing a good clean abort of a render job, and does any of you have recommandations as how to best approach it.

[inlife]

I've been thinking about this for a while now. So far, there are problems with identifying which process relates to which instance, but I hope there is a solution for that as well.

This issue also touches on the AME render engine handler idea that I've been thinking of for quite a while now. I think eventually, that could result in a conceptually new approach. But so far, those are still only ideas.

[olekristensen]

It could maybe be possible to keep track of the processes by actively tracking ps and do some bookkeeping, eg waiting a couple of seconds for aerender to spin up one or more aerendercore processes and tie them to the job. Then next time look for new aerendercore processes that are not already claimed by an earlier job, and add those to the new job. And so forth.

[olekristensen]

How would you approach using AME - would it be possible to inject scripted builds into that?
In our use case we build the full comp from script, even starting with an almost empty skeleton project, and letting the script import master aep projects. So our main benefit is that we can run jsx straight in the aerender pass. I haven't looked into refactoring to use nexrenders data assets and the NX object, but how do you see that apply to an AME render, would that not have to be a two-step process starting with aerender anyway?

[inlife]

The concept that I had in mind was to actually run the aerender and execute all of the JSX scripts, but then from there as a final step submit the job to the AME instead. So as a result AME would receive a fully finalized version of the project with all of the scripts executed. (Same as you would do via GUI)