Allow a custom task to run between prepare and publish steps of GitHub Pages plugin

[mojavelinux]

Currently, the GitHub Pages plugin copies files from a generated directory into the cloned repository (presumably after the task runs that generates those files). For large sites, this extra copy step can add measurable time to the build. This could be avoided if the task that generates the site would be allowed to generate directly into the cloned repository (after the prepareGhPages task wipes existing files away).

Consider allowing a custom task to be executed as an alternate to the pages CopySpec. To put it in concrete terms, consider a task such as JBake running after the doFirst of the prepareGhPages task and before the publishGhPages task.

[ajoberstar]

Do you have an example repo that I could use to verify any changes against? It would helpful for seeing a real world before and after improvement.

[mojavelinux]

This project is a good example:

https://github.com/msgilligan/msgilligan.github.io

Ideally, the jbake task would generate files directly into the working tree of the repository instead of into an intermediate folder followed by another copy operation. While the current arrangement is fine for small sites, for sites that exceed several hundred MB, this extra copy operation takes a significant amount of time and disk space.

This really comes down to an ordering issue. While the prepareGhPages can be made to depend on jbake, the jbake task runs before the doFirst block of the prepareGhPages task. What's needed is for the jbake task to run after the doFirst block (effectively replacing the native copy task in prepareGhPages).

Probably the simplest way to explain this is that we want to replace the copy task in prepareGhPages with a custom task.

[mojavelinux]

I think the problem is coupling doFirst with the CopySpec. I think the correct way to do this is to have these be separate tasks. That allows the Gradle configuration to reorder the operations by enforcing a task order.

Perhaps we need to following tasks:

• prepareGhPages
• copyGhPages
• commitGhPages
• publishGhPages

By default, these tasks would depend on each other in reverse order. However, a build master could modify the dependsOn to remove the copyGhPages task and replace it with something like compileGhPages (or any task). That way, the plugin doesn't need to worry about custom tasks. That's up to the build master.

commitGhPages.dependsOn = [bake]
jbake.dependsOn = [prepareGhPages]

Is this the right approach?

[ysb33r]

Woudn't it just work to have the following?

// Assuming a 'jbakeTask` exists

prepareGhPages.dependsOn jbakeTask

jbakeTask {
  // Assuming outputDir can handle lazy evaluation
  // (otherwise will always have to run jbakeTask configuration after 
  // prepareGhPages configuration)
  outputDir  {prepareGhPages.destinationDir}
}

This will hook jbakeTask to run before prepareGhPages. It will also ensure it copies to the same location.

[mojavelinux]

That's what I thought at first glance too. There are two complications:

1. We're moving the commit to the doLast of the prepareGhPages, so it will no longer work to have publishGhPages depend on a task before commit.
2. Even without that change, it's still necessary to suppress the built-in CopySpec. If it does no work, the git repository is never setup, and ext.repo never set, so the publishGhPages task fails.

That's why I think we need to separate these concerns so that we respect the task chain.

[mojavelinux]

See https://github.com/ajoberstar/gradle-git/blob/master/src/main/groovy/org/ajoberstar/gradle/git/ghpages/GithubPagesPlugin.groovy#L58-L81

In other words, the CopySpec isn't just coupled to the prepareGhPages task, the prepareGhPages task doesn't even run if there's nothing to copy (up-to-date).

[mojavelinux]

This is a sidebar, but I really think the gh-pages plugin should be a) moved to its own repository and b) renamed to git-publisher (since nothing about this task is specific to GitHub Pages, other than the default branch name). @ajoberstar should I open a separate issue for that?

[ysb33r]

Even without that change, it's still necessary to suppress the built-in CopySpec. If it does no work, the git repository is never setup, and ext.repo never set, so the publishGhPages task fails

That's true. Empty copySpec will cause task to be up to date.

[ysb33r]

It might be worthwhile to split some of the concerns out into distinctive tasks with the following relationships:

• prepareGhPages: Only checks out / updates from Git repo
• copyGhPages: Copy stuff that is in pages.
  • dependsOn prepareGhPages
• publishGhPages: Push to Git.
  • dependsOn prepareGhPages
  • mustRunAfter copyGhPages

Now you should be able to hook any task in between. If I return to my jbakeTask examples from earlier then

jbakeTask {
  outputDir  githubPages.workingDir
  dependsOn prepareGhPages
}

publishGhPages.dependsOn jbakeTask

[mojavelinux]

👍

The prepareGhPages task will have to always run, regardless of what the copy or build task does, since it has to go first (especially if we want to avoid the extra copy step). But since we now reuse the existing git clone, that shouldn't be too much of a problem.

[ajoberstar]

Sorry for the delayed response, it's good to see all of the discussion on this.

Ideally, we'd keep some backwards compatibility by having the prepareGhPages and publishGhPages still result in the same actions existing users are used to. In that model, we'd maybe break it as @ysb33r described but with different naming:

• refreshGhPages - fresh clone or fetch/reset the existing to get an initial clean state
• copyGhPages (dependsOn refreshGhPages) copy from the pages CopySpec.
• prepareGhPages - no-op task to represent all contents are ready in the repo. I still think this task should depend on copyGhPages, we just need something to ensure an empty copyspec doesn't cause an issue.
• commitGhPages - commit step, depends on prepareGhPages
• publishGhPages- push, depends oncommitGhPages`

Extra copying gets inserted after refresh and before prepare. One additional challenge this causes is #94. Seems to get harder to understand which parts of the repo would get cleaned out in this model.

Thoughts?

And yes, @mojavelinux I agree with splitting this into a separate repo. Added #203 for this.

[jnorthr]

Would really love to see another stand-alone repo for this, can think of some use cases to automate my docu toolchain 😊 +1 for Dan's idea ✅

[ajoberstar]

This will be addressed in gradle-git-publish (the new standalone plugin) instead of here.

[mojavelinux]

👍

[ajoberstar]

0.1.0 of gradle-git-publish implements this.

[jnorthr]

one question please:
in this code:
gitPublish {
// where to publish to
repoUri = '[email protected]/ajoberstar/test-repo.git'

does the target repo need to already exist or will it be built if it does not exist?
thx

[ajoberstar]

The target repo needs to exist, but the branch does not.