Blueprints: Directory Resources

[adamziel]

Description

Adds a Directory resource type to enable loading file trees from git repositories, local hard drive, deeply nested zip archives etc:

{
	"steps": [
		{
			"step": "installPlugin",
			"pluginData": {
				"resource": "git:directory",
				"repositoryUrl": "https://github.com/WordPress/wordpress-playground.git",
				"ref": "HEAD",
				"path": "packages/docs"
			}
		},
		{
			"step": "installPlugin",
			"pluginData": {
				"resource": "literal:directory",
				"name": "hello-world",
				"files": {
					"README.md": "Hello, World!",
					"index.php": "<?php\n/**\n* Plugin Name: Hello World\n* Description: A simple plugin that says hello world.\n*/",
				}
			}
		}
	]
}

Motivation

This PR opens the door to:

• Seamless Git integration. Import path mapping from git to Playground is now just a few steps referencing specific git directories. No more custom logic required!
• Blueprint-based site imports and exports without any Playground webapp-specific logic.
• Runtime-specific "resource overrides", e.g. --resource-override=GUTENBERG:./gutenberg.zip in CLI to test a Blueprint with a my local version of Gutenberg. The same logic would be used by the Blueprints builder to use files selected via <input type="file"> controls.

Schema

Every step can declare which kinds of resources it accepts (file-based resources vs directory-based resources). Using a single pluginData property in the installPlugin step means less choices for the developer. It also makes local resource overrides easy, e.g. we could tell Playground CLI to load a local Gutenberg directory instead of a remote Gutenberg zip. This wouldn't be as easy had we used separate options for passing ZIP-based and directory-based resources.

On one hand, pluginData is less informative than pluginZipFile. On the other, the name accommodates for non-zip resources such as directories.

Developer notes about specific API changes introduced in this PR

This PR changes introduces a new literal:directory resource that can be used in Blueprints as follows:

{
	"steps": [
		{
			"step": "installPlugin",
			"pluginData": {
				"resource": "literal:directory",
				"name": "hello-world",
				"files": {
					"README.md": "Hello, World!",
					"index.php": "<?php\n/**\n* Plugin Name: Hello World\n* Description: A simple plugin that says hello world.\n*/",
				}
			}
		}
	]
}

Or via the JS API:

await installTheme(php, {
	themeData: {
		name: 'test-theme',
		files: {
			'index.php': `/**\n * Theme Name: Test Theme`,
		},
	},
	ifAlreadyInstalled: 'overwrite',
	options: {
		activate: false,
	},
});

It also introduces a new writeFiles step:

{
	"steps": [
		{
			"step": "writeFiles",
			"writeToPath": "/wordpress/wp-content/plugins/my-plugin",
			"filesTree": {
				"name": "my-plugin",
				"files": {
					"index.php": "<?php echo '<a>Hello World!</a>'; ?>",
					"public": {
						"style.css": "a { color: red; }"
					}
				}
			}
		}
	]
}

Specific changes:

• Adds a Resource<Directory> resource type that provides a name: string and files: FileTree.
• Renames pluginZipFile to pluginData in the installPlugin step
• Renames themeZipFile to themeData in the installPlugin step
• Adds a new writeFiles step for writing entire directory trees
• Adds a new literal:directory resource type where an entire file tree can be specified inline
• Adds a new git:directory resource type that throws an error for now, but will load arbitrary directories from git repositories once Native git support: lsRefs(), sparseCheckout(), GitPathControl #1764 lands

Remaining work

• Discuss the scope and the ideas
• Add unit tests
• Update the documentation
• Adjust the installPlugin and installTheme step for compatibility with it's former signature. Ensure the existing packages consuming those functions from the @wp-playground/blueprints package will continue to work.
• Confirm we can safely omit streaming from the system design at this point without setting ourselves up for a grand refactor a few months down the road.
  • I think we can! Streaming support could be an addition to the system, not a change in how the system works. For example, there could be a new DirectoryStream resource type producing an AsyncDirectoryIterator with streamable File or Blob objects as its leafs. It would work nicely with remote APIs or the ZIP streaming plumbing in @php-wasm/stream-compression. Any existing code expecting a DirectoryResource should be relatively easily adaptable to use these async iterators instead.

Follow-up work

• Include actual git support once the Git sparse checkout PR lands
• Ship a Playground CORS proxy to enable using git checkout in the webapp
• Once we have a use-able git:directory resource, expand the developer notes from this PR and other related PRs and write a post on https://make.wp.org/playground

Tangent – Streaming and a shorthand URL notation

Without streaming, the entire directory must be loaded into memory. Our git sparse checkout implementation buffers everything anyway, but we will want to stream-read directory resources in the future. For example:

{
	"steps": [
		// Stream plugin files directly from a doubly zipped Git artifact
		{
			"step": "installPlugin",
			"pluginData": {
				"resource": "zip:github-artifact",
				"zipFile": {
					"resource": "url",
					"url": "https://github.com/WordPress/guteneberg/pr/54713/artifacts/build.zip"
				}
			}
		}
	}
}

That's extremely verbose, I'd love to explore a shorthand notation. One idea would be to make it a valid URI shaped after the data URL syntax:

const dataUri = `data:text/html;base64,%3Cscript%3Ealert%28%27hi%27%29%3B%3C%2Fscript%3E`;
const githubArtifactUri = `zip-github-artifact+url:https://github.com/WordPress/gutenberg/pr/54713/artifacts/build.zip`;
const gitResourceUri = `git:branch=HEAD;path=src,https://github.com/WordPress/hello-dolly.git`;

It wouldn't allow easy composition of the resources, e.g. a directory inside a zip sourced from a GitHub repo. Maybe that's for the best, though, since such a string would be extremely dense and difficult for humans to read. The object-based syntax might still be the most convenient way of to declare those.

[akirk]

I think the proposal is the right call to make as a step for evolving the plugin file source. I am not sure I am sold on the data-URL like syntax although I don't have a better proposal at this point.

Regarding backwards compatibility: I think at this point there are already a lot of blueprints in circulation that use pluginZipFile, so I believe we need to continue to support this legacy syntax.

[adamziel]

Regarding backwards compatibility: I think at this point there are already a lot of blueprints in circulation that use pluginZipFile, so I believe we need to continue to support this legacy syntax.

Absolutely

[adamziel]

Technically this PR is ready for merging – the build errors are related to #1766. Practically, I'll wait a bit more for feedback on the API shape.

[bgrgicak]

Great PR description!

[asirota]

Excellent improvement.