Composer-managed directories other than "vendor" get lost from workspace during a rebuild from cache

[effulgentsia]

Hi. I'm one of the maintainers of Drupal. I noticed a couple issues (#92 and #46) in this project's queue mentioning that installing Drupal via the Paketo buildpack doesn't work, and I'd like to help identify the reasons why. The following is one of the problems I ran into so far.

Expected Behavior

Drupal's recommended Composer project template uses Composer Installers to install some packages into the "web" directory, while also leaving some packages to be installed in the "vendor" directory.

I would expect both the "web" and "vendor" directories generated from running composer install to be in the final image.

Current Behavior

When running pack build with the --clear-cache option, I get both of those directories in the final image as expected.

When running pack build without the --clear-cache option, if I have already previously built the image then I only end up with the "vendor" directory in the final image. I do not end up with the "web" directory in the final image, and since the "web" directory is also the value of BP_PHP_WEB_DIR, attempting to docker run the final image errors out.

Possible Solution

I don't know if there's a reasonable way for the buildpack to automatically track all of the directories that were created or affected by a composer install execution, so that they can be copied back to the workspace from a cached layer. If there isn't, then perhaps another solution would be to provide an environment variable that could be set to a list of paths (e.g., so as to include both "vendor" and "web")? Or as a final fallback, would it make sense to add an environment variable or some other mechanism to disable caching of the composer install layer?

Steps to Reproduce

# Create just the composer.json and composer.lock files for a Drupal app.
composer create-project drupal/recommended-project my_app --no-install
cd my_app

# Not sure why these weren't auto-discovered from composer check-platform-reqs,
# but that's a separate issue.
mkdir .php.ini.d
echo "extension=gd.so" > .php.ini.d/extensions.ini
echo "extension=pdo.so" >> .php.ini.d/extensions.ini

# Initial build: the container starts up successfully and you can visit
# http://localhost:8000/ in the browser which takes you to Drupal's
# website installer. That installer doesn't work, but that's a separate issue.
pack build my_app_image --builder=paketobuildpacks/builder-jammy-full --env BP_PHP_SERVER=httpd --env BP_PHP_WEB_DIR=web --env BPE_DEFAULT_PORT=8080
docker run --rm -p 8000:8080 my_app_image

# Second build: the docker run command fails with the following message:
# AH00526: Syntax error on line 5 of /layers/paketo-buildpacks_php-httpd/php-httpd-config/httpd.conf:
# DocumentRoot '/workspace/web' is not a directory, or is not readable
# process httpd exited, status: exit status 1
# error running procs: exit status 1
pack build my_app_image --builder=paketobuildpacks/builder-jammy-full --env BP_PHP_SERVER=httpd --env BP_PHP_WEB_DIR=web --env BPE_DEFAULT_PORT=8080
docker run --rm -p 8000:8080 my_app_image

Motivations

Wanting to use Paketo builder for creating images of Drupal applications :)

[sophiewigmore]

Hey @effulgentsia thanks for looking into this and for the helpful explanation. You're right that the vendor directory is the only directory that we do caching for, so the case you're describing seems completely valid