Skip to content

Commit

Permalink
TechnologyBrewery#146 copying transitive dependencies functionality f…
Browse files Browse the repository at this point in the history
…or containerize-dependencies goal
  • Loading branch information
Cho-William committed Jun 21, 2024
1 parent 86b7fb4 commit 78648ec
Show file tree
Hide file tree
Showing 25 changed files with 789 additions and 8 deletions.
47 changes: 43 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,44 @@ For example, developers may use this feature to bind a Habushu module's `compile
</plugin>
```

### Leveraging the containerize-dependencies Goal to Prepare a Containerized Virtual Environment ###
If the execution is specified (does not run by default), the `containerize-dependencies` goal will collect the monorepo python dependency specified in the dependencies block of the project's `pom.xml`, including all transitive dependencies. After collecting the set of necessary dependencies, Habushu will copy the source files to the build directory, while preserving the original structure of the dependency modules to ensure that any path-based dependencies can be leveraged as-is.

```xml
<plugin>
<groupId>org.technologybrewery.habushu</groupId>
<artifactId>habushu-maven-plugin</artifactId>
<extensions>true</extensions>
<configuration>
...
</configuration>
<executions>
<execution>
<id>containerize-deps</id>
<phase>validate</phase>
<goals>
<goal>containerize-dependencies</goal>
</goals>
</execution>
</executions>
</plugin>
```

The plugin will only examine dependencies that are of the type `habushu` in the dependencies block of the `pom.xml` file.

```xml
<dependencies>
<dependency>
<groupId>your-group-id</groupId>
<artifactId>your-artifact-id</artifactId>
<version>your-version</version>
<type>habushu</type>
</dependency>
</dependencies>
```

Any transitive monorepo dependency `pom.xml` files shoud also use this convention to ensure they are captured by the plugin.

### Leveraging Maven Build Cache for Faster Builds ###

Habushu enables support for faster builds via
Expand Down Expand Up @@ -682,13 +720,15 @@ Folder in which Python test files are located - should align with Poetry's proje

Default: `${project.basedir}/tests`

#### cacheWheels ####
#### cacheWheels (deprecated) ####
The `cache-wheels` goal has been `deprecated`, please see the `containerize-dependencies` goal instead.

Enables or Disables the copying of wheels into Poetry cache.

Default: `false`

#### wheelDependencies ####
#### wheelDependencies (deprecated) ####
The `retrieve-wheels` goal has been `deprecated`, please see the `containerize-dependencies` goal instead.

Optional set of wheel dependencies to retrieve from poetry cache. This allows previously cached external
wheel dependencies to be copied into a given target directory if it exists in poetry cache. This logic
Expand Down Expand Up @@ -876,8 +916,7 @@ Uses [behave](https://github.com/behave/behave) to execute BDD scenarios that ar
Builds the `sdist` and `wheel` archives of this project using `poetry build`. It also generates a `requirements.txt` file which is useful when installing the package in a Docker container where you may want to install the dependencies in a specific Docker layer to optimize caching.

##### install #####
Publishes the `pom.xml` for the module into your local Maven Repository (`~/.m2/repository`). If the **cacheWheels** configuration is set to True, the `wheel` archive will be copied to the poetry cache directory (`~/{poetry-cache-dir}/cache/repositories/wheels/{artifact-id}/`). The **cacheWheels** configuration default behavior is not to cache the `wheel` archive. If the **wheelDependencies** list is set, each specified wheel dependency will be
retrieve and placed into the given target directory.
Publishes the `pom.xml` for the module into your local Maven Repository (`~/.m2/repository`).

##### deploy #####

Expand Down
21 changes: 21 additions & 0 deletions habushu-maven-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,24 @@
<artifactId>cucumber-junit-platform-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-compat</artifactId>
<version>${version.maven}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.maven.plugin-testing</groupId>
<artifactId>maven-plugin-testing-harness</artifactId>
<version>4.0.0-alpha-2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-suite</artifactId>
Expand Down Expand Up @@ -191,6 +209,9 @@
cucumber.junit-platform.naming-strategy=long
</configurationParameters>
</properties>
<excludes>
<exclude>**/ContainerizeDependenciesMojoTestCase.java</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@
* wheel files in poetry.
*
* @throws HabushuException
* @deprecated This mojo is deprecated, use the containerize-dependencies goal instead
*/
@Deprecated
@Mojo(name = "cache-wheels", defaultPhase = LifecyclePhase.INSTALL)
public class CacheWheelsMojo extends AbstractHabushuMojo {
/**
Expand All @@ -31,6 +33,8 @@ public class CacheWheelsMojo extends AbstractHabushuMojo {

@Override
public void doExecute() throws MojoExecutionException, MojoFailureException {
getLog().warn("The cache-wheels goal has been deperecated," +
"please see the containerize-dependencies goal instead." );
if (cacheWheels) {
cacheWheels();
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
package org.technologybrewery.habushu;

import org.apache.maven.execution.MavenSession;
import org.apache.maven.model.Dependency;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugin.MojoFailureException;
import org.apache.maven.plugins.annotations.Component;
import org.apache.maven.plugins.annotations.LifecyclePhase;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.project.MavenProject;
import org.apache.maven.shared.model.fileset.FileSet;
import org.apache.maven.shared.model.fileset.util.FileSetManager;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;

/**
* Stages the source files of a monorepo dependency
* to the target directory along with the source files of
* any transitive path-based dependencies.
*/

@Mojo(name = "containerize-dependencies", defaultPhase = LifecyclePhase.VALIDATE)
public class ContainerizeDependenciesMojo extends AbstractHabushuMojo {

private static final Logger logger = LoggerFactory.getLogger(ContainerizeDependenciesMojo.class);

@Component
protected MavenSession session;

protected File anchorSourceDirectory;
protected File anchorOutputDirectory;
protected final String HABUSHU = "habushu";

/**
* Overriding to allow execution in non-habushu projects.
*/
@Override
public void execute() throws MojoExecutionException, MojoFailureException {
doExecute();
}

@Override
protected void doExecute() throws MojoExecutionException, MojoFailureException {
if (this.anchorSourceDirectory == null) {
this.anchorSourceDirectory = new File(session.getExecutionRootDirectory());
}

Set<MavenProject> habushuProjects = getHabushuProjects();
try {
copyRelevant(habushuProjects);
} catch (IOException e) {
throw new RuntimeException(e);
}
}

/**
* Copies the relevant source files by leveraging {@link FileSet}s to filter appropriately.
* @param habushuProjects corresponding projects of the pom's habushu-type dependencies
* @throws IOException
*/
private void copyRelevant(Set<MavenProject> habushuProjects) throws IOException {
this.anchorOutputDirectory = new File(project.getBuild().getDirectory()
+ "/venv-support/" + this.anchorSourceDirectory.getName());
Path srcRoot = this.anchorSourceDirectory.toPath();
Path destRoot = this.anchorOutputDirectory.toPath();

Map<Path, FileSet> dependencyFileSets = new HashMap<>();
for (MavenProject project : habushuProjects) {
Path projectPath = project.getFile().getParentFile().toPath();
Path relativeProjectPath = srcRoot.relativize(projectPath);
FileSet fileSet = getDefaultFileSet();
fileSet.setDirectory(projectPath.toString());
dependencyFileSets.put(relativeProjectPath, fileSet);
}

FileSetManager fileSetManager = new FileSetManager();
for (Path project : dependencyFileSets.keySet()) {
FileSet fileSet = dependencyFileSets.get(project);
logger.info("Copying {} files from {}",
fileSetManager.getIncludedFiles(fileSet).length,
project.getFileName()
);
for (String includedFile : fileSetManager.getIncludedFiles(fileSet)) {
Path relativePath = project.resolve(includedFile);
Files.createDirectories(destRoot.resolve(relativePath).getParent());
Files.copy(srcRoot.resolve(relativePath), destRoot.resolve(relativePath));
}
}
}

private FileSet getDefaultFileSet() {
FileSet fileSet = new FileSet();
fileSet.addExclude("target/**");
fileSet.addExclude(".venv/**");
fileSet.addExclude("dist/**");
return fileSet;
}

/**
* Checks listed habushu-type dependencies against the set of projects included in the Maven build's session
* @return the corresponding Maven projects that match the habushu-type dependencies
*/
protected Set<MavenProject> getHabushuProjects() {
HashSet<MavenProject> habushuProjects = new HashSet<>();
collectHabushuDependenciesAsProjects(project, habushuProjects);
return habushuProjects;
}

/**
* Collects the projects of the habushu-type dependencies and adds them to the given project set
* @param currentProject the project to interrogate the habushu-type dependencies against
* @param habushuProjects the set to append to with matching projects
*/
private void collectHabushuDependenciesAsProjects(MavenProject currentProject, Set<MavenProject> habushuProjects) {
Set<String> habushuDeps = currentProject.getDependencies().stream()
.filter(d -> HABUSHU.equals(d.getType()))
.map(ContainerizeDependenciesMojo::toGav)
.collect(Collectors.toSet());
for (MavenProject project : getSession().getProjects()) {
if (habushuDeps.contains(toGav(project))) {
logger.info("Found habushu-type dependency's project {}.", project);
habushuProjects.add(project);
collectHabushuDependenciesAsProjects(project, habushuProjects);
}
}
}

private static String toGav(Dependency dependency) {
return dependency.getGroupId() + ":" + dependency.getArtifactId() + ":" + dependency.getVersion();
}
private static String toGav(MavenProject project) {
return project.getGroupId() + ":" + project.getArtifactId() + ":" + project.getVersion();
}

protected MavenSession getSession() {
return this.session;
}

protected void setAnchorSourceDirectory(File anchorSourceDirectory) {
this.anchorSourceDirectory = anchorSourceDirectory;
}

public File getAnchorOutputDirectory() {
return anchorOutputDirectory;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@
* wheel artifacts cached by the {@param cacheWheels} parameter and REQUIRES
* the requested wheel to have first been cached prior to setting this config
* @throws HabushuException
* @deprecated This mojo is deprecated, use the containerize-dependencies goal instead
*/
@Deprecated
@Mojo(name = "retrieve-wheels", defaultPhase = LifecyclePhase.VALIDATE)
public class RetrieveWheelsMojo extends AbstractHabushuMojo {
/**
Expand All @@ -42,6 +44,8 @@ public void execute() throws MojoExecutionException, MojoFailureException {

@Override
public void doExecute() throws MojoExecutionException, MojoFailureException {
getLog().warn("The cache-wheels goal has been deperecated," +
"please see the containerize-dependencies goal instead." );
if (!wheelDependencies.isEmpty()) {
processWheelDependencies();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,6 @@
org.technologybrewery.habushu:habushu-maven-plugin:${project.version}:initialize-habushu,
org.technologybrewery.habushu:habushu-maven-plugin:${project.version}:baton-migrate
</initialize>
<validate>
org.technologybrewery.habushu:habushu-maven-plugin:${project.version}:retrieve-wheels
</validate>
<compile>org.technologybrewery.habushu:habushu-maven-plugin:${project.version}:install-dependencies</compile>
<process-classes>
org.technologybrewery.habushu:habushu-maven-plugin:${project.version}:format-python,
Expand All @@ -30,7 +27,6 @@
<test>org.technologybrewery.habushu:habushu-maven-plugin:${project.version}:behave-bdd-test</test>
<package>org.technologybrewery.habushu:habushu-maven-plugin:${project.version}:build-deployment-artifacts</package>
<install>
org.technologybrewery.habushu:habushu-maven-plugin:${project.version}:cache-wheels,
org.apache.maven.plugins:maven-install-plugin:install
</install>
<deploy>
Expand Down
Loading

0 comments on commit 78648ec

Please sign in to comment.