diff --git a/README.md b/README.md index 682df7d..59574da 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,19 @@ # DataWeave CLI -**DataWeave CLI** is a command-line interface that allows `querying`, `filtering`, and `mapping` structured data from different data sources like `JSON`, `XML`, `CSV`, `YML` to other data formats. It also allows to easily create data in such formats. +**DataWeave CLI** is a command-line interface that allows `querying`, `filtering`, and `mapping` structured data from +different data sources like `JSON`, `XML`, `CSV`, `YML` to other data formats. It also allows to easily create data in +such formats. -For more info about the `DataWeave` language visit the [documenation site](https://docs.mulesoft.com/mule-runtime/latest/dataweave) +For more info about the `DataWeave` language visit +the [documenation site](https://docs.mulesoft.com/mule-runtime/latest/dataweave) ## What is Included? -The binary distribution already ships with a set of modules and data formats that makes this useful for a very interesting and broad set of use cases. + +The binary distribution already ships with a set of modules and data formats that makes this useful for a very +interesting and broad set of use cases. ### Included Modules + - [DataWeave Standard Library](https://github.com/mulesoft/data-weave/tree/master/wlang) ### Supported Data Formats @@ -25,7 +31,7 @@ The binary distribution already ships with a set of modules and data formats tha | `text/plain` | `text` | [Text Plain Format](https://docs.mulesoft.com/dataweave/latest/dataweave-formats-text) | | `text/x-java-properties` | `properties` | [Text Java Properties](https://docs.mulesoft.com/dataweave/latest/dataweave-formats-properties) | -## Installation +## Installation ### Homebrew (Mac) @@ -35,18 +41,22 @@ brew install dw ``` ### Manual Installation + 1. Download the latest [release version](https://github.com/mulesoft-labs/data-weave-cli/releases) according to your OS. 2. Unzip the file on your `/.dw` 3. Add `/.dw/bin` to your **PATH** ### Build and Install -To build the project, you need to run gradlew with the graalVM distribution based on Java 11. You can download it at https://github.com/graalvm/graalvm-ce-builds/releases +To build the project, you need to run gradlew with the graalVM distribution based on Java 11. You can download it +at https://github.com/graalvm/graalvm-ce-builds/releases Set: + ```bash export GRAALVM_HOME=/graalvm-ce-java11-21.2.0/Contents/Home export JAVA_HOME=/graalvm-ce-java11-21.2.0/Contents/Home ``` + Execute the gradle task `nativeCompile` ```bash @@ -59,11 +69,12 @@ Once it finishes you will find the `dw` binary in `native-cli/build/native/nativ ## How to Use It -If the directory containing the `dw` executable is in your _PATH_, you can run `dw` from anywhere. +If the directory containing the `dw` executable is in your _PATH_, you can run `dw` from anywhere. If it is not, go to the `bin` directory referenced in the installation instructions and run `dw` from there. -The following example shows the DataWeave CLI documentation +The following example shows the DataWeave CLI documentation + ```bash dw --help ``` @@ -141,6 +152,28 @@ The following are the DataWeave CLI environment variables that you can set in yo | `DW_DEFAULT_INPUT_MIMETYPE` | The default `mimeType` that is going to be used for the standard input. If not defined `application/json` will be used. | | `DW_DEFAULT_OUTPUT_MIMETYPE` | The default output `mimeType` that is going to be if not defined. If not defined `application/json` will be used. | +## Dependency Manager + +In order for a spell to depend on a library it can include a library it can use the dependencies.dwl to specify the list of dependencies that it should be included and download + +```data-weave +%dw 2.0 +var mavenRepositories = [{ + url: "https://maven.anypoint.mulesoft.com/api/v3/maven" +}] +--- +{ + dependencies: [ + { + kind: "maven", + artifactId: "data-weave-analytics-library", + groupId: "68ef9520-24e9-4cf2-b2f5-620025690913", + version: "1.0.1", + repositories: mavenRepositories // By default mulesoft, exchange and central are being added + } + ] +} +``` ### Querying Content From a File @@ -198,7 +231,6 @@ dw -i payload "output application/json --- payload filter cat | dw "output application/json --- payload filter (item) -> item.age > 17" ``` - ### Redirecting the Output to a File ```bash @@ -226,9 +258,11 @@ curl "https://api.github.com/repos/mulesoft/mule/commits?per_page=5" | dw "{mess ``` ### Generate a Request with Body + This example uses the [jsonplaceholder API](https://jsonplaceholder.typicode.com/) to update a resource. Steps: + 1. Search the post resource with the `id = 1`. 2. Use DataWeave CLI to create a JSON output changing the post title `My new title`. 3. Finally, update the post resource. @@ -238,6 +272,7 @@ curl https://jsonplaceholder.typicode.com/posts/1 | dw "output application/json ``` #### Output + ```json { "id": 1, @@ -258,4 +293,5 @@ Before creating a pull request review the following: * [SECURITY](SECURITY.md) * [CODE_OF_CONDUCT](CODE_OF_CONDUCT.md) -When you submit your pull request, you are asked to sign a contributor license agreement (CLA) if we don't have one on file for you. \ No newline at end of file +When you submit your pull request, you are asked to sign a contributor license agreement (CLA) if we don't have one on +file for you. \ No newline at end of file diff --git a/gradle.properties b/gradle.properties index b9cab2f..b3ab521 100644 --- a/gradle.properties +++ b/gradle.properties @@ -2,7 +2,7 @@ weaveVersion=2.5.0-SNAPSHOT nativeVersion=100.100.100 scalaVersion=2.12.11 ioVersion=1.0.0-SNAPSHOT -graalvmVersion=22.0.0.2 +graalvmVersion=22.2.0 #Libaries scalaTestVersion=3.0.1 scalaTestPluginVersion=0.32 diff --git a/native-cli-integration-tests/src/test/scala/org/mule/weave/native/NativeCliRuntimeIT.scala b/native-cli-integration-tests/src/test/scala/org/mule/weave/native/NativeCliRuntimeIT.scala index afac9ff..2e00125 100644 --- a/native-cli-integration-tests/src/test/scala/org/mule/weave/native/NativeCliRuntimeIT.scala +++ b/native-cli-integration-tests/src/test/scala/org/mule/weave/native/NativeCliRuntimeIT.scala @@ -363,6 +363,7 @@ class NativeCliRuntimeIT extends FunSpec "module-singleton", "multipart-write-binary", "read-binary-files", + "underflow", "try", "urlEncodeDecode") ++ // Uses resource name that is different on Cli than in the Tests diff --git a/native-cli/build.gradle b/native-cli/build.gradle index 1915c05..e55a776 100644 --- a/native-cli/build.gradle +++ b/native-cli/build.gradle @@ -3,7 +3,7 @@ plugins { id "com.github.maiflai.scalatest" version "${scalaTestPluginVersion}" id 'application' // Apply GraalVM Native Image plugin - id 'org.graalvm.buildtools.native' version '0.9.13' + id 'org.graalvm.buildtools.native' version '0.9.14' } sourceSets { @@ -19,6 +19,9 @@ mainClassName = 'org.mule.weave.dwnative.cli.DataWeaveCLI' dependencies { api group: 'org.mule.weave', name: 'runtime', version: weaveVersion compileOnly group: 'org.graalvm.sdk', name: 'graal-sdk', version: graalvmVersion + implementation group: 'io.get-coursier', name: 'coursier-core_2.12', version: '1.1.0-M14-7' + implementation group: 'io.get-coursier', name: 'coursier_2.12', version: '1.1.0-M14-7' + implementation group: 'io.get-coursier', name: 'coursier-cache_2.12', version: '1.1.0-M14-7' implementation group: 'org.mule.weave', name: 'core-modules', version: weaveVersion implementation group: 'org.mule.weave', name: 'yaml-module', version: weaveVersion implementation group: 'org.mule.weave', name: 'http-module', version: ioVersion @@ -116,6 +119,7 @@ graalvmNative { "org.asynchttpclient," + "org.mule.weave.v2.module.http.netty.HttpAsyncClientService," + "scala.util.Random," + + "coursier.," + "org.mule.weave.v2.sdk.SPIBasedModuleLoaderProvider\$") buildArgs.add("--initialize-at-build-time=" + "sun.instrument.InstrumentationImpl," + @@ -143,6 +147,7 @@ graalvmNative { // "org.mule.weave.v2.model.types.," // "org.mule.weave.v2.core.functions." // option "-H:+TraceClassInitialization" + buildArgs.add("--trace-class-initialization=coursier.core.Type\$") buildArgs.add("-H:DeadlockWatchdogInterval=1000") buildArgs.add("--report-unsupported-elements-at-runtime") buildArgs.add("-H:CompilationExpirationPeriod=0") diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/NativeRuntime.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/NativeRuntime.scala index 9c22bce..8a73633 100644 --- a/native-cli/src/main/scala/org/mule/weave/dwnative/NativeRuntime.scala +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/NativeRuntime.scala @@ -42,6 +42,7 @@ import org.mule.weave.v2.runtime.ModuleComponentsFactory import org.mule.weave.v2.runtime.ParserConfiguration import org.mule.weave.v2.runtime.ScriptingBindings import org.mule.weave.v2.runtime.ScriptingEngineSetupException +import org.mule.weave.v2.sdk.NameIdentifierHelper import org.mule.weave.v2.sdk.SPIBasedModuleLoaderProvider import org.mule.weave.v2.sdk.TwoLevelWeaveResourceResolver import org.mule.weave.v2.sdk.WeaveResourceResolver @@ -58,7 +59,7 @@ class NativeRuntime(libDir: File, path: Array[File], console: Console) { private val dataWeaveUtils = new DataWeaveUtils(console) private val pathBasedResourceResolver: PathBasedResourceResolver = PathBasedResourceResolver(path ++ Option(libDir.listFiles()).getOrElse(new Array[File](0))) - + private val weaveScriptingEngine: DataWeaveScriptingEngine = { setupEnv() DataWeaveScriptingEngine(new NativeModuleComponentFactory(() => pathBasedResourceResolver, systemFirst = true), ParserConfiguration()) @@ -68,6 +69,10 @@ class NativeRuntime(libDir: File, path: Array[File], console: Console) { weaveScriptingEngine.enableProfileParsing() } + def addJarToClassPath(file: File): Unit = { + pathBasedResourceResolver.addContent(ContentResolver(file)) + } + /** * Setup initialization properties */ @@ -103,9 +108,9 @@ class NativeRuntime(libDir: File, path: Array[File], console: Console) { private def compileScript(script: String, inputs: ScriptingBindings, nameIdentifier: NameIdentifier, defaultOutputMimeType: String) = { weaveScriptingEngine.compile(script, nameIdentifier, inputs.entries().map(wi => new InputType(wi, None)).toArray, defaultOutputMimeType) } - + private def createServiceManager(maybePrivileges: Option[Seq[String]] = None): ServiceManager = { - + val charsetProviderService = new CharsetProviderService { override def defaultCharset(): Charset = { StandardCharsets.UTF_8 @@ -120,7 +125,7 @@ class NativeRuntime(libDir: File, path: Array[File], console: Console) { if (maybePrivileges.isDefined) { val privileges = maybePrivileges.get val weaveRuntimePrivileges = privileges.map(WeaveRuntimePrivilege(_)).toArray - customServices = customServices + (classOf[SecurityManagerService] -> new DefaultSecurityManagerService(weaveRuntimePrivileges)) + customServices = customServices + (classOf[SecurityManagerService] -> new DefaultSecurityManagerService(weaveRuntimePrivileges)) } ServiceManager(new ConsoleLogger(console), customServices) } @@ -143,12 +148,7 @@ class WeavePathProtocolHandler(path: PathBasedResourceResolver) extends ReadFunc override def createSourceProvider(url: String, locatable: LocationCapable, charset: Charset): SourceProvider = { val uri = url.stripPrefix(CLASSPATH_PREFIX) - val wellFormedUri = if (uri.startsWith("/")) { - uri.substring(1) - } else { - uri - } - val maybeResource = path.resolve(wellFormedUri) + val maybeResource = path.resolve(uri) maybeResource match { case Some(value) => { SourceProvider(value, charset) diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/PathBasedResourceResolver.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/PathBasedResourceResolver.scala index 7d5a06c..a5b8af3 100644 --- a/native-cli/src/main/scala/org/mule/weave/dwnative/PathBasedResourceResolver.scala +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/PathBasedResourceResolver.scala @@ -22,11 +22,12 @@ class PathBasedResourceResolver(paths: mutable.ArrayBuffer[ContentResolver]) ext } override def resolve(name: NameIdentifier): Option[WeaveResource] = { - val filePath = NameIdentifierHelper.toWeaveFilePath(name) + val iterator = paths.iterator while (iterator.hasNext) { - val maybeResource = iterator.next().resolve(filePath) + val maybeResource: Option[InputStream] = iterator.next().resolve(name) if (maybeResource.isDefined) { + val filePath = NameIdentifierHelper.toWeaveFilePath(name, "/") //Use unix based system return Some(WeaveResource(filePath, toString(maybeResource.get))) } } @@ -43,9 +44,10 @@ class PathBasedResourceResolver(paths: mutable.ArrayBuffer[ContentResolver]) ext } def resolve(filePath: String): Option[InputStream] = { + val ni = NameIdentifierHelper.fromWeaveFilePath(filePath, "/") val iterator = paths.iterator while (iterator.hasNext) { - val maybeResource = iterator.next().resolve(filePath) + val maybeResource = iterator.next().resolve(ni) if (maybeResource.isDefined) { return maybeResource } @@ -55,10 +57,12 @@ class PathBasedResourceResolver(paths: mutable.ArrayBuffer[ContentResolver]) ext override def resolveAll(name: NameIdentifier): Seq[WeaveResource] = { - val filePath = NameIdentifierHelper.toWeaveFilePath(name) paths - .flatMap(_.resolve(filePath)) - .map((content) => WeaveResource(filePath, toString(content))) + .flatMap(_.resolve(name)) + .map((content) => { + val path = NameIdentifierHelper.toWeaveFilePath(name, "/") + WeaveResource(path, toString(content)) + }) } } @@ -66,18 +70,7 @@ class PathBasedResourceResolver(paths: mutable.ArrayBuffer[ContentResolver]) ext * */ trait ContentResolver { - def resolve(path: String): Option[InputStream] -} - -class CompositeContentResolver(contents: Seq[ContentResolver]) extends ContentResolver { - override def resolve(path: String): Option[InputStream] = { - contents - .toStream - .flatMap((content) => { - content.resolve(path) - }) - .headOption - } + def resolve(path: NameIdentifier): Option[InputStream] } @@ -93,7 +86,8 @@ object ContentResolver { class DirectoryContentResolver(directory: File) extends ContentResolver { - override def resolve(path: String): Option[InputStream] = { + override def resolve(ni: NameIdentifier): Option[InputStream] = { + val path = NameIdentifierHelper.toWeaveFilePath(ni, File.separator) //Use unix based system val file = new File(directory, path) if (file.isFile) { Some(new FileInputStream(file)) @@ -107,12 +101,15 @@ class JarContentResolver(jarFile: => File) extends ContentResolver { lazy val zipFile = new ZipFile(jarFile) - override def resolve(path: String): Option[InputStream] = { - val zipEntry = if (path.startsWith("/")) { - path.substring(1) - } else { - path - } + override def resolve(ni: NameIdentifier): Option[InputStream] = { + val path = NameIdentifierHelper.toWeaveFilePath(ni, "/") //Use unix based system + + val zipEntry: String = + if (path.startsWith("/")) { + path.substring(1) + } else { + path + } val pathEntry = zipFile.getEntry(zipEntry) if (pathEntry != null) { Some(zipFile.getInputStream(pathEntry)) diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/cli/CLIArgumentsParser.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/cli/CLIArgumentsParser.scala index 781137b..b5b8cdd 100644 --- a/native-cli/src/main/scala/org/mule/weave/dwnative/cli/CLIArgumentsParser.scala +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/cli/CLIArgumentsParser.scala @@ -16,6 +16,8 @@ import org.mule.weave.dwnative.cli.commands.WeaveCommand import org.mule.weave.dwnative.cli.commands.WeaveModule import org.mule.weave.dwnative.cli.commands.WeaveRunnerConfig import org.mule.weave.dwnative.cli.utils.SpellsUtils +import org.mule.weave.dwnative.dependencies.DependencyResolutionResult +import org.mule.weave.dwnative.dependencies.SpellDependencyManager import org.mule.weave.v2.io.FileHelper import org.mule.weave.v2.parser.ast.variables.NameIdentifier import org.mule.weave.v2.runtime.utils.AnsiColor.red @@ -38,6 +40,8 @@ class CLIArgumentsParser(console: Console) { var eval: Boolean = false var maybePrivileges: Option[Seq[String]] = None + var dependencyResolver: Option[(NativeRuntime) => Array[DependencyResolutionResult]] = None + val inputs: mutable.Map[String, File] = mutable.Map() val params: mutable.Map[String, String] = mutable.Map() @@ -109,7 +113,7 @@ class CLIArgumentsParser(console: Console) { } else { null } - var spellName = if (spell.contains("/")) { + var spellName: String = if (spell.contains("/")) { spell.split("/")(1) } else { spell @@ -140,11 +144,14 @@ class CLIArgumentsParser(console: Console) { return Right(s"Unable to get Wise `$wizardName's` Grimoire.") } - val spellFolder = new File(wizardGrimoire, spellName) + val spellFolder: File = new File(wizardGrimoire, spellName) if (!spellFolder.exists()) { new UpdateGrimoireCommand(UpdateGrimoireConfig(wizardGrimoire), console).exec() } + val manager = new SpellDependencyManager(spellFolder, console) + dependencyResolver = Some(manager.resolveDependencies) + if (!spellFolder.exists()) { return Right(s"Unable find $spellName in Wise `$wizardName's` Grimoire.") } @@ -184,6 +191,10 @@ class CLIArgumentsParser(console: Console) { if (!spellFolder.exists()) { return Right(s"Unable find `$spellName` folder.") } + + val manager = new SpellDependencyManager(spellFolder, console) + dependencyResolver = Some(manager.resolveDependencies) + val srcFolder = new File(spellFolder, "src") val mainFile = new File(srcFolder, fileName) @@ -278,7 +289,8 @@ class CLIArgumentsParser(console: Console) { if (scriptToRun.isEmpty) { Right(s"Missing or -f or --spell ") } else { - val config: WeaveRunnerConfig = WeaveRunnerConfig(paths, eval, scriptToRun.get, params.toMap, inputs.toMap, output, maybePrivileges) + + val config: WeaveRunnerConfig = WeaveRunnerConfig(paths, eval, scriptToRun.get, dependencyResolver ,params.toMap, inputs.toMap, output, maybePrivileges) Left(new RunWeaveCommand(config, console)) } } diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/cli/commands/RunWeaveCommand.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/cli/commands/RunWeaveCommand.scala index 463ff18..fcba1bd 100644 --- a/native-cli/src/main/scala/org/mule/weave/dwnative/cli/commands/RunWeaveCommand.scala +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/cli/commands/RunWeaveCommand.scala @@ -3,6 +3,8 @@ package org.mule.weave.dwnative.cli.commands import org.mule.weave.dwnative.NativeRuntime import org.mule.weave.dwnative.WeaveExecutionResult import org.mule.weave.dwnative.cli.Console +import org.mule.weave.dwnative.dependencies.ResolutionErrorHandler +import org.mule.weave.dwnative.dependencies.DependencyResolutionResult import org.mule.weave.dwnative.utils.DataWeaveUtils import org.mule.weave.dwnative.utils.DataWeaveUtils.DW_DEFAULT_INPUT_MIMETYPE_VAR import org.mule.weave.dwnative.utils.DataWeaveUtils.DW_DEFAULT_OUTPUT_MIMETYPE_VAR @@ -28,14 +30,41 @@ class RunWeaveCommand(val config: WeaveRunnerConfig, console: Console) extends W val weaveUtils = new DataWeaveUtils(console) private val DEFAULT_MIME_TYPE: String = "application/json" - + @volatile private var keepRunning = true def exec(): Int = { + var exitCode = ExitCodes.SUCCESS val path: Array[File] = config.path.map(new File(_)) val nativeRuntime: NativeRuntime = new NativeRuntime(weaveUtils.getLibPathHome(), path, console) - doRun(config, nativeRuntime) + config.dependencyResolver.foreach((dep) => { + val results = dep(nativeRuntime) + results.foreach((dm) => { + val resolvedDependencies = dm.resolve( + new ResolutionErrorHandler { + override def onError(id: String, message: String): Unit = { + console.error(s"Unable to resolve: `${id}`. Reason: ${message}") + exitCode = ExitCodes.FAILURE + } + } + ) + if (resolvedDependencies.isEmpty) { + console.error(s"${dm.id} didn't resolve to any artifact.") + exitCode = ExitCodes.FAILURE + } else { + + resolvedDependencies.foreach((a) => { + nativeRuntime.addJarToClassPath(a) + }) + } + }) + }) + if (exitCode == ExitCodes.SUCCESS) { + doRun(config, nativeRuntime) + } else { + exitCode + } } @@ -62,13 +91,13 @@ class RunWeaveCommand(val config: WeaveRunnerConfig, console: Console) extends W }) } - val value = config.params.toSeq.map( prop => + val value = config.params.toSeq.map(prop => KeyValuePair(KeyValue(prop._1), StringValue(prop._2)) ).to - + val params = ObjectValue(value) scriptingBindings.addBinding("params", params) - + val module: WeaveModule = config.scriptToRun(nativeRuntime) if (config.eval) { keepRunning = true @@ -136,9 +165,11 @@ class RunWeaveCommand(val config: WeaveRunnerConfig, console: Console) extends W case class WeaveRunnerConfig(path: Array[String], eval: Boolean, scriptToRun: NativeRuntime => WeaveModule, + dependencyResolver: Option[(NativeRuntime) => Array[DependencyResolutionResult]], params: Map[String, String], inputs: Map[String, File], outputPath: Option[String], maybePrivileges: Option[Seq[String]]) + case class WeaveModule(content: String, nameIdentifier: String) \ No newline at end of file diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/DependencyManager.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/DependencyManager.scala new file mode 100644 index 0000000..4f6691b --- /dev/null +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/DependencyManager.scala @@ -0,0 +1,65 @@ +package org.mule.weave.dwnative.dependencies + +import java.io.File +import scala.concurrent.Await +import scala.concurrent.Future +import scala.concurrent.duration.Duration + +/** + * Dependency manager will resolve an artifact with from a given ID. + */ +trait DependencyManager { + /** + * Returns true if this dependency manager support the given model + * + * @param dep the dependency + * @return True if it is supported else false + */ + def support(dep: DependencyModel): Boolean + + /** + * Retrieves the given artifact + * + * @param artifactId The artifactId to retrieve + * @param controller The callback to be used to resolving an artifact + * @param messageCollector This message collector + */ + def retrieve(artifactId: DependencyModel): DependencyResolutionResult + + /** + * Returns the kind of dependencies it handles i.e maven + * + * @return The kind of dependencies it handles + */ + def kind: String +} + +case class DependencyResolutionResult(id: String, kind: String, artifact: Future[Seq[File]]) { + def resolve(errorHandler: ResolutionErrorHandler): Array[File] = { + try { + val files = Await.result(artifact, Duration.Inf) + files.toArray + } catch { + case e: Exception => { + errorHandler.onError(s"${id}", e.getMessage) + Array.empty + } + } + } +} + + +/** + * Callback that handles all messages when resolving an artifact + */ +trait ResolutionErrorHandler { + + /** + * When an error occurred when trying to resolve an artifact id + * + * @param id The id of the artifact + * @param message The error message + */ + def onError(id: String, message: String): Unit +} + diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/DependencyModel.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/DependencyModel.scala new file mode 100644 index 0000000..d985bf7 --- /dev/null +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/DependencyModel.scala @@ -0,0 +1,12 @@ +package org.mule.weave.dwnative.dependencies + +sealed trait DependencyModel + +case class MavenDependencyModel(artifactId: String, groupId: String, version: String, repository: Array[MavenRepositoryModel]) extends DependencyModel { + def fullArtifactID(): String = s"${artifactId}:${groupId}:${version}" + +} + +case class MavenRepositoryModel(url: String, credentials: Option[MavenCredentialsModel]) + +case class MavenCredentialsModel(username: String, password: String) diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/MavenDependencyManager.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/MavenDependencyManager.scala new file mode 100644 index 0000000..3c75be5 --- /dev/null +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/MavenDependencyManager.scala @@ -0,0 +1,85 @@ +package org.mule.weave.dwnative.dependencies + +import coursier.Dependency +import coursier.LocalRepositories.Dangerous +import coursier.MavenRepository +import coursier.Module +import coursier.ModuleName +import coursier.Organization +import coursier.Repositories +import coursier.Resolution +import coursier.ResolutionProcess +import coursier._ +import coursier.cache.Cache +import coursier.cache.CacheLogger +import coursier.cache.FileCache +import coursier.util.Gather +import coursier.util.Task +import org.mule.weave.dwnative.cli.Console + +import java.io.File +import java.util.concurrent.ExecutorService +import scala.concurrent.ExecutionContext +import scala.concurrent.ExecutionContextExecutor +import scala.concurrent.Future +import scala.concurrent.duration._ + +class MavenDependencyManager( + console: Console, + cacheDirectory: File, + executor: ExecutorService, //The executor service to be used for resolving the dependencies in parallel + ) extends DependencyManager { + + + val context: ExecutionContextExecutor = ExecutionContext.fromExecutor(executor) + + override def retrieve(dep: DependencyModel): DependencyResolutionResult = { + + + val cache: Cache[Task] = FileCache() + .withLocation(cacheDirectory) + .withLogger(new CacheLogger { + override def downloadingArtifact(url: String): Unit = { + console.info(s"Downloading: `${url}`.") + } + }) + .withPool(executor) + .withTtl(1.day) + .noCredentials + + val mdm = dep match { + case mdm: MavenDependencyModel => { + mdm + } + case _ => throw new RuntimeException(s"Unsupported dependency model `${dep.getClass}`.") + } + + val repositories: Seq[MavenRepository] = if (mdm.repository.isEmpty) { + Seq( + MavenRepository("https://maven.anypoint.mulesoft.com/api/v3/maven"), + MavenRepository("https://repository.mulesoft.org/nexus/content/repositories/releases/"), + MavenRepository("https://repository.mulesoft.org/nexus/content/repositories/snapshots/", changing = Some(true)), + Repositories.central) + } else { + mdm.repository.map((repository) => { + MavenRepository(repository.url) + }).toSeq + } + + val moduleName = Module(Organization(mdm.groupId), ModuleName(mdm.artifactId)) + val files: Future[Seq[File]] = Fetch(cache) + .addRepositories(repositories: _*) + .addDependencies(Dependency(moduleName, mdm.version, attributes = Attributes(classifier = Classifier("dw-library")))) + .future() + + DependencyResolutionResult(mdm.fullArtifactID(), kind, files) + } + + override def kind: String = { + "maven" + } + + override def support(dep: DependencyModel): Boolean = dep.isInstanceOf[MavenDependencyModel] +} + + diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/SpellDependencyManager.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/SpellDependencyManager.scala new file mode 100644 index 0000000..1ec025f --- /dev/null +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/dependencies/SpellDependencyManager.scala @@ -0,0 +1,142 @@ +package org.mule.weave.dwnative.dependencies + +import org.mule.weave.dwnative.NativeRuntime +import org.mule.weave.dwnative.cli.Console +import org.mule.weave.dwnative.utils.DataWeaveUtils +import org.mule.weave.v2.runtime.ArrayDataWeaveValue +import org.mule.weave.v2.runtime.DataWeaveValue +import org.mule.weave.v2.runtime.ObjectDataWeaveValue +import org.mule.weave.v2.runtime.ScriptingBindings +import org.mule.weave.v2.runtime.SimpleDataWeaveValue + +import java.io.File +import java.io.FileInputStream +import java.util.concurrent.Executors +import scala.io.Source + +class SpellDependencyManager(projectHome: File, console: Console) { + + private val CACHE_FOLDER: File = new DataWeaveUtils(console).getCacheHome() + private val DEPENDENCY_MANAGER = Array(new MavenDependencyManager(console, CACHE_FOLDER, Executors.newCachedThreadPool())) + + def asRepository(v: DataWeaveValue): MavenRepositoryModel = { + v match { + case value: ObjectDataWeaveValue => { + val url = selectRequiredStringValue(value, "url") + MavenRepositoryModel(url, None) + } + case _ => throw new RuntimeException(s"Expecting `Object` but got `${v.typeName()}`") + } + } + + def asDependency(value: ObjectDataWeaveValue): DependencyModel = { + val kind = selectRequiredStringValue(value, "kind") + kind match { + case s if s equalsIgnoreCase "maven" => { + val artifactId = selectRequiredStringValue(value, "artifactId") + val groupId = selectRequiredStringValue(value, "groupId") + val version = selectRequiredStringValue(value, "version") + val repositories = selectArrayValue(value, "repositories") + val repositoriesDef: Array[MavenRepositoryModel] = repositories.map((v) => v.map((v) => asRepository(v))).getOrElse(Array()) + MavenDependencyModel(artifactId, groupId, version, repositoriesDef) + } + case _ => throw new RuntimeException(s"Invalid kind field `${kind}`.") + } + } + + def resolveDep(dep: DependencyModel, console: Console): DependencyResolutionResult = { + DEPENDENCY_MANAGER + .find((dm) => { + dm.support(dep) + }) + .map((dm) => { + dm.retrieve(dep) + }) + .getOrElse(throw new RuntimeException(s"Unable to find support for: `${dep.getClass}`.")) + } + + def resolveDependencies(nr: NativeRuntime): Array[DependencyResolutionResult] = { + val dependenciesFile = new File(projectHome, "dependencies.dwl") + if (dependenciesFile.exists()) { + val dependencies: Array[_ <: DependencyModel] = collectDependencies(dependenciesFile, nr) + dependencies + .map((dep) => { + resolveDep(dep, console) + }) + } else { + Array.empty + } + } + + def collectDependencies(file: File, runtime: NativeRuntime): Array[_ <: DependencyModel] = { + val source = Source.fromInputStream(new FileInputStream(file), "UTF-8") + try { + val scriptContent = source.mkString + val executeResult = runtime.eval(scriptContent, ScriptingBindings(), file.getName, None) + executeResult.asDWValue() match { + case value: ObjectDataWeaveValue => { + selectValue(value, "dependencies") + .map { + case value: ArrayDataWeaveValue => { + value.elements().map { + case value: ObjectDataWeaveValue => { + asDependency(value) + } + case _ => throw new RuntimeException("Expecting dependency") + } + } + case value: ObjectDataWeaveValue => { + Array(asDependency(value)) + } + case _ => { + throw new RuntimeException("Expecting Array of Dependencies") + } + }.getOrElse(Array()) + } + case value: SimpleDataWeaveValue if (value.value() == null) => { + Array() + } + case _ => { + throw new RuntimeException("Invalid dependencies.dwl data structure.") + } + + } + } finally { + source.close() + } + } + + private def selectValue(value: ObjectDataWeaveValue, fieldName: String) = { + value.entries() + .find((dw) => { + dw.name.name.equals(fieldName) + }).map(_.value) + } + + private def selectArrayValue(value: ObjectDataWeaveValue, fieldName: String) = { + selectValue(value, fieldName).map { + case value: ArrayDataWeaveValue => { + value.elements() + } + case v => throw new RuntimeException(s"Expecting `${fieldName}` to be `Array` but was `${v.typeName()}``") + } + } + + private def selectRequiredArrayValue(value: ObjectDataWeaveValue, fieldName: String): Array[DataWeaveValue] = { + selectArrayValue(value, fieldName).getOrElse(throw new RuntimeException(s"Missing required `Array` field `${fieldName}`")) + } + + private def selectStringValue(value: ObjectDataWeaveValue, fieldName: String): Option[String] = { + selectValue(value, fieldName).map { + case value: SimpleDataWeaveValue => { + value.value().toString + } + case v => throw new RuntimeException(s"Expecting `${fieldName}` to be `String` but was `${v.typeName()}`") + } + } + + private def selectRequiredStringValue(value: ObjectDataWeaveValue, fieldName: String) = { + selectStringValue(value, fieldName) + .getOrElse(throw new RuntimeException(s"Missing required `String` field `${fieldName}``")) + } +} diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/utils/DataWeaveUtils.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/utils/DataWeaveUtils.scala index d09a419..4941a68 100644 --- a/native-cli/src/main/scala/org/mule/weave/dwnative/utils/DataWeaveUtils.scala +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/utils/DataWeaveUtils.scala @@ -29,7 +29,6 @@ class DataWeaveUtils(console: Console) { } home } else { - console.debug("Env not working trying home directory") val defaultDWHomeDir: File = getDefaultDWHome() if (defaultDWHomeDir.exists()) { @@ -70,7 +69,7 @@ class DataWeaveUtils(console: Console) { if (weavehome.isDefined) { val home = new File(weavehome.get) if (!home.exists()) { - console.envVar(s"Weave Working Home Directory `${weavehome}` declared on environment variable `$DW_WORKING_DIRECTORY_VAR` does not exists.") + console.error(s"Weave Working Home Directory `${weavehome}` declared on environment variable `$DW_WORKING_DIRECTORY_VAR` does not exists.") } home } else { @@ -92,7 +91,7 @@ class DataWeaveUtils(console: Console) { if (weavehome.isDefined) { val home = new File(weavehome.get) if (!home.exists()) { - console.envVar(s"Weave Library Home Directory `${weavehome}` declared on environment variable `$DW_LIB_PATH_VAR` does not exists.") + console.error(s"Weave Library Home Directory `${weavehome}` declared on environment variable `$DW_LIB_PATH_VAR` does not exists.") } home } else { @@ -100,5 +99,7 @@ class DataWeaveUtils(console: Console) { } } - def sanitizeFilename(inputName: String): String = inputName.replaceAll("[^a-zA-Z0-9-_.]", "_") + def getCacheHome(): File = { + new File(getDWHome(), "cache") + } } \ No newline at end of file diff --git a/native-cli/src/test/resources/spells/SimpleSpellWithDependencies/dependencies.dwl b/native-cli/src/test/resources/spells/SimpleSpellWithDependencies/dependencies.dwl new file mode 100644 index 0000000..d16240a --- /dev/null +++ b/native-cli/src/test/resources/spells/SimpleSpellWithDependencies/dependencies.dwl @@ -0,0 +1,16 @@ +%dw 2.0 +var mavenRepositories = [{ + url: "https://maven.anypoint.mulesoft.com/api/v3/maven" +}] +--- +{ + dependencies: [ + { + kind: "maven", + artifactId: "data-weave-analytics-library", + groupId: "68ef9520-24e9-4cf2-b2f5-620025690913", + version: "1.0.1", + repositories: mavenRepositories + } + ] +} \ No newline at end of file diff --git a/native-cli/src/test/resources/spells/SimpleSpellWithDependencies/src/Main.dwl b/native-cli/src/test/resources/spells/SimpleSpellWithDependencies/src/Main.dwl new file mode 100644 index 0000000..0930c56 --- /dev/null +++ b/native-cli/src/test/resources/spells/SimpleSpellWithDependencies/src/Main.dwl @@ -0,0 +1,6 @@ +%dw 2.0 +output application/json + +import median from analytics::Statistics +--- +median([3, 1, 4]) \ No newline at end of file diff --git a/native-cli/src/test/scala/org/mule/weave/dwnative/cli/DataWeaveCLITest.scala b/native-cli/src/test/scala/org/mule/weave/dwnative/cli/DataWeaveCLITest.scala index c3c8cdf..ad2764b 100644 --- a/native-cli/src/test/scala/org/mule/weave/dwnative/cli/DataWeaveCLITest.scala +++ b/native-cli/src/test/scala/org/mule/weave/dwnative/cli/DataWeaveCLITest.scala @@ -8,6 +8,7 @@ import org.scalatest.Matchers import java.io.ByteArrayInputStream import java.io.ByteArrayOutputStream import java.io.File +import java.nio.file.Files import scala.io.Source class DataWeaveCLITest extends FreeSpec with Matchers { @@ -23,7 +24,6 @@ class DataWeaveCLITest extends FreeSpec with Matchers { "should take into account the env variable for default output" in { val console = new TestConsole(System.in, System.out, Map()) new DataWeaveCLIRunner().run(Array("--list-spells"), console) - console.fatalMessages.isEmpty shouldBe true } @@ -58,6 +58,27 @@ class DataWeaveCLITest extends FreeSpec with Matchers { result.trim shouldBe "\"DW Rules\"" } + "should be able to run a local spell with a dependency" in { + val stream = new ByteArrayOutputStream() + val localSpell: File = TestUtils.getSimpleSpellWithDependencies + val console = new TestConsole(System.in, stream) + val exitCode = new DataWeaveCLIRunner().run(Array("--local-spell", localSpell.getAbsolutePath), console) + console.infoMessages.foreach((m) => { + println(s"[INFO] ${m}") + }) + console.errorMessages.foreach((m) => { + println(s"[ERROR] ${m}") + }) + + console.fatalMessages.foreach((m) => { + println(s"[FATAL] ${m}") + }) + exitCode shouldBe 0 + val source = Source.fromBytes(stream.toByteArray, "UTF-8") + val result: String = source.mkString + result.trim shouldBe "3" + } + "should work with simple script and not output" in { val stream = new ByteArrayOutputStream() new DataWeaveCLIRunner().run(Array("(1 to 3)[0]"), new TestConsole(System.in, stream)) @@ -124,10 +145,11 @@ class DataWeaveCLITest extends FreeSpec with Matchers { val source = Source.fromBytes(stream.toByteArray, "UTF-8") val result = source.mkString.trim source.close() - val expected = """ - |{ - | "isEmpty": false - |}""".stripMarginAndNormalizeEOL.trim + val expected = + """ + |{ + | "isEmpty": false + |}""".stripMarginAndNormalizeEOL.trim result shouldBe expected } @@ -140,13 +162,14 @@ class DataWeaveCLITest extends FreeSpec with Matchers { val source = Source.fromBytes(stream.toByteArray, "UTF-8") val result = source.mkString.trim source.close() - val expected = """ - |{ - | "isEmpty": false - |}""".stripMarginAndNormalizeEOL.trim + val expected = + """ + |{ + | "isEmpty": false + |}""".stripMarginAndNormalizeEOL.trim result shouldBe expected } - + "should fail running a script with requires privileges in untrusted mode" in { val stream = new ByteArrayOutputStream() val script = """import props from dw::Runtime output application/json --- {isEmpty: isEmpty(props())}""".stripMargin @@ -173,15 +196,16 @@ class DataWeaveCLITest extends FreeSpec with Matchers { new DataWeaveCLIRunner().run(Array( "-p", "name", "Mariano", "-p", "lastname", "Lischetti", - "{fullName: params.name ++ \" \" ++ params.lastname}"), + "{fullName: params.name ++ \" \" ++ params.lastname}"), new TestConsole(System.in, stream)) val source = Source.fromBytes(stream.toByteArray, "UTF-8") val result = source.mkString.trim source.close() - val expected = """ - |{ - | "fullName": "Mariano Lischetti" - |} + val expected = + """ + |{ + | "fullName": "Mariano Lischetti" + |} """.stripMarginAndNormalizeEOL.trim result shouldBe expected } diff --git a/native-cli/src/test/scala/org/mule/weave/dwnative/cli/DependencyManagerTest.scala b/native-cli/src/test/scala/org/mule/weave/dwnative/cli/DependencyManagerTest.scala new file mode 100644 index 0000000..c0cfa48 --- /dev/null +++ b/native-cli/src/test/scala/org/mule/weave/dwnative/cli/DependencyManagerTest.scala @@ -0,0 +1,55 @@ +package org.mule.weave.dwnative.cli + +import org.mule.weave.dwnative.NativeRuntime +import org.mule.weave.dwnative.dependencies.ResolutionErrorHandler +import org.mule.weave.dwnative.dependencies.DependencyModel +import org.mule.weave.dwnative.dependencies.DependencyResolutionResult +import org.mule.weave.dwnative.dependencies.MavenDependencyModel +import org.mule.weave.dwnative.dependencies.SpellDependencyManager +import org.scalatest.FreeSpec +import org.scalatest.Matchers + +import java.io.File + +class DependencyManagerTest extends FreeSpec with Matchers { + + "it should parse the build definition correctly" in { + val simpleSpellWithDependencies = new File(TestUtils.getSpellsFolder(), "SimpleSpellWithDependencies") + val dependencies = new File(simpleSpellWithDependencies, "dependencies.dwl") + + val testConsole = new TestConsole() + val manager = new SpellDependencyManager(simpleSpellWithDependencies, testConsole) + val nativeRuntime = new NativeRuntime(TestUtils.getMyLocalSpellWithLib, Array.empty, testConsole) + val deps: Array[_ <: DependencyModel] = manager.collectDependencies(dependencies, nativeRuntime) + assert(deps.length == 1) + deps(0) match { + case MavenDependencyModel(artifactId, groupId, version, repository) => { + assert(artifactId == "data-weave-analytics-library") + assert(groupId == "68ef9520-24e9-4cf2-b2f5-620025690913") + assert(version == "1.0.1") + assert(repository.length == 1) + assert(repository(0).url == "https://maven.anypoint.mulesoft.com/api/v3/maven") + } + case _ => fail("Expecting maven model") + } + } + + + "it should resolve the artifacts correctly" in { + val simpleSpellWithDependencies = new File(TestUtils.getSpellsFolder(), "SimpleSpellWithDependencies") + val testConsole = DefaultConsole + val manager = new SpellDependencyManager(simpleSpellWithDependencies, testConsole) + val nativeRuntime = new NativeRuntime(TestUtils.getMyLocalSpellWithLib, Array.empty, testConsole) + val results: Array[DependencyResolutionResult] = manager.resolveDependencies(nativeRuntime) + assert(results.length == 1) + val artifacts = results.flatMap((a) => { + a.resolve(new ResolutionErrorHandler { + override def onError(id: String, message: String): Unit = { + fail(s"${id} : ${message}") + } + }) + }) + assert(!artifacts.isEmpty) + } + +} diff --git a/native-cli/src/test/scala/org/mule/weave/dwnative/cli/TestConsole.scala b/native-cli/src/test/scala/org/mule/weave/dwnative/cli/TestConsole.scala index 0def7ba..6e9a2e8 100644 --- a/native-cli/src/test/scala/org/mule/weave/dwnative/cli/TestConsole.scala +++ b/native-cli/src/test/scala/org/mule/weave/dwnative/cli/TestConsole.scala @@ -33,7 +33,9 @@ class TestConsole(val in: InputStream = System.in, val out: OutputStream = Syste clearCount = clearCount + 1 } - override def envVar(name: String): Option[String] = envVars.get(name) + override def envVar(name: String): Option[String] = { + envVars.get(name).orElse(Option(System.getenv(name))) + } override def doDebug(message: String): Unit = DefaultConsole.debug(message) } diff --git a/native-cli/src/test/scala/org/mule/weave/dwnative/cli/TestUtils.scala b/native-cli/src/test/scala/org/mule/weave/dwnative/cli/TestUtils.scala index 0cc0bed..00fab90 100644 --- a/native-cli/src/test/scala/org/mule/weave/dwnative/cli/TestUtils.scala +++ b/native-cli/src/test/scala/org/mule/weave/dwnative/cli/TestUtils.scala @@ -20,6 +20,12 @@ object TestUtils { localSpell } + def getSimpleSpellWithDependencies = { + val file = getSpellsFolder() + val localSpell = new File(file, "SimpleSpellWithDependencies") + localSpell + } + def getSpellsFolder(): File = { val spellsUrls = getClass.getClassLoader.getResource("spells/spells.txt")