Adding dep management (#47)

* Adding dep management * Cleanup print * Removing local maven * Adding logs * Using tmp folder * It should fail when it can not resolve * It should fail when it can not resolve * Fixing logging * Adding more logs * Trying to fix build * Cleanup
mulesoft-labs · Sep 28, 2022 · 20e76ae · 20e76ae
1 parent 842bbe6
commit 20e76ae
Show file tree

Hide file tree

Showing 19 changed files with 570 additions and 74 deletions.
diff --git a/README.md b/README.md
@@ -1,13 +1,19 @@
 # DataWeave CLI
 
-**DataWeave CLI** is a command-line interface that allows `querying`, `filtering`, and `mapping` structured data from different data sources like `JSON`, `XML`, `CSV`, `YML` to other data formats. It also allows to easily create data in such formats.
+**DataWeave CLI** is a command-line interface that allows `querying`, `filtering`, and `mapping` structured data from
+different data sources like `JSON`, `XML`, `CSV`, `YML` to other data formats. It also allows to easily create data in
+such formats.
 
-For more info about the `DataWeave` language visit the [documenation site](https://docs.mulesoft.com/mule-runtime/latest/dataweave)
+For more info about the `DataWeave` language visit
+the [documenation site](https://docs.mulesoft.com/mule-runtime/latest/dataweave)
 
 ## What is Included?
-The binary distribution already ships with a set of modules and data formats that makes this useful for a very interesting and broad set of use cases.
+
+The binary distribution already ships with a set of modules and data formats that makes this useful for a very
+interesting and broad set of use cases.
 
 ### Included Modules
+
 - [DataWeave Standard Library](https://github.com/mulesoft/data-weave/tree/master/wlang)
 
 ### Supported Data Formats
@@ -25,7 +31,7 @@ The binary distribution already ships with a set of modules and data formats tha
 | `text/plain`                        |    `text`    |                        [Text Plain Format](https://docs.mulesoft.com/dataweave/latest/dataweave-formats-text) |
 | `text/x-java-properties`            | `properties` |               [Text Java Properties](https://docs.mulesoft.com/dataweave/latest/dataweave-formats-properties) |
 
-## Installation 
+## Installation
 
 ### Homebrew (Mac)
 
@@ -35,18 +41,22 @@ brew install dw
 ```
 
 ### Manual Installation
+
 1. Download the latest [release version](https://github.com/mulesoft-labs/data-weave-cli/releases) according to your OS.
 2. Unzip the file on your `<user.home>/.dw`
 3. Add `<user.home>/.dw/bin` to your **PATH**
 
 ### Build and Install
 
-To build the project, you need to run gradlew with the graalVM distribution based on Java 11. You can download it at https://github.com/graalvm/graalvm-ce-builds/releases
+To build the project, you need to run gradlew with the graalVM distribution based on Java 11. You can download it
+at https://github.com/graalvm/graalvm-ce-builds/releases
 Set:
+
 ```bash
 export GRAALVM_HOME=<pathToGraalVMFolder>/graalvm-ce-java11-21.2.0/Contents/Home
 export JAVA_HOME=<pathToGraalVMFolder>/graalvm-ce-java11-21.2.0/Contents/Home
 ```
+
 Execute the gradle task `nativeCompile`
 
 ```bash
@@ -59,11 +69,12 @@ Once it finishes you will find the `dw` binary in `native-cli/build/native/nativ
 
 ## How to Use It
 
-If the directory containing the `dw` executable is in your _PATH_, you can run `dw` from anywhere. 
+If the directory containing the `dw` executable is in your _PATH_, you can run `dw` from anywhere.
 
 If it is not, go to the `bin` directory referenced in the installation instructions and run `dw` from there.
 
-The following example shows the DataWeave CLI documentation 
+The following example shows the DataWeave CLI documentation
+
 ```bash
 dw --help
 ```
@@ -141,6 +152,28 @@ The following are the DataWeave CLI environment variables that you can set in yo
 | `DW_DEFAULT_INPUT_MIMETYPE`  | The default `mimeType` that is going to be used for the standard input. If not defined `application/json` will be used. |
 | `DW_DEFAULT_OUTPUT_MIMETYPE` | The default output `mimeType` that is going to be if not defined. If not defined `application/json` will be used.       |
 
+## Dependency Manager
+
+In order for a spell to depend on a library it can include a library it can use the dependencies.dwl to specify the list of dependencies that it should be included and download
+
+```data-weave
+%dw 2.0
+var mavenRepositories = [{
+    url: "https://maven.anypoint.mulesoft.com/api/v3/maven"
+}]
+---
+{
+  dependencies: [
+    {
+      kind: "maven",
+      artifactId: "data-weave-analytics-library",
+      groupId: "68ef9520-24e9-4cf2-b2f5-620025690913",
+      version: "1.0.1",
+      repositories: mavenRepositories // By default mulesoft, exchange and central are being added
+    }
+  ]
+}
+```
 
 ### Querying Content From a File
 
@@ -198,7 +231,6 @@ dw -i payload <fullpathToUsers.json> "output application/json --- payload filter
 cat <fullpathToUser.json> | dw "output application/json --- payload filter (item) -> item.age > 17"
 ```
 
-
 ### Redirecting the Output to a File
 
 ```bash 
@@ -226,9 +258,11 @@ curl "https://api.github.com/repos/mulesoft/mule/commits?per_page=5" | dw "{mess
 ``` 
 
 ### Generate a Request with Body
+
 This example uses the [jsonplaceholder API](https://jsonplaceholder.typicode.com/) to update a resource.
 
 Steps:
+
 1. Search the post resource with the `id = 1`.
 2. Use DataWeave CLI to create a JSON output changing the post title `My new title`.
 3. Finally, update the post resource.
@@ -238,6 +272,7 @@ curl https://jsonplaceholder.typicode.com/posts/1 | dw "output application/json
 ```
 
 #### Output
+
 ```json
 {
   "id": 1,
@@ -258,4 +293,5 @@ Before creating a pull request review the following:
 * [SECURITY](SECURITY.md)
 * [CODE_OF_CONDUCT](CODE_OF_CONDUCT.md)
 
-When you submit your pull request, you are asked to sign a contributor license agreement (CLA) if we don't have one on file for you.
+When you submit your pull request, you are asked to sign a contributor license agreement (CLA) if we don't have one on
+file for you.
diff --git a/gradle.properties b/gradle.properties
@@ -2,7 +2,7 @@ weaveVersion=2.5.0-SNAPSHOT
 nativeVersion=100.100.100
 scalaVersion=2.12.11
 ioVersion=1.0.0-SNAPSHOT
-graalvmVersion=22.0.0.2
+graalvmVersion=22.2.0
 #Libaries
 scalaTestVersion=3.0.1
 scalaTestPluginVersion=0.32

diff --git a/native-cli-integration-tests/src/test/scala/org/mule/weave/native/NativeCliRuntimeIT.scala b/native-cli-integration-tests/src/test/scala/org/mule/weave/native/NativeCliRuntimeIT.scala
@@ -363,6 +363,7 @@ class NativeCliRuntimeIT extends FunSpec
         "module-singleton",
         "multipart-write-binary",
         "read-binary-files",
+        "underflow",
         "try",
         "urlEncodeDecode") ++
       // Uses resource name that is different on Cli than in the Tests

diff --git a/native-cli/build.gradle b/native-cli/build.gradle
@@ -3,7 +3,7 @@ plugins {
     id "com.github.maiflai.scalatest" version "${scalaTestPluginVersion}"
     id 'application'
     // Apply GraalVM Native Image plugin
-    id 'org.graalvm.buildtools.native' version '0.9.13'
+    id 'org.graalvm.buildtools.native' version '0.9.14'
 }
 
 sourceSets {
@@ -19,6 +19,9 @@ mainClassName = 'org.mule.weave.dwnative.cli.DataWeaveCLI'
 dependencies {
     api group: 'org.mule.weave', name: 'runtime', version: weaveVersion
     compileOnly group: 'org.graalvm.sdk', name: 'graal-sdk', version: graalvmVersion
+    implementation group: 'io.get-coursier', name: 'coursier-core_2.12', version: '1.1.0-M14-7'
+    implementation group: 'io.get-coursier', name: 'coursier_2.12', version: '1.1.0-M14-7'
+    implementation group: 'io.get-coursier', name: 'coursier-cache_2.12', version: '1.1.0-M14-7'
     implementation group: 'org.mule.weave', name: 'core-modules', version: weaveVersion
     implementation group: 'org.mule.weave', name: 'yaml-module', version: weaveVersion
     implementation group: 'org.mule.weave', name: 'http-module', version: ioVersion
@@ -116,6 +119,7 @@ graalvmNative {
                     "org.asynchttpclient," +
                     "org.mule.weave.v2.module.http.netty.HttpAsyncClientService," +
                     "scala.util.Random," +
+                    "coursier.," +
                     "org.mule.weave.v2.sdk.SPIBasedModuleLoaderProvider\$")
             buildArgs.add("--initialize-at-build-time=" +
                     "sun.instrument.InstrumentationImpl," +
@@ -143,6 +147,7 @@ graalvmNative {
 //            "org.mule.weave.v2.model.types.,"
 //            "org.mule.weave.v2.core.functions."
 //    option "-H:+TraceClassInitialization"
+            buildArgs.add("--trace-class-initialization=coursier.core.Type\$")
             buildArgs.add("-H:DeadlockWatchdogInterval=1000")
             buildArgs.add("--report-unsupported-elements-at-runtime")
             buildArgs.add("-H:CompilationExpirationPeriod=0")

diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/NativeRuntime.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/NativeRuntime.scala
@@ -42,6 +42,7 @@ import org.mule.weave.v2.runtime.ModuleComponentsFactory
 import org.mule.weave.v2.runtime.ParserConfiguration
 import org.mule.weave.v2.runtime.ScriptingBindings
 import org.mule.weave.v2.runtime.ScriptingEngineSetupException
+import org.mule.weave.v2.sdk.NameIdentifierHelper
 import org.mule.weave.v2.sdk.SPIBasedModuleLoaderProvider
 import org.mule.weave.v2.sdk.TwoLevelWeaveResourceResolver
 import org.mule.weave.v2.sdk.WeaveResourceResolver
@@ -58,7 +59,7 @@ class NativeRuntime(libDir: File, path: Array[File], console: Console) {
   private val dataWeaveUtils = new DataWeaveUtils(console)
 
   private val pathBasedResourceResolver: PathBasedResourceResolver = PathBasedResourceResolver(path ++ Option(libDir.listFiles()).getOrElse(new Array[File](0)))
-  
+
   private val weaveScriptingEngine: DataWeaveScriptingEngine = {
     setupEnv()
     DataWeaveScriptingEngine(new NativeModuleComponentFactory(() => pathBasedResourceResolver, systemFirst = true), ParserConfiguration())
@@ -68,6 +69,10 @@ class NativeRuntime(libDir: File, path: Array[File], console: Console) {
     weaveScriptingEngine.enableProfileParsing()
   }
 
+  def addJarToClassPath(file: File): Unit = {
+    pathBasedResourceResolver.addContent(ContentResolver(file))
+  }
+
   /**
     * Setup initialization properties
     */
@@ -103,9 +108,9 @@ class NativeRuntime(libDir: File, path: Array[File], console: Console) {
   private def compileScript(script: String, inputs: ScriptingBindings, nameIdentifier: NameIdentifier, defaultOutputMimeType: String) = {
     weaveScriptingEngine.compile(script, nameIdentifier, inputs.entries().map(wi => new InputType(wi, None)).toArray, defaultOutputMimeType)
   }
-  
+
   private def createServiceManager(maybePrivileges: Option[Seq[String]] = None): ServiceManager = {
-    
+
     val charsetProviderService = new CharsetProviderService {
       override def defaultCharset(): Charset = {
         StandardCharsets.UTF_8
@@ -120,7 +125,7 @@ class NativeRuntime(libDir: File, path: Array[File], console: Console) {
     if (maybePrivileges.isDefined) {
       val privileges = maybePrivileges.get
       val weaveRuntimePrivileges = privileges.map(WeaveRuntimePrivilege(_)).toArray
-      customServices = customServices + (classOf[SecurityManagerService] -> new DefaultSecurityManagerService(weaveRuntimePrivileges)) 
+      customServices = customServices + (classOf[SecurityManagerService] -> new DefaultSecurityManagerService(weaveRuntimePrivileges))
     }
     ServiceManager(new ConsoleLogger(console), customServices)
   }
@@ -143,12 +148,7 @@ class WeavePathProtocolHandler(path: PathBasedResourceResolver) extends ReadFunc
 
   override def createSourceProvider(url: String, locatable: LocationCapable, charset: Charset): SourceProvider = {
     val uri = url.stripPrefix(CLASSPATH_PREFIX)
-    val wellFormedUri = if (uri.startsWith("/")) {
-      uri.substring(1)
-    } else {
-      uri
-    }
-    val maybeResource = path.resolve(wellFormedUri)
+    val maybeResource = path.resolve(uri)
     maybeResource match {
       case Some(value) => {
         SourceProvider(value, charset)

diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/PathBasedResourceResolver.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/PathBasedResourceResolver.scala
@@ -22,11 +22,12 @@ class PathBasedResourceResolver(paths: mutable.ArrayBuffer[ContentResolver]) ext
   }
 
   override def resolve(name: NameIdentifier): Option[WeaveResource] = {
-    val filePath = NameIdentifierHelper.toWeaveFilePath(name)
+
     val iterator = paths.iterator
     while (iterator.hasNext) {
-      val maybeResource = iterator.next().resolve(filePath)
+      val maybeResource: Option[InputStream] = iterator.next().resolve(name)
       if (maybeResource.isDefined) {
+        val filePath = NameIdentifierHelper.toWeaveFilePath(name, "/") //Use unix based system
         return Some(WeaveResource(filePath, toString(maybeResource.get)))
       }
     }
@@ -43,9 +44,10 @@ class PathBasedResourceResolver(paths: mutable.ArrayBuffer[ContentResolver]) ext
   }
 
   def resolve(filePath: String): Option[InputStream] = {
+    val ni = NameIdentifierHelper.fromWeaveFilePath(filePath, "/")
     val iterator = paths.iterator
     while (iterator.hasNext) {
-      val maybeResource = iterator.next().resolve(filePath)
+      val maybeResource = iterator.next().resolve(ni)
       if (maybeResource.isDefined) {
         return maybeResource
       }
@@ -55,29 +57,20 @@ class PathBasedResourceResolver(paths: mutable.ArrayBuffer[ContentResolver]) ext
 
 
   override def resolveAll(name: NameIdentifier): Seq[WeaveResource] = {
-    val filePath = NameIdentifierHelper.toWeaveFilePath(name)
     paths
-      .flatMap(_.resolve(filePath))
-      .map((content) => WeaveResource(filePath, toString(content)))
+      .flatMap(_.resolve(name))
+      .map((content) => {
+        val path = NameIdentifierHelper.toWeaveFilePath(name, "/")
+        WeaveResource(path, toString(content))
+      })
   }
 }
 
 /**
   *
   */
 trait ContentResolver {
-  def resolve(path: String): Option[InputStream]
-}
-
-class CompositeContentResolver(contents: Seq[ContentResolver]) extends ContentResolver {
-  override def resolve(path: String): Option[InputStream] = {
-    contents
-      .toStream
-      .flatMap((content) => {
-        content.resolve(path)
-      })
-      .headOption
-  }
+  def resolve(path: NameIdentifier): Option[InputStream]
 }
 
 
@@ -93,7 +86,8 @@ object ContentResolver {
 
 class DirectoryContentResolver(directory: File) extends ContentResolver {
 
-  override def resolve(path: String): Option[InputStream] = {
+  override def resolve(ni: NameIdentifier): Option[InputStream] = {
+    val path = NameIdentifierHelper.toWeaveFilePath(ni, File.separator) //Use unix based system
     val file = new File(directory, path)
     if (file.isFile) {
       Some(new FileInputStream(file))
@@ -107,12 +101,15 @@ class JarContentResolver(jarFile: => File) extends ContentResolver {
 
   lazy val zipFile = new ZipFile(jarFile)
 
-  override def resolve(path: String): Option[InputStream] = {
-    val zipEntry = if (path.startsWith("/")) {
-      path.substring(1)
-    } else {
-      path
-    }
+  override def resolve(ni: NameIdentifier): Option[InputStream] = {
+    val path = NameIdentifierHelper.toWeaveFilePath(ni, "/") //Use unix based system
+
+    val zipEntry: String =
+      if (path.startsWith("/")) {
+        path.substring(1)
+      } else {
+        path
+      }
     val pathEntry = zipFile.getEntry(zipEntry)
     if (pathEntry != null) {
       Some(zipFile.getInputStream(pathEntry))