diff --git a/modules/ROOT/assets/image-source-files/mruntime-component-foreach-example.graffle b/modules/ROOT/assets/image-source-files/mruntime-component-foreach-example.graffle new file mode 100644 index 0000000000..56103fa23b Binary files /dev/null and b/modules/ROOT/assets/image-source-files/mruntime-component-foreach-example.graffle differ diff --git a/modules/ROOT/assets/images/mruntime-component-foreach-example.png b/modules/ROOT/assets/images/mruntime-component-foreach-example.png new file mode 100644 index 0000000000..d5efdb6ecf Binary files /dev/null and b/modules/ROOT/assets/images/mruntime-component-foreach-example.png differ diff --git a/modules/ROOT/pages/for-each-scope-concept.adoc b/modules/ROOT/pages/for-each-scope-concept.adoc index 1236ac7a9f..ed5b295df9 100644 --- a/modules/ROOT/pages/for-each-scope-concept.adoc +++ b/modules/ROOT/pages/for-each-scope-concept.adoc @@ -4,182 +4,315 @@ include::_attributes.adoc[] endif::[] :page-aliases: for-each-scope-xml-reference.adoc -The For Each scope splits a payload into elements and processes them one by one through the components that you place in the scope. -It is similar to a `for-each`/`for` loop code block in most programming languages and can process any collection, including lists and arrays. -The collection can be any supported content type, such as `application/json`, `application/java`, or `application/xml`. +Processing within the For Each scope (``) is similar to a `for-each` or `for` loop in most programming languages. -//// -TODO? Splitter not in Studio 7 as of GA. Not clear when it will be in. -== Differences With a Splitter +By default, For Each splits the `payload` it receives as input into separate elements and processes them one by one or within separate batches containing a configured number of elements. Alternatively, you can use a DataWeave expression to collect a portion of the input to split (such as, `payload._someField_`). The scope accepts collections, including lists and arrays, of any supported content type, such as `application/json`, `application/java`, or `application/xml`. -The For Each scope performs a similar task to using a Splitter and then an Aggregator. The main difference is that the For Each scope outputs a collection just like the one it receives, the Aggregator outputs a message where the payload is a list of mule messages (each with its own payload and attributes). -//// -General considerations about the For Each scope: +Within the scope, each element or batch of elements is treated as a separate `payload`. Instead of outputting processed input, the scope outputs the entire Mule event that it received as input, along with any additional Mule variables created within the scope. The input event includes the un-split payload, attributes of the payload, and any Mule variables associated with the event. By default, the scope stores that event in a variable (`rootMessage`) that you can access from within For Each. Note that For Each consumes this variable, so it is not available outside For Each. -* By default, For Each tries to split the payload. If the payload is a simple Java collection, the For Each scope can split it without any configuration. The payload inside the For Each scope is each of the split elements. Attributes within the original message are ignored because they are related to the entire message. -* For Each does not modify the current payload. The output payload is the same as the input. -* For non-Java collections, such as XML or JSON, use a DataWeave expression to split data. Use the *Collection* field for this purpose. -+ -In the following example, the *Collection* field in For Each is set to iterate over an array stored in `payload.topics`: -+ -image::component-foreach-example.png[For Each Component] +The scope's splitter ignores attributes of the input to For Each. These attributes are metadata in the Mule message, such as the content type or status code in a response to an HTTP request. Mule attributes are not accessible from the split elements, so `attributes` returns `null` from within For Each. However, you can use the For Each variable `rootMessage` to access the attributes from within For Each, for example, with `vars.rootMessage.attributes."content-type"` to return a content type such as `application/json; charset=utf-8` from the header attributes of an HTTP response. + +Unlike attributes in the Mule message, Mule variables are directly accessible within For Each using the `vars` selector, such as `vars._someVariable_`. See <> to understand how modifications to values of Mule variables propagate from within and outside of the For Each component. + +[[example]] +== Basic Flow With For Each -The For Each scope stores each item of the collection in `payload` during each iteration. -//// -Note that if the input contains information outside the collection you tell it to split, this information is lost. -//// +The following example illustrates a basic flow with For Each in Anypoint Studio. Assume that the scope receives its input from a scheduled HTTP request for this https://jsonplaceholder.typicode.com/users[JSON array of objects]. -You can also split an array into batches to enable quicker processing. Each batch is treated as a separate Mule message. For example, if a collection has 200 elements and you set *Batch Size* to `50`, the For Each scope iteratively processes 4 batches of 50 elements, each as a separate Mule message. +[[studio_example]] +image::mruntime-component-foreach-example.png[For Each Component] -=== Example XML +As the following XML for this example shows, For Each (``) collects and splits the entire array (`payload`) it receives into separate objects. Components within the scope select a portion of each object, write a portion of that selected content into separate files, and log the new `payload` that results within For Each. -This is an example XML based on the For Each scope configuration detailed above: -[source,xml,linenums] +[source,xml] ---- -... - - - - -... + + + + + + + + + + + + + ---- +. For Each (``) collects the entire input payload and splits it into the ten top-level JSON objects within it, such as this one: ++ +[source,json] +---- +{ + "id": 1, + "name": "Leanne Graham", + "username": "Bret", + "email": "Sincere@april.biz", + "address": { + "street": "Kulas Light", + "suite": "Apt. 556", + "city": "Gwenborough", + "zipcode": "92998-3874", + "geo": { + "lat": "-37.3159", + "lng": "81.1496" + } + }, + "phone": "1-770-736-8031 x56442", + "website": "hildegard.org", + "company": { + "name": "Romaguera-Crona", + "catchPhrase": "Multi-layered client-server neural-net", + "bs": "harness real-time e-markets" + } +} +---- +. As For Each iterates over each object, the DataWeave expression (`payload.company`) in the xref:transform-component-about.adoc[Transform Message] component (``) uses DataWeave selectors to extract the value of each `"company"` key and transform the `payload` to the value of each of those keys. As in a Mule flow, the next component within For Each receives the transformed payload as its input. +. The File Write operation (``) uses `payload.name` within a larger DataWeave expression to select the `name` of each company from the transformed object. The name of each file follows the pattern specified in the expression, which hyphenates the extracted company name and appends a date-time stamp to make the file name unique. For example, the name of a file for Abernathy Group data looks like this: ++ +[source,log] +---- +Abernathy-Group-20221007113314.json +---- +By default, the operation also loads the payload received from Transform Message to each file, for example: ++ +[source,log] +---- +{ + "name": "Abernathy Group", + "catchPhrase": "Implemented secondary concept", + "bs": "e-enable extensible e-tailers" +} +---- + +For configurable For Each properties, see <>. + +== Use Cases and Examples + +You can learn more about using For Each from several Mule projects within https://www.mulesoft.com/exchange/[Anypoint Exchange^]: + +* _Authenticating Salesforce using OAuth2_ +* _Import contacts into Microsoft Dynamics CRM_ +* _Importing a CSV file into mongoDB_ +* _Importing an Email Attachment using the IMAP Connector_ +* _Importing Email Attachments using the POP3 Connector_ +* _Querying a Database and Attaching Results to an Email_ + +You can also open these projects from xref:studio::import-project-exchange.adoc[Anypoint Studio]. + +[[variable_propagation]] == Variable Propagation -Every execution of the For Each scope starts with the variables and values -from the previous execution of the block. New variables or modifications -to existing variables that take place when processing one element are visible -during the processing of another element. These changes to variables continue -to be available outside the For Each scope. +Each execution within For Each begins with the values of Mule variables +from the previous execution. New Mule variables or modifications to the values of existing variables that take place when processing one element are accessible during the processing other elements. Changes to Mule variables continue to be available outside the For Each scope. -[source,xml,linenums] +[source,xml] ---- - - + + - - + + - + - - + + + + ---- -After aggregation, the variables are: +As the xref:choice-router-concept.adoc[Choice router] (``) executes the processors within each condition, the Logger (`Logger After Choice`) prints the following variable values: -[source,json,linenums] ----- -{var1: "var1", var2: "newValue", var3: "otherVal", var4: "val4"} +[source,log] ---- +// Condition: when payload == 'apple' +.LoggerMessageProcessor: +[var1-BeforeForEach, var2-newValue, var3-appleVal, null] -== Error Handling +// Condition: when payload == 'banana' +.LoggerMessageProcessor: +[var1-BeforeForEach, var2-newValue, var3-appleVal bananaVal, null] -If one of the elements in a collection throws an exception, the For Each scope stops processing that collection and invokes the error handler. +// Condition: otherwise +.LoggerMessageProcessor: +[var1-BeforeForEach, var2-newValue, var3-otherVal, var4-val4] +---- -== Example Projects +The last values are propagated outside of For Each. The logger (`Logger After For Each`) prints the same values as the ones printed after the final choice condition (`otherwise`). -There are several example projects in Anypoint Exchange that you can open in Anypoint Studio to learn more about how to use the For Each scope: +[source,log] +---- +[var1-BeforeForEach, var2-newValue, var3-otherVal, var4-val4] +---- -* _Authenticating Salesforce using OAuth2_ -* _Import contacts into Microsoft Dynamics CRM_ -* _Importing a CSV file into mongoDB_ -* _Importing an Email Attachment using the IMAP Connector_ -* _Importing Email Attachments using the POP3 Connector_ -* _Querying a Database and Attaching Results to an Email_ +== Error Handling -To download and open an example project while you are in Anypoint Studio, click the Exchange icon in the upper-left corner. Then, in the window that opens, log into Anypoint Exchange and search on the name of the project. +If one of the elements in a collection throws an exception, For Each stops processing that collection and invokes the error handler. -== XML Reference +See xref:on-error-scope-concept.adoc[] for information about Mule error handlers. -For Each scopes open and close with a `` tag. Components that are affected by this scope are defined as child elements of the `` tag. +[[reference]] +== Reference -=== Configurable Properties +The For Each scope (``) provides a number of configurable properties: -[%header,cols="35,20,45"] +[%header,cols="1a,1a,1a,4a"] |=== -|Property | Default | Description +| Field Name | XML | Default | Description + +| *Collection* | `collection` | `payload` -| An expression that returns a Java collection, object array, map, or DOM - nodes. +| A DataWeave expression that returns a Java collection, object, array, map, or DOM nodes. For Each automatically splits the collection into elements, such as the highest-level objects in a JSON array of objects. Each element becomes the `payload` on which a processor within For Each acts. For Each accepts collections of any supported content type, such as `application/json`, `application/java`, or `application/xml`. +| *Counter Variable Name* | `counterVariableName` | `counter` -| Name of the property that stores the number of messages over which it iterates. +| Name of the variable that assigns a number to each iteration over a given element. The sequence starts with `1` and proceeds sequentially. Using the default name of this property (`counter`), you can access this number with `vars.counter`. For Each consumes this variable, so it is not available and returns `null` when used outside of the scope. +| Batch Size | `batchSize` | `1` -| Partitions the collection into sub-collections of the specified - size. For example, if a collection has 200 elements and you set the batch size to 50, it processes 4 batches of 50 elements each. +| Number that the scope uses to partition the collected elements from the input into sub-collections of a specified size. For example, if input to For Each is an array of 199 elements and you set the batch size to 50, the scope produces three arrays of 50 elements each and one array of 49 elements. +| *Root Message Variable Name* | `rootMessageVariableName` | `rootMessage` -| Name of the property that stores the parent message. The parent is the complete, non-split message. +| Configurable name of the variable that stores the complete, un-split Mule message from the original input to For Each. The message contains the payload and any attributes but does not contain any variables (`vars`), which are part of the Mule event that contains the Mule message. Within For Each, you can gain access to the value of the payload of the message with `vars.rootMessage.payload` and to attributes of the message with `vars.rootMessage.attributes`. For Each consumes this variable, so you can access it only from within the scope. Outside For Each, the variable returns `null`. |=== +The following example illustrates the use of several attributes that are available to For Each: -//// -=== Iterate a JSON Array - -[source,xml,linenums] +[source,xml] ---- -include::{examplesdir}/for-each-scope-concept_1.xml[] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ---- -In this case, the payload contains a JSON array. For Each iterates over the payload because the `collection` attribute has not been defined. The payload within For Each is each of the users in the JSON array. - -=== Iterate over a Java Map that is Placed in a Variable +After an HTTP request (``), the remaining components in the flow are configured to illustrate how For Each properties work and how access to the `payload` and `attributes` takes place within and outside of For Each: -[source,xml,linenums] +. For Each collects company objects (`collection="payload.company"`) from the JSON array returned from the HTTP request's `url`. For Each iterates over each of these objects. +. In Logger 1, `iterationCounter : vars.counter` returns an object with the iteration number, starting with `1` for the first company object and ending with `10` for the last, for example, `{iterationCounter=1}`. +. In Logger 2, `sharedStatusCode : vars.rootMessage.attributes.statusCode` retrieves the value of the status code attribute that is stored in the `rootMessage` variable: `{sharedStatusCode=200}`. The status code comes from the original input to For Each and is not included in the split payload, so the Logger prints the same status code during each iteration of For Each. +. In ``, the DataWeave expression extracts the value of the `payload.name` from each company object, hyphenates, and appends a unique date-time stamp to the transformed company name. Examples of names of files created by the expression are `Abernathy-Group-20221007120752.json` for the first company and `Yost-and-Sons-20221007120752.json` for the last. ++ +For the file content, the setting uses the default (`payload`) implicitly, so the created files contain company data in JSON format, such as the following data for the first company object: ++ +[source,log] ---- -include::{examplesdir}/for-each-scope-concept_2.xml[] +{ + "name": "Abernathy Group", + "catchPhrase": "Implemented secondary concept", + "bs": "e-enable extensible e-tailers" +} ---- - -The example uses the `ee:transform` to set a variable named `users` with a Java map that is created with DataWeave. Then it uses a built-in DataWeave function to get the entries of a map as a collection in order to iterate over it. - -=== Split a Collection into Batches - -[source,xml,linenums] +. In Logger 3: `vars.rootMessage.payload..company.name` retrieves an array of company name values stored in the `rootMessage` variable. Like the `statusCode` value, this data is part of the original, un-split input to For Each that is stored in the `rootMessage` variable. All company names are accessible from that variable, so they can all be listed in the array. ++ +[source,log] ---- -include::{examplesdir}/for-each-scope-concept_3.xml[] +[Romaguera-Crona, Deckow-Crist, Romaguera-Jacobson, Robel-Corkery, + Keebler LLC, Considine-Lockman, Johns Group, Abernathy Group, + Yost and Sons, Hoeger LLC] ---- +. Logger 4 is located outside For Each. The expression `payload..company.name` in this logger produces the same array produced by Logger 3, but Logger 4 is able to select `payload` directly. For Each consumes the `rootMessage` variable, so the expression `vars.rootMessage.payload..company.name` from Logger 4 logger would return `null`. +. Logger 5 is located outside For Each. The expression `attributes.statusCode` produces the same status code value as the expression in Logger 2, but Logger 5 is able to select `attributes` directly. For Each consumes the `rootMessage` variable, so the expression `vars.rootMessage.attributes.statusCode` from Logger 5 would return `null`. -The `batchSize` attribute creates sub-arrays of two elements. So inside `foreach`, the payload will have an array of two elements. - -=== Aggregate the Process of Each Element into a Variable +The next example illustrates the batch size (`batchsize`) functionality. Assume that the flow collects its input payload from the preceding flow through a Flow Reference (``) component: -[source,xml,linenums] +[source,xml] ---- -include::{examplesdir}/for-each-scope-concept_4.xml[] + + ... + + + + +