Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spelling and links #60

Merged
merged 1 commit into from
Mar 8, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions blog/2022-01-24-image-processing-workflow-with-conductor.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ tags: [Netflix Conductor, Orkes, Conductor, orchestration, image processing, 202

There are many tools available to work with images - resizing, changing the format, cropping, changing colors, etc. Tools like Photoshop require a lot of manual work to create image. Online tools for image processing are also extremely popular. But, rather than doing the work manually, or paying for a service to modify your images, wouldn't it be cool to have a workflow that does image resizing for you automatically? In this post, we'll build just this using Conductor to orchestrate the microservices involved, and to create an API-like surface for image processing.

In this post, we'll run Conductor locally on your computer. The Conductor workflow consists of two tasks. The first task reads in an image and resizes it according to the parameters provided (labeled "image_conver_resize_ref" in the image below). The second task ("image_toS3_ref" below) takes the resized image and saves the it to an Amazon S3 bucket.
In this post, we'll run Conductor locally on your computer. The Conductor workflow consists of two tasks. The first task reads in an image and resizes it according to the parameters provided (labeled "image_convert_resize_ref" in the image below). The second task ("image_toS3_ref" below) takes the resized image and saves the it to an Amazon S3 bucket.

<img src="/content/img/blogassets/imageprocessing-workflow-diagram.png" alt="Diagram of our image processing workflow" width="350" style={{paddingBottom: 40, paddingTop: 0}} />

Expand Down Expand Up @@ -41,7 +41,7 @@ Now we can build and run the Conductor server:

```
cd conductor
$ cd serverserver
$ cd server
$ ../gradlew bootRun
```

Expand Down Expand Up @@ -308,7 +308,7 @@ curl -X 'POST' \
}'
```

The response will be a Workflow ID. If you have ElasticSeacrh running on your system, the results will appear at localhost:5000. If you do not (and it was not a part of this tutorial) - you can browse to the workflow manually by visiting ```http://localhost:5000/execution/[workflowid]```
The response will be a Workflow ID. If you have ElasticSearch running on your system, the results will appear at localhost:5000. If you do not (and it was not a part of this tutorial) - you can browse to the workflow manually by visiting ```http://localhost:5000/execution/[workflowid]```


You'll see a page similar to the one below with “completed” in green next to the workflowId:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Conductor allows you to define your stateful applications that can handle failur
Each task in Conductor can be configured how it responds to availability events such as: 1) Failures 2) Timeouts and 3) Rate limits.


Here is a sample task defintion:
Here is a sample task definition:

```
{
Expand Down
16 changes: 8 additions & 8 deletions blog/2022-01-31-dynamic-workflows.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
---
slug: image-processing-multiple-images-dynamic
title: Dyanmic forks- scaling your system at runtime - an image processing example
title: Dynamic forks- scaling your system at runtime - an image processing example
authors: doug
tags: [Netflix Conductor, Orkes, Conductor, orchestration, image processing, fork, dynamic fork, tutorial, 2022]
---

In recent posts, we have built several image processesing workflows with Conductor. In our first post, we created an [image processing workflow for one image](image-processing-workflow-with-conductor) - where we provide an image along with the desired output dimensions and format. The workflow output a link on Amazon S3 to the desired file.
In recent posts, we have built several image processing workflows with Conductor. In our first post, we created an [image processing workflow for one image](image-processing-workflow-with-conductor) - where we provide an image along with the desired output dimensions and format. The workflow output a link on Amazon S3 to the desired file.

In the 2nd example, we used the FORK System task to create [multiple images](image-processing-multiple-images-forks) in parallel. The number of images was hardcoded in the workflow - as FORK generates exactly as many paths as are coded into the workflow.

As number of images is hardcoded in the workflow - only 2 images are created. When it ocmes to image generation, there is often a need for more images (as new formats become popular) or sizes - as more screens are supported.
As number of images is hardcoded in the workflow - only 2 images are created. When it comes to image generation, there is often a need for more images (as new formats become popular) or sizes - as more screens are supported.

Luckily, Conductor supports this flexibility, and has a feature to specify the number of tasks to be created at runtime. In this post, we'll demonstrate the use of [dynamic forks](https://orkes.io/content/docs/reference-docs/dynamic-fork-task), where the workflow splitting is done at runtime.
Luckily, Conductor supports this flexibility, and has a feature to specify the number of tasks to be created at runtime. In this post, we'll demonstrate the use of [dynamic forks](/content/docs/reference-docs/dynamic-fork-task), where the workflow splitting is done at runtime.

Learn how to create a dynamic fork workflow in this post!

Expand Down Expand Up @@ -175,7 +175,7 @@ This task is also present in the [Github repository](https://github.com/orkes-io

We will need to create 2 tasks (and for simplicity, we can just copy the files from Github): ```image_multiple_convert_resize.json``` and ```image_convert_resize.json```.

You can add these definintions using curl:
You can add these definitions using curl:

```
curl -X 'POST' \
Expand Down Expand Up @@ -228,9 +228,9 @@ This will spawn 9 different processes and create 9 images.

### Not quite the same

In this workflow, our dynamic task creates the resized and reformatted images. In the earlier workflows, there is a second task that uploads the videos to S3. A Dyanmic task will only run one task, so in this case, we are just generating the image.
In this workflow, our dynamic task creates the resized and reformatted images. In the earlier workflows, there is a second task that uploads the videos to S3. A Dynamic task will only run one task, so in this case, we are just generating the image.

To run several tasks per dyanmica task, we'll need to create a [subworkflow](https://orkes.io/content/docs/reference-docs/sub-workflow-task) inside the dynamic task.
To run several tasks per dynamic task, we'll need to create a [subworkflow](https://orkes.io/content/docs/reference-docs/sub-workflow-task) inside the dynamic task.

For simplicity, we've introduced the dyanmica task here, and in our next post, we will combine the dynamic task with a subworkflow to create the images AND upload them to S3.
For simplicity, we've introduced the dynamic task here, and in our next post, we will combine the dynamic task with a subworkflow to create the images AND upload them to S3.

18 changes: 9 additions & 9 deletions blog/2022-01-31-image-processing-fork.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,23 @@ tags: [Netflix Conductor, Orkes, Conductor, orchestration, image processing, for
---


In our [previous post](image-processing-workflow-with-conductor) on image processesing workflows, we built a Netflix Conductor workflow that took an image input, and then ran 2 tasks: The first task resizes and reformats the image, and the second task uploads the image to an AWS S3 bucket.
In our [previous post](image-processing-workflow-with-conductor) on image processing workflows, we built a Netflix Conductor workflow that took an image input, and then ran 2 tasks: The first task resizes and reformats the image, and the second task uploads the image to an AWS S3 bucket.

With today's varied screen sizes, and varied browser support, it is a common requirement that the image processing pipeline must create multiple images with different sizes and formats of each image.

To do this with a Conductor workflow, we'll utilize the [FORK](https://orkes.io/content/docs/reference-docs/fork-task) operation to create parallel processes to generate multiple versions of the same image. The FORK task creates multiple parallel processes, so each iamge will be created asynchronously - ensuring a fast and efficient process.
To do this with a Conductor workflow, we'll utilize the [FORK](/content/docs/reference-docs/fork-task) operation to create parallel processes to generate multiple versions of the same image. The FORK task creates multiple parallel processes, so each image will be created asynchronously - ensuring a fast and efficient process.

In this post, our workflow will create 2 versions of the same image - a jpg and webp.

<!-- truncate -->

> NOTE: This demo is provided to explain the FORK task in Conductor, but is not the best workflow to generate multiple images. For that - please read the [Image processesing with dynamic workflows](image-processing-multiple-images-dynamic) post.
> NOTE: This demo is provided to explain the FORK task in Conductor, but is not the best workflow to generate multiple images. For that - please read the [Image processing with dynamic workflows](image-processing-multiple-images-dynamic) post.

## Geting Started
## Getting Started

In this demonstration, we'll be running Conductor locally. Once you have followed the steps for [setting up a local Conductor instance](https://orkes.io/content/docs/getting-started/install/running-locally), you'll be ready to go.
In this demonstration, we'll be running Conductor locally. Once you have followed the steps for [setting up a local Conductor instance](/content/docs/getting-started/install/running-locally), you'll be ready to go.

Since we are doing image processesing, you'll also want to have ImageMagick [installed on your machine](https://imagemagick.org/script/download.php).
Since we are doing image processing, you'll also want to have ImageMagick [installed on your machine](https://imagemagick.org/script/download.php).

The workflow, tasks and Java workers are all a part of the [orkesworkers](https://github.com/orkes-io/orkesworkers) GitHub repository.

Expand Down Expand Up @@ -128,7 +128,7 @@ When the 2 workflows have completed, the last task in each flow will fire that i
```
This creates a JSON object with our 2 AWS S3 URLs where our new jpg and webp images are now hosted.

> The JOIN and FORK tasks are System tasks, and therefore do not require an additional task definitio- the workflow definition is suffcient.
> The JOIN and FORK tasks are System tasks, and therefore do not require an additional task definition- the workflow definition is sufficient.


## The Fork Tasks
Expand Down Expand Up @@ -157,7 +157,7 @@ In the workflow above, we omitted the workflow tasks. Let's look at the first w
]
```

These two tasks should be familiar to those who read the first image processesing post. They are very nearly the same - the only difference is the addition of ```_jpg_``` to the taskReferenceName, and the ```"outputFormat"``` term) is now hardcoded to ```jpg```.
These two tasks should be familiar to those who read the first image processing post. They are very nearly the same - the only difference is the addition of ```_jpg_``` to the taskReferenceName, and the ```"outputFormat"``` term) is now hardcoded to ```jpg```.


In the first fork, we have 2 tasks: ```image_convert_resize``` and ```upload_toS3```.
Expand Down Expand Up @@ -237,7 +237,7 @@ This task is described in our [previous post](image-processing-workflow-with-con
}
```

Here's how to add it to your lcoal Conductor instance:
Here's how to add it to your local Conductor instance:

```
curl -X 'POST' \
Expand Down
8 changes: 4 additions & 4 deletions blog/2022-02-02-image-processing-subworkflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,20 @@ In our 2nd post, we [utilized a fork to create two images in parallel](image-pro
<img src="/content/img/blogassets/workflow_fork.png" width="400" style={{paddingBottom: 40, paddingTop: 0}} />


In both of these workflows, two tasks are reused: ```image_convert_resize``` and ```upload_toS3```. This is onegreat advantage of using microservices - we create the service once, and reuse it many times in different ways.
In both of these workflows, two tasks are reused: ```image_convert_resize``` and ```upload_toS3```. This is one great advantage of using microservices - we create the service once, and reuse it many times in different ways.

In this post, we'll take that abstraction a step further, and replace the tasks in the two forks with a ```SUB_WORKFLOW```. This allows us to simplify the full workflow by abstracting a frequently used set of tasks into a single task.

<!--truncate -->

## Subworkflow

A [subworkflow](https://orkes.io/content/docs/reference-docs/sub-workflow-task) is exactly what it sounds like: a fully functional workflow called inside a larger workflow.
A [subworkflow](/content/docs/reference-docs/sub-workflow-task) is exactly what it sounds like: a fully functional workflow called inside a larger workflow.

There are a number of advantages to calling a sub-workflow:

1. Simplicity. In the current case, we replace 2 tasks with one workflow. In more complicated examples, the definition will be even further simplified.
2. Continutity. Our [simple workflow](image-processing-workflow-with-conductor) and [forked workflow](image-processing-multiple-images-forks) use the same tasks. If a change is made to the simple workflow (but not the forked one) - the two workflows are now different - perhaps unintentionally. If we utilize the ```simple workflow``` as a subworkflow - any changes to the workflow are immediately propagated to the forked workflow with no user intervention.
2. Continuity. Our [simple workflow](image-processing-workflow-with-conductor) and [forked workflow](image-processing-multiple-images-forks) use the same tasks. If a change is made to the simple workflow (but not the forked one) - the two workflows are now different - perhaps unintentionally. If we utilize the ```simple workflow``` as a subworkflow - any changes to the workflow are immediately propagated to the forked workflow with no user intervention.

## Creating our Subworkflow

Expand Down Expand Up @@ -142,6 +142,6 @@ Converting the 2 webp tasks to a subworkflow is not included in this post, but b

## Conclusion

Integrating subworkflows can simplify your workflow, allowing you to extract (and reuse) complex steps thhroughout your workflow. They also have the added advantage of beaing easily updated - one change to the workflow will propgate automatically to every location it is referenced in all of your orchestration.
Integrating subworkflows can simplify your workflow, allowing you to extract (and reuse) complex steps throughout your workflow. They also have the added advantage of being easily updated - one change to the workflow will propagate automatically to every location it is referenced in all of your orchestration.


4 changes: 2 additions & 2 deletions blog/2022-02-10-OurFirst-Meetup.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ In early Feb 2022, we had our first meetup on Netflix Conductor: [Using Conducto

We had two excellent talks from Maros Marasalek at FRINX and Nick Tomlin at Netflix.

After the two talks, we had 2 roadmap sessions. Ther first session was by the Netflix Conductor team - where they discussed recent releases, and walked through the Open Source roadmap for the coming months. The second session, by our own Viren Baraiya, introduced Orkes, and our plans for extending Netflix Conductor.
After the two talks, we had 2 roadmap sessions. The first session was by the Netflix Conductor team - where they discussed recent releases, and walked through the Open Source roadmap for the coming months. The second session, by our own Viren Baraiya, introduced Orkes, and our plans for extending Netflix Conductor.


## Conductor in FRINX
Expand All @@ -33,7 +33,7 @@ Our third talk was from the Netflix Conductor team - where they presented the ro

## Introducing Orkes

Finally, one of our founders (and the commiter of the first line of Conductor code) Viren Baraiya presented Orkes and *our* roadmap:
Finally, one of our founders (and the committer of the first line of Conductor code) Viren Baraiya presented Orkes and *our* roadmap:

<iframe width="560" height="315" src="https://www.youtube.com/embed/MnC25X0jtYE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Expand Down
6 changes: 3 additions & 3 deletions blog/2022-03-03-loan-banking-using-conductor.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ tags: [Netflix Conductor, Orkes, Conductor, orchestration, 2022]

## Introduction

As the [Introduction](../docs/introduction) documentation shows, Netflix Conductor can be used for a variety of uses cases, solving complex workflow problems that plague many companies worldwide.
As the [Introduction](/content/docs/introduction) documentation shows, Netflix Conductor can be used for a variety of uses cases, solving complex workflow problems that plague many companies worldwide.

Let's now try to understand how we can use Conductor to solve a Lending problem that exists in the Banking and Fintech sector.

Expand Down Expand Up @@ -60,15 +60,15 @@ If the credit score of the customer is below average score, banks don't give loa

The task `credit_score_risk` fetches data from one of the credit bureau and generates the result on the basis of credit score and other parameters.
Result can be - ` decision : possible` if it possible to give loan and ` decision : reject` if it not possible to give loan.
On the basis of result generated by `credit_score_risk` the [switch_task](../docs/reference-docs/switch-task.md) `credit_result` proceeds the workflow.
On the basis of result generated by `credit_score_risk` the [switch_task](/content/docs/reference-docs/switch-task.md) `credit_result` proceeds the workflow.

In the above diagram, the result generated is `decision : possible` hence, the workflow goes to the task `principal_interest_calculation` (if the result was `decision : reject` the flow would have went to the task `terminate_due_to_bank_rejection` which is a terminate task).

In the task `principal_interest_calculation` the amount, duration, interest and other banking computations are done and in the next step `loan_offered_to_the_customer` details of the loan are sent to the customer and then the customer has to either accept it or reject it.

![](./assets/loan_transfer_Medium.png)

Whatever the Customer decision, on the basis of it the `customer_decision` a [(switch-task)](../docs/reference-docs/switch-task) takes decision of proceeding the workflow. If Customer rejects, the workflow proceeds to the `terminate_due_to_customer_rejection)` a [(terminate-task)](../docs/reference-docs/terminate-task) and the workflow ends. If Customer accepts the loan offered by bank the workflow goes in the direction as shown in the above figure. And in the task `loan_transfer_to_customer_account` money is transferred to the customer's account and workflow ends.
Whatever the Customer decision, on the basis of it the `customer_decision` a [(switch-task)](/content/docs/reference-docs/switch-task) takes decision of proceeding the workflow. If Customer rejects, the workflow proceeds to the `terminate_due_to_customer_rejection)` a [(terminate-task)](/content/docs/reference-docs/terminate-task) and the workflow ends. If Customer accepts the loan offered by bank the workflow goes in the direction as shown in the above figure. And in the task `loan_transfer_to_customer_account` money is transferred to the customer's account and workflow ends.


## Workflow after successful execution
Expand Down
2 changes: 1 addition & 1 deletion docs/codelab/helloworld2.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Hello World Codelab
# Hello World Codelab 2
## Part 2


Expand Down
2 changes: 1 addition & 1 deletion docs/codelab/helloworld3.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Hello World Codelab
# Hello World Codelab 3

We've made it to Part 3! Thanks for keeping at it! What we've covered so far:

Expand Down
2 changes: 1 addition & 1 deletion docs/codelab/helloworld4.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Hello World Codelab
# Hello World Codelab 4


Welcome to part 4 of the codelab.
Expand Down
2 changes: 1 addition & 1 deletion docs/codelab/helloworld5.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Hello World Codelab Part 5
# Hello World Codelab 5

What we've covered so far:

Expand Down