Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nodes hub to store and reuse commonly used node #569

Merged
merged 18 commits into from
Jul 26, 2024
Merged

Nodes hub to store and reuse commonly used node #569

merged 18 commits into from
Jul 26, 2024

Conversation

haixuanTao
Copy link
Collaborator

This PR creates a node hub to store commonly used nodes so that we can make it extremely easy for people to get started with dora.

@Hennzau
Copy link
Collaborator

Hennzau commented Jul 10, 2024

I'm currently working on this PR on the latest version of Dora (0.3.5), so I bumped it from rc0 to release.

Structure

  • node-hub folder containing usefuls nodes:
    • dora-record
    • dora-rerun
    • opencv-video-capture
    • opencv-plot
    • ultralytics-yolo

Each python node respects the following structure:

node-hub/
└── node/
    ├── main.py
    ├── README.mdr
    └── pyproject.toml

The pyproject.toml can be installed with pip (pip install node-hub/node), it installs all dependencies and setup a script to directly execute the node from a python env:

  - id: object-detection
    build: pip install ../../node-hub/ultralytics-yolo
    path: ultralytics-yolo

Usage

cd examples/python-dataflow
dora build ./dataflow.yml (or dora build ./dataflow_dynamic.yml)
dora up
dora start ./dataflow.yml (or dora start ./dataflow_dynamic.yml)

Note: if you're running the dynamic dataflow, you will need to start manually the ultralytics-yolo node:

# activate your virtual environment in another terminal
python ultralytics-yolo --name object-detection --model yolov5n.pt

Sorry, something went wrong.

Copy link
Collaborator Author

@haixuanTao haixuanTao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

I made couple of small comments.

FYI, in a follow up PR we can look at: https://arrow.apache.org/docs/python/extending_types.html#custom-scalar-conversion

We can create a new project called something like pyarrow-extension-types and define some standard format such as ImageRGB, BboxXYXY ... So that, the abstraction of types is more solid than dictionary.

node-hub/ultralytics-yolo/main.py Outdated Show resolved Hide resolved
node-hub/ultralytics-yolo/main.py Outdated Show resolved Hide resolved
node-hub/opencv-plot/main.py Outdated Show resolved Hide resolved
node-hub/opencv-video-capture/main.py Outdated Show resolved Hide resolved
examples/python-dataflow/dataflow_yolo.yml Outdated Show resolved Hide resolved
examples/python-dataflow/run.rs Outdated Show resolved Hide resolved
@Hennzau Hennzau marked this pull request as ready for review July 17, 2024 14:44
Copy link
Collaborator Author

@haixuanTao haixuanTao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Sorry I looked more into details and found more things to update.

node-hub/opencv-plot/main.py Outdated Show resolved Hide resolved
node-hub/opencv-plot/main.py Outdated Show resolved Hide resolved
node-hub/opencv-plot/main.py Outdated Show resolved Hide resolved
node-hub/ultralytics-yolo/main.py Outdated Show resolved Hide resolved
examples/python-dataflow/dataflow_yolo.yml Outdated Show resolved Hide resolved
node-hub/opencv-plot/README.md Outdated Show resolved Hide resolved
node-hub/opencv-plot/README.md Outdated Show resolved Hide resolved
examples/python-dataflow/dataflow.yml Outdated Show resolved Hide resolved
Copy link
Collaborator

@phil-opp phil-opp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please squash the commits a bit before merging, we don't need all those "Fix CI" commits in our git history.

examples/python-dataflow/README.md Outdated Show resolved Hide resolved
@haixuanTao haixuanTao force-pushed the nodes_hub branch 2 times, most recently from 07229c4 to 6e508ec Compare July 19, 2024 15:06
@Hennzau Hennzau self-requested a review July 19, 2024 21:33
Copy link
Collaborator

@Hennzau Hennzau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything is good for me. Maybe we should think about the ZeroCopy dictionary issue for a next PR (or this one), tell me your ideas when you are working on it 😄

haixuanTao and others added 18 commits July 26, 2024 09:37

Verified

This commit was signed with the committer’s verified signature.
evenyag Yingwen
.idea

.idea

README.md and better outputs/inputs

README.md and ultralytics-yolo

fix queue_size and dynamic

README.md

Fix CI

Fix CI

Fix CI

Fix CI

Fix CI

Fix CI

Fix CI

Fix CI

Fix CI
…etadata as well as enable more flexibility in managing metadata
…inition to make our input more generalistic.

Rewrite README documentation to reflect metadata changes
@haixuanTao haixuanTao enabled auto-merge July 26, 2024 07:44
@haixuanTao haixuanTao merged commit 42f1c7e into main Jul 26, 2024
38 checks passed
@haixuanTao haixuanTao deleted the nodes_hub branch July 26, 2024 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants