Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DAG flow #9

Merged
merged 3 commits into from
Apr 26, 2024
Merged

Add DAG flow #9

merged 3 commits into from
Apr 26, 2024

Conversation

Spycsh
Copy link
Member

@Spycsh Spycsh commented Apr 26, 2024

Type of Change

feature

Description

Microservices dataflow is organized in a DAG, we expose two ways: Python API or Yaml config to define that. Further features are to be done including Ingress/Egress, megaservice API etc.
usage e.g.

Yaml

    - (s1, s5) >> s2
    - s2 >> (s3, s4)
    - s3 >> s4

equivalent to

Python

service_builder = ServiceBuilder(port=1234, hostfile=None)
s1 = BaseService(id="s1", endpoint="http://localhost:8081/v1/add")
s2 = BaseService(id="s2", endpoint="http://localhost:8082/v1/add")
s3 = BaseService(id="s3", endpoint="http://localhost:8083/v1/add")
s4 = BaseService(id="s4", endpoint="http://localhost:8084/v1/add")
s5 = BaseService(id="s5", endpoint="http://localhost:8085/v1/add")
service_builder.add(s1).add(s2).add(s3).add(s4).add(s5)
service_builder.flow_to(s1, s2)
service_builder.flow_to(s5, s2)
service_builder.flow_to(s2, s3)
service_builder.flow_to(s2, s4)
service_builder.flow_to(s3, s4)

How has this PR been tested?

local test

Dependency Change?

None

@lvliang-intel lvliang-intel merged commit b3f83dd into main Apr 26, 2024
3 checks passed
lkk12014402 pushed a commit that referenced this pull request Jul 22, 2024
lkk12014402 pushed a commit that referenced this pull request Aug 8, 2024
* add guardrails

Signed-off-by: lvliang-intel <[email protected]>

* fix dockerfile

Signed-off-by: lvliang-intel <[email protected]>

* fix integration issues

Signed-off-by: lvliang-intel <[email protected]>

* add package lock json file

Signed-off-by: lvliang-intel <[email protected]>

---------

Signed-off-by: lvliang-intel <[email protected]>
@chensuyue chensuyue deleted the genaicomps_flow branch August 19, 2024 02:16
Spycsh pushed a commit that referenced this pull request Nov 8, 2024
* Adds an endpoint for image ingestion

Signed-off-by: Melanie Buehler <[email protected]>

* Combined image and video endpoint

Signed-off-by: Melanie Buehler <[email protected]>

* Add test and update README

Signed-off-by: Melanie Buehler <[email protected]>

* fixed variable name for embedding model (#1)

Signed-off-by: okhleif-IL <[email protected]>

* Fixed test script

Signed-off-by: Melanie Buehler <[email protected]>

* Remove redundant function

Signed-off-by: Melanie Buehler <[email protected]>

* get_videos, delete_videos --> get_files, delete_files (#3)

Signed-off-by: okhleif-IL <[email protected]>

* Updates test per review feedback

Signed-off-by: Melanie Buehler <[email protected]>

* Fixed test

Signed-off-by: Melanie Buehler <[email protected]>

* Add support for audio files multimodal data ingestion (#4)

* Add support for audio files multimodal data ingestion

Signed-off-by: dmsuehir <[email protected]>

* Update function name

Signed-off-by: dmsuehir <[email protected]>

---------

Signed-off-by: dmsuehir <[email protected]>

* Change videos_with_transcripts to ingest_with_text

Signed-off-by: Melanie Buehler <[email protected]>

* Add image support to video ingestion with transcript functionality

Signed-off-by: Melanie Buehler <[email protected]>

* Update test and README

Signed-off-by: Melanie Buehler <[email protected]>

* Updated for review suggestions

Signed-off-by: Melanie Buehler <[email protected]>

* Add two tests for ingest_with_text

Signed-off-by: Melanie Buehler <[email protected]>

* LVM TGI Gaudi update for prompts without images (#7)

* LVM Gaudi TGI update for prompts without images

Signed-off-by: dmsuehir <[email protected]>

* Wording

Signed-off-by: dmsuehir <[email protected]>

* Add a test

Signed-off-by: dmsuehir <[email protected]>

---------

Signed-off-by: dmsuehir <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change dummy image to be b64 encoded instead of the url (#9)

Signed-off-by: dmsuehir <[email protected]>

* Updates based on review feedback (#10)

Signed-off-by: dmsuehir <[email protected]>

* Test fix (#11)

Signed-off-by: dmsuehir <[email protected]>

---------

Signed-off-by: Melanie Buehler <[email protected]>
Signed-off-by: okhleif-IL <[email protected]>
Signed-off-by: dmsuehir <[email protected]>
Co-authored-by: dmsuehir <[email protected]>
Co-authored-by: Omar Khleif <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants