Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Add ability to schedule a datasource ingestion job #8

Closed
mawandm opened this issue Apr 6, 2024 · 0 comments · Fixed by #27
Closed

[Feature] Add ability to schedule a datasource ingestion job #8

mawandm opened this issue Apr 6, 2024 · 0 comments · Fixed by #27
Assignees
@mawandm
Labels
enhancement New feature or request

Comments

@mawandm
Copy link
Contributor

mawandm commented Apr 6, 2024

Currently all ingestion jobs run on the same schedule. This isn't practical as some document repositories might be expensive to access and or are not updated frequently and it doesn't scale.

This feature introduces the ability to schedule an ingestion job at a regular interval. The schedule should be expressible in a unix cron format.

Acceptance Criteria

  1. An endpoint for scheduling. This will be /v1/tasks. It should accept POST, PUT, GET, DELETE
  2. Introduce RBAC policy to allow for creating tasks. The RBAC resource id is in the format tasks/*
@mawandm mawandm added the enhancement New feature or request label Apr 6, 2024
@mawandm mawandm changed the title Add ability to schedule a datasource ingestion job [Feature] Add ability to schedule a datasource ingestion job Apr 6, 2024
@mawandm mawandm self-assigned this Apr 16, 2024
This was referenced Apr 16, 2024
Merged
Merged
mawandm added a commit that referenced this issue Apr 18, 2024
@mawandm
fix(api): rename llm endpoint (#28)
36b917c 
In this PR, we refactor the llm endpoint to rag endpoint

Part of #8
mawandm added a commit that referenced this issue Apr 19, 2024
@mawandm
feat(api)(frontend): add task scheduling feature (#27)
5099ffb 
This PR introduces the concept of tasks within the API module. A use case for task is running an ingestion pipeline from a datasource.

The result of this PR is that we will not need to current uniform scheduling mechanism. Rather, users are able to pick when to schedule the ingestion of a specific datasource.

Closes of #8
@mawandm mawandm closed this as completed Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mawandm mawandm

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(api)(frontend): add task scheduling feature ametnes/nesis
1 participant
@mawandm