Skip to content
This repository has been archived by the owner on Aug 14, 2023. It is now read-only.

Create jobs per media file, not per media #4

Open
4 tasks
jaredallard opened this issue Jul 23, 2019 · 0 comments
Open
4 tasks

Create jobs per media file, not per media #4

jaredallard opened this issue Jul 23, 2019 · 0 comments
Labels
enhancement New feature or request

Comments

@jaredallard
Copy link
Contributor

We should stop creating a single media job for one media item. This causes scale problems with massive media collections requiring converter instances to download the entire media before starting (on every retry).

We should, instead, create a job for every single media file in a logical "media" object.

Work to do

  • Twilight: Make stateless (don't do any ++ logic, since ordering is no longer guaranteed)
  • Downloader: Create job during upload loop, not in main.js
  • Converter: support actually almost exists, so restructure code to handle just one.
  • core: add v2.converter object that is geared towards a single file..
@jaredallard jaredallard added the enhancement New feature or request label Jul 23, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant