Enable users to register Processing Services & Pipelines

[vanessavmac]

Summary of Changes

• Users can register an ML Processing Service (a collection of pipelines)
• Register pipelines to the Processing Service using an 'Register Pipelines' button
• To process an image, users select the pipeline. The job checks which services are online before selecting one with lowest latency/response time.

Example User Flow

Users can register processing services associated with an endpoint URL. The endpoint URL has an info endpoint which returns a list of pipelines that should be in this processing service. The "Register Pipelines" button pings the info endpoint and adds the pipelines and algorithms to the processing service.

Pipelines are listed in the overview page along with the number of processing services available.

When a processing service is available, the user can select a pipeline to process a capture with. The processing service with the lowest latency is used to process the images with the selected pipeline.

Follow-Up ToDos (Requires feature spec-ing)

• Revamp the processing page.
  • Currently, users select a pipeline and the function choose_processing_service_for_pipeline() selects a service that has the lowest latency. The UI could display what services are online and prevent users from using pipelines with no available processing services.
  • Recommend better pipelines (i.e. most recently checked, lowest latency, etc)
• Come up with a better solution than "Stages" for showing the sequential order of algorithms in a pipeline. This processing service feature PR just uses the default values for "Stages."

Follow-Up Issues

• ami/ml/models/pipeline.py Address the TODO (# TODO the algorithms list be retrieved by querying the pipeline endpoint) Update Pipeline model Algorithm Field #681
• ami/ml/views.py Rather than select a random image to test the pipeline processing (test_process), filter images by projects user has access to. Improve test_process pipeline view #680
• Currently we just add and create pipelines. Once pipelines are added, users cannot at the moment modify the pipelines in that processing service. For example, if the user registers pipelines then changes the info endpoint on the service, when the user registers pipelines again, it will only add additional pipelines (pipelines won't be deleted or updated). Extend Register Pipelines Functionality #678
• Use celery beat for periodic task to check when processing services are last seen online (similar to this http://localhost:8000/admin/django_celery_beat/periodictask/1/change/) Create periodic task to check processing services #676
• Add section to pipeline details with a json field of pipeline parameters Add section to pipeline details with a json field of pipeline parameters #677

[netlify]

✅ Deploy Preview for ami-dev ready!

Name	Link
🔨 Latest commit	fb0539a
🔍 Latest deploy log	https://app.netlify.com/sites/ami-dev/deploys/678c85ac5ff0fa0008dbac78
😎 Deploy Preview	https://deploy-preview-632--ami-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 73 Accessibility: 89 Best Practices: 92 SEO: 100 PWA: 80 View the detailed breakdown and full score reports

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[vanessavmac]

@mihow I'm getting an error for this test in ami-platform/ami/jobs/tests.py:

def test_run_job_unauthenticated(self):
        jobs_run_url = reverse_with_params("api:job-run", args=[self.job.pk])
        self.client.force_authenticate(user=None)
        resp = self.client.post(jobs_run_url)
        self.assertEqual(resp.status_code, 401)

The console logs:

======================================================================
FAIL: test_run_job_unauthenticated (ami.jobs.tests.TestJobView.test_run_job_unauthenticated)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/app/ami/jobs/tests.py", line 176, in test_run_job_unauthenticated
    self.assertEqual(resp.status_code, 401)
AssertionError: 403 != 401

----------------------------------------------------------------------

I think this error is occurring now after I modified the image processing to use the choose_backend_for_pipeline() function in ami-platform/ami/ml/models/pipeline.py. However, I'm not sure why exactly it's occurring. I think it might have to do with how the job status is determined?

Does this only happen when you have SessionAuthentication commented-in in the base.py settings?

Forgot that that was not commented out -- the tests pass when I comment out "rest_framework.authentication.SessionAuthentication",

[mihow]

@vanessavmac I merged main into this branch and fixed several conflicts from other updates that have happened. I got into the groove and made several small changes & fixes as well. I think tomorrow we should do one more pass and test on a snapshot of the live database. Then hopefully merge!