🧑‍✈️ Run AWS SageMaker pilot for our users

[bagg3rs]

User Story

As a Data Scientist
I want to explore using LLM Large Language Models and GPU backed instances
So that there is a secure managed location to explore using these tool which can aid MoJ's data driven strategy

Context

Our user want analytical features not available on our existing platform. The types of tools and underlying compute is changing rapidly. SageMaker provides a managed service for these tools and provides instances with higher resources and GPU to speed and aid research.

There is also a lot of interest in using LLMs for e.g.semantic search of free text. SageMaker in VPC isolation mode makes sure sensitive workloads are secured and stay within the instance, we can further secure data using private VPC and PrivateLink

Value and consequences

• SageMaker costs are based on usage and can vary significantly month-to-month depending on your application's usage, instance type. We can provide proactive notifications that will for the first time allow our users to understand the cost of the work that they are doing
• Improved Security, workloads will run in isolation mode (without access to internet) to further secure sensitive data and reduce fear of data exposure for Data Owners
• Reduced operational cost and complexity
• Agility and change readiness, additional analytical services can be offered when available without considerable effort that has lead to users having to work elsewhere (extensive development of front and backend services) e.g. Control Panel
• Better cost transparency, we will understand our tooling compute costs which is currently very difficult to calculate
• For Foundation Models, SageMaker JumpStart does not download models from a public model zoo, it can be used in fully locked-down e.g. no internet access
• Network access can be limited and scoped down for SageMaker JumpStart models, this helps teams improve the security posture of the environment
• Due to the VPC boundaries, access to the endpoint can also be limited via subnets and security groups, which adds an extra layer of security
• Leverage managed services like SageMaker Studio
• If successful we can close down our current tooling EKS. EKS although managed still requires a considerable amount of effort to run with the endless upgrades and get on with more useful tasks

Disadvantages

• RStudio on Amazon SageMaker is a paid product and requires that each user is appropriately licensed. As part of the pilot we will need to understand our users need for RStudio

Hypothesis

If we provide tooling to our data scientists
Then they can tell better stories about our data

Proposal

👥 AWS onsite session with DP and Data Scientist ✅
🚗 Deploy SageMaker with Terraform in compute account
👤 Create any required roles?
🎛️ Add to Control Panel
🚀 Release to users
🗣️ Gather user feedback

Definition of done

• Send over to AWS current number of RStudio and JupyterLabs users.
• Arrange dates with AWS and get out invites (Mon 27th Nov)
• If successful complete spike 🌱 Spike: SageMaker #1922
• Cost analysis explored aws cost here vs EKS GPU instances AWS mention 54% lower TCO over 3yrs.
• Feedback from users captured from pilot

Reference

POC implementation was implemented Control Panel see closed PR

How to write good user stories

[alex-vonfeldmann]

I'm not sure this requires UCD at this point, the Data Science guys were pretty clear what they want (which is better infrastructure that supports LLM). @bagg3rs Am I wrong?

Does this fall under AP BAU or DP?

[bagg3rs]

@alex-vonfeldmann correct no UCD is required at this time (removed label), we will gather feedback once we start PoC.
This falls DP? as it looks at potentially replacing all current AP managed tooling (Jupyter/RStudio).

[bagg3rs]

referencing this feature request as it could be a good way to host vscode / coder

[github-actions]

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.

[github-actions]

This issue is being closed because it has been open for a further 7 days with no activity. If this is still a valid issue, please reopen it, Thank you!