🛡️ Investigate IAM Identity Center with Lake Formation + tags for access management

[bagg3rs]

User Story

As a Analytical Platform user
I want to use one identity to access data resources
So that accessing and requesting access to data and using analytical applications is simplified

Value / Purpose

We need the availability of IAM Identity Centre being integrated with EntraID with groups to allow and test tagged based access

IAM Identity Center along with identity propagation and Lake Formation tags should give the Data Platform Service area a unified way to grant access to AWS services including S3, Athena, QuickSight and Glue Catalogue.

• Allow the use of our users main identity to access data resources
• Manage access to AWS services, without creating multiple IAM roles
• Row and column LF-Tags based permissions to remove sensitive information without the need to e.g. duplicate etc
• Query datasets across accounts with Lake Formation
• Dynamically Manage permissions based on users EntraID group membership
• Simplified auditing with CloudTrail
• Allow Data Owners to control access to their assets in conjunction with 🔄 Revisit AWS DataZone #3843
• Single Entra app registration and certificate to manage access for our applications and services and other teams also ✨

Useful Contacts

RichB, Julia

Hypothesis

If we use AWS IDAM Identity Centre and Lake Formation
Then we can simplify access management for Data Platform services

Proposal

Can IAM Identity Center can be the centre of Data Platform services?

• Create a MVP lake formation tag strategy, get feedback from DE. Whats the smallest number needed see here for example
• demonstrate SSO user with QuickSight to S3 bucket access using LM-Tags with identity propagation
• Use Lake Formation tags to control access to folders within S3 buckets and test fine-grained access policies based on tags.
• Use tags for RLS and CLS
• Test cross account access and think about how this could allow direct owner to consumer (this links into 🔄 Revisit AWS DataZone #3843 ) with sso

Additional Information

• AWS re:Invent 2023 - Simplify and improve access control for your AWS analytics services (SEC245)
• Why use LF-Tags
• Lake formation tags best practices and from aws
• IAM Identity Center integration limitations

Definition of Done

• Create ADR on using AWS Identity Centre for accessing platform, data and services
• Create ADR on using lake formation tags

[michaeljcollinsuk]

Slack thread of initial discussions with the team https://mojdt.slack.com/archives/C04M8224WCV/p1713364819258839

Summary of my initial thoughts:

• Lake formation feels more like a solution for managing access to data-engineering databases, but may not work for the "data warehouses" functionality that CP offers.

"AWS Lake Formation is for use cases where you need to manage access for tabular data (e.g., Glue tables), where you might want to enforce row- and column-level access."

• Also unsure if this would be compatible with our current tooling offering (Rstudio/Jupyter/Vscode) as the docs for the hybrid workflow state describe an example where querires come from an AWS managaged service:

"using an integrated service such as Amazon Athena, AWS Glue, Amazon EMR, or Amazon Redshift Spectrum"

• S3 Access grants may be a way of managing access to S3 "data warehouses" rather than managing users IAM roles.

[michaeljcollinsuk]

Some thoughts/discussion on Identity Centre with the team in https://mojdt.slack.com/archives/C04M8224WCV/p1713537790475079

[jacobwoffenden]

Closing as per https://mojdt.slack.com/archives/C04M8224WCV/p1714549958985199?thread_ts=1714549071.413819&cid=C04M8224WCV