RFC: Sensitive data masking utility

[seshubaws]

Is this related to an existing feature request or issue?

#1173

Which AWS Lambda Powertools utility does this relate to?

Other

Summary

Customers would like to obfuscate incoming data for known fields that contain PII, so that they're not passed downstream or accidentally logged. With the increase of batch processing utilities and GDPR, this is one of the hardest tasks for customers today, specially when considering multi-account users.

AWS Encryption SDK is a good starting point but it can be too complex for the average developer, data engineer, or DevOps persona to use. As such, it is highly requested that the Powertools library have a utility to easily mask and/or encrypt sensitive data.

Use case

The use case for this utility would be for developers who want to mask or encrypt sensitive data such as names, addresses, SSNs, etc. in order for them to not be logged in CloudWatch so such data is not compromised, and so that downstream systems like S3, DynamoDB, RDBMS, etc. will not need any additional work on handling PII data.

Additionally, developers should be able to recover encrypted sensitive data to its original form so that they can handle sensitive requests around that data on a as-needed basis.

Proposal

The data masking utility should allow users to mask data, or encrypt and decrypt it. If they would like to encrypt their data, customers should be able to decide for themselves which encryption provider they want to use, though we will provide an out-of-the-box integration with the AWS Encryption SDK. The below code snippet is a rudimentary look at how this utility can be used and how it will function.

Usage

from aws_lambda_powertools.utilities.data_masking.constants import KMS_KEY_ARN
from aws_lambda_powertools.utilities.data_masking import DataMasking
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import AwsEncryptionSdkProvider


def lambda_handler():
    
   data_masker = DataMasking()
    data = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "[email protected]",
               "address": {
                    "street": "123 Main St", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }

    masked = data_masker.mask(data=data, fields=["email", "address.street"])
    """
    masked = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "*****",
               "address": {
                    "street": "*****", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }
   """

    encryption_provider = AwsEncryptionSdkProvider(keys=[KMS_KEY_ARN])
    data_masker = DataMasking(provider=encryption_provider)

    encrypted = data_masker.encrypt(data=data, fields=["email", "address.street"])
    """
    encrypted = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "InRoaXMgaXMgYSBzdHJpbmciHsLZGx2na-XzP_TB5Bf2LNU1bLc",
               "address": {
                    "street": "XMgYSB_KDddaDJYMb-JpbmGnagTklwQ-msdaDLP", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }
   """
 
    decrypted = data_masker.decrypt(data=encrypted, fields=["email", "address.street"])
    """
    data = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "[email protected]",
               "address": {
                    "street": "123 Main St", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }
   """

AWS Encryption SDK

The AWS Encryption SDK is a client-side encryption library that makes it easier to encrypt and decrypt data of any type in your application. The Encryption SDK is available in all the languages that Powertools supports. You can use it with customer master keys in AWS Key Management Service (AWS KMS), though the library does not require any AWS service. When you encrypt data, the SDK returns a single, portable encrypted message that includes the encrypted data and encrypted data keys. This object is designed to work in many different types of applications. You can specify many of the encryption options, including selecting an encryption and signing algorithm.

Latencies
Latencies for using this utility with the AWS Encryption SDK in Lambda functions configured with 128MB, 1024MB, and 1769MB, respectively.

plugins.metrics-by-endpoint.response_time./Prod/function128:
  min: ......................................................................... 0
  max: ......................................................................... 5373
  median: ...................................................................... 561.2
  p95: ......................................................................... 713.5
  p99: ......................................................................... 1620
plugins.metrics-by-endpoint.response_time./Prod/function1024:
  min: ......................................................................... 0
  max: ......................................................................... 5039
  median: ...................................................................... 89.1
  p95: ......................................................................... 144
  p99: ......................................................................... 232.8
plugins.metrics-by-endpoint.response_time./Prod/function1769:
  min: ......................................................................... 0
  max: ......................................................................... 1726
  median: ...................................................................... 82.3
  p95: ......................................................................... 133
  p99: ......................................................................... 183.1

Custom encryption
If customer would like to use another encryption provider, or define their own encrypt and decrypt functions, we will define an interface that the customer can implement and pass in to the DataMaskingUtility class.

from aws_lambda_powertools.utilities.data_masking.provider import BaseProvider
from itsdangerous.url_safe import URLSafeSerializer

class MyCustomEncryption(BaseProvider):
    def __init__(self, secret):
        self.secret = URLSafeSerializer(secret)

    def encrypt(self, value: str) -> str:
        if value is None:
            return value
        return self.secret.dumps(value)

    def decrypt(self, value: str) -> str:
        if value is None:
            return value
        return self.secret.loads(value)


def lambda_handler():
    data = {
        "id": 1,
        "name": "John Doe",
        "age": 30,
        "email": "[email protected]",
        "address": {
            "street": "123 Main St", 
            "city": "Anytown", 
            "state": "CA", 
            "zip": "12345"
        },
    }

    masking_provider = MyCustomEncryption(secret="secret-key")
    data_masker = DataMasking(provider=masking_provider)

    encrypted = data_masker.encrypt(data, fields=["email", "address.street"])
    """
    encrypted = {
        "id": 1,
        "name": "John Doe",
        "age": 30,
        "email": "InRoaXMgaXMgYSBzdHJpbmciHsLZGx2na-XzP_TB5Bf2LNU1bLc",
        "address": {
            "street": "XMgYSB_KDddaDJYMb-JpbmGnagTklwQ-msdaDLP", 
            "city": "Anytown", 
            "state": "CA", 
            "zip": "12345"
        },
    }
   """

   decrypted = data_masker.decrypt(data=encrypted, fields=["email", "address.street"])
    """
    decrypted = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "[email protected]",
               "address": {
                    "street": "123 Main St", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }
   """

Out of scope

Traversing an arbitrary dictionary will be out of scope for the initial launch of this tool. This feature will be to receive instructions as to where in the given dictionary it should mask/unmask the data.

We still need to determine the most efficient method of taking input JSON path masking or encrypting the value at that path. JMESPath or JSON path can be considered for simple use cases but we need to find the fastest method.

Potential challenges

• We will need to discuss the design and implementation with a security expert to ensure safety.
• Also need to determine if we should support envelope encryption for customers using the AWS Encryption SDK
  • Envelope encryption is the practice of encrypting plaintext data with a data key, and then encrypting the data key under another key. But, eventually, one key must remain in plaintext so you can decrypt the keys and your data. This top-level plaintext key encryption key is known as the root key. You can store root keys in AWS KMS, known as AWS KMS keys.
• Need to decide what the best name for the methods are. Should they still be called encrypt and decrypt even in the case where the customer only masks and won't be able to decrypt later? Should we have a mask method in the interface that is always accessible so that users can irretrievably mask some data and also encrypt some data?

Dependencies and Integrations

Integration with the AWS Encryption SDK.

Alternative solutions

No response

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[heitorlessa]

Prioritising for review tomorrow - missed our typical response time due to team offsite.

We'll revert with comments by tomorrow COB

[rubenfonseca]

Hi @seshubaws thank you for your great work on the RFC. I especially liked the way you already pre-tested some solutions to find some bottlenecks.

I have two concerns:

• it seems to me the "JSON path" method to identify fields that can contain PII is very limiting, compared to the offer we have on CloudWatch right now. With CloudWatch you instead declare categories of PII you are interested in masking, and AWS will mask automatically no matter where they appear. I believe Heitor's proposal where we use pydantic's parser could be a little better, but still do you think we can inverse the responsibilities here, and help the user declare "what to mask" vs "where to mask"?

• regarding ItsDangerous, I understand the benefit of having something simpler. However, handling private keys and secrets on our own seems scary to me, and I would much prefer to build on top of the existing SDK and mechanisms, even if it means it will be slower. Alternatively, ItsDangerous could probably be used as long as we leave all the secret management to KMS.

Last, one thing I would love to see, is to augment objects that have PII with the corresponding encrypted version. Example:

{ "email": "[email protected]", name: "John" }

If we are interested in masking emails, this could turn into

{ "email": "*** 1 ***", name: "John", "*** 1 ***": "encrypted/signed version of the PII" }

This way, anyone with access to the decryption keys would be able to just augment the object with the original data, instead of having to find it on external systems.

[heitorlessa]

This is great, I'm so excited we're moving along with this now. A few immediate comments:

• We should strive for cross-language compatibility. It's not a hard requirement, but it'd be great if ~50-70% is achievable to aid feature parity in the future. Other Powertools languages will follow the docs and features, it doesn't have to be a 1:1 mapping, otherwise we risk becoming half-good everywhere - we delight customers in their language of choice.
  • Q: For AST, are you using stdlib ast or a 3rd party library? If the latter, what's their license and package size?
• Batteries included but swappable. I agree with Ruben that AWS Encryption SDK however heavy it addresses common security concerns we are not typically aware. We should be mindful to support 80% of customers looking for correctness and security while also allowing 20% of customers to bring their own implementation (e.g. ItsDangerous is everywhere) without bloating their dependencies with our recommended choice. For example, ItsDangerous is roughly 36K zipped while AWS Encryption SDK brings 18M zipped, largely due to AWS SDK and other heavy hitters. Encryption SDK alone might make it difficult for some Data Engineers/Scientists to use it, hence having a Provider helps us meet them where they are.
  • Q: Could you include what the experience would look like to bring-your-own-masking provider?
  • Q: Based on your research with both Encryption SDK and ItsDangerous, what common nomenclature could we use for interfaces? e.g., sign, dumps, etc.
• What vs Where to mask. Unless Ruben has suggestions on how to address it for customers not using Parser (SRE, DevOps, Data Scientists), we'd want to support - however, it doesn't have to be at launch. [What] We could start with the Pydantic above and focus on getting a Provider interface, and correctness approach first. [Where] Once that's out of the picture, we can hear from customers on how they'd plan to use the JSON Path-like to scrub data recursively.
  • NOTE: They should be easily composable, since one traverses and one mask. If one doesn't have a dict data structure, you simply use a Provider to mask data.
• Value of CloudWatch Logs PII. We recently announced native support for masking sensitive data in CloudWatch Logs, quite similarly to SNS. Part of Powertools exists because of integration and feature gaps between services, however, we should not assume that everyone using Lambda will want their logs dumped in CloudWatch - see Observability providers top demanded feature. Instead, we should recognize the plurality of customers we have. Depending on volume and need, it might be more economically viable to use this feature vs a managed service.
  • NOTE: Alternatively, if a customer wants to mask PII before they send it to downstream services via REST/GraphQL APIs, or non-SNS messaging services (EventBridge, Kinesis, SQS, Kafka, MQ), this is still a value add for them to rely on this feature. Over time, we will continue to voice our customers feedback to make it into a native feature to reduce operational overhead - until then, I'd be happy to hear otherwise why not.

On UX:

• A draft proof of concept would be great to improve this discussion.
• We'd want to avoid high-level top imports to prevent cold starts, like what happened to Tracer and thus requiring additional logic for lazy imports. Instead, use aws_lambda_powertools.utilities.data_masking.
• Let's not mutate implicitly and always return the data masked, despite Python using pass-by-reference. This makes it easier for cross-languages too.
• Prefer data parameter over event parameter. This enables a near future where we officially support this feature beyond Lambda runtimes - this feature could easily be used anywhere.
• Customers have more than a Dict, so let's account for those too. This makes it easier to interop with existing systems they might have too to ease migration (on/off), if necessary - e.g., Signer.sign("[email protected]")

Super looking forward to this.

Thank you for all the hard work you already put into this.

[lpsuerj]

I'm excited about all the possibilities we can uncover with this feature on the security front! Few things that come to my mind about it:

1. When logs are not configured to be sent to CloudWatch logs, we need to ensure there's a way to show evidence of security control implementation effectiveness. Encryption of sensitive data is a high priority for security audit teams and to enabled an easy path to comply with regulator requirements (internal and external) would be a big win.

2. AWS is broadly adopting a new security industry-wide logging format called OCSF. I't be great if we could embrace the same format for a future potential integration with services like AWS Security Lake for encryption events logs. Ideally we should still log the encryption activities following defined patterns in the Encryption SDK. This is NOT a hard requirement but a great nice to have

I'm looking forward to deep dive in this topic and join the team bringing this to life!

[ran-isenberg]

Hey, really liking this idea!
I'd like to share my two cents as this is an issue I also deal with in my day to day.
I'd usually use a KMS CMK and boto to encrypt PII and in many cases either the customer brings its own key or i have a key per tenant but the encryption itself is required to be FIPS certified and other goodies.
For safe logging, if you parse your input with the Parser utility you can define any sensitive information as SecretStr (https://docs.pydantic.dev/usage/types/#secret-types) which causes it to printed as '****' when serialized to a string in the logger.

[seshubaws]

@rubenfonseca Thank you for your comments, let me know if I haven't answered all your questions!

It seems to me the “JSON path” method to identify fields that can contain PII is very limiting, compared to the offer we have on CloudWatch right now. With CloudWatch you instead declare categories of PII you are interested in masking, and AWS will mask automatically no matter where they appear. I believe Heitor’s proposal where we use pydantic’s parser could be a little better, but still do you think we can inverse the responsibilities here, and help the user declare “what to mask” vs “where to mask”?

Yes, that makes sense to me! I played around a little with CW’s new data masking tool and it seems they might be using regex for pattern matching for the desired categories users want to mask, but not exactly sure (let me know if you know what how they implemented this). I’m not very familiar with Pydantic but if it will help users identify more generic PII fields, we should definitely use that.

regarding ItsDangerous, I understand the benefit of having something simpler. However, handling private keys and secrets on our own seems scary to me, and I would much prefer to build on top of the existing SDK and mechanisms, even if it means it will be slower. Alternatively, ItsDangerous could probably be used as long as we leave all the secret management to KMS.

Yes, I think this goes along with what Heitor mentioned about designing a bring-your-own-masking-provider type interface. As both the AWS Encryption SDK and ItsDangerous have their tradeoffs, we should leave it up to the customer to decide which would fit best with their specific use case. I’ll work on coming up with a UX draft for this.

Last, one thing I would love to see, is to augment objects that have PII with the corresponding encrypted version. Example:

We could definitely do this! I am a little confused on what use case would require a field to be masked and also encrypted though, as I am under the impression that if it’s encrypted, then it’s just as good as being masked with the added benefit of only users with access can decrypt it.

[seshubaws]

Had an offline discussion with @heitorlessa (thanks for the thorough review and comments Heitor!) and thought I'd add it here for the public:

What vs Where to mask. Unless Ruben has suggestions on how to address it for customers not using Parser (SRE, DevOps, Data Scientists), we’d want to support - however, it doesn’t have to be at launch. [What] We could start with the Pydantic above and focus on getting a Provider interface, and correctness approach first. [Where] Once that’s out of the picture, we can hear from customers on how they’d plan to use the JSON Path-like to scrub data recursively.

Seshu: So do we want to launch first with customers choosing categories of things to mask like what CW is offering right now? ie categories like DOB, credit cards, SSN? Or would they be more general categories like Financial?

Heitor: Categories will prolly have false positives, so I’d wait until customers actually request it so we can brainstorm a bit more. Right now, masking any input that can be unmasked with appropriate permissions later is the way forward — for irreversibly masking (****), I’m still torn on whether we do this now with AST, or simply recommend Pydantic SecretStr for now until other languages have bandwidth to port this utility.

Basic UX of this:

from pydantic import BaseModel, validator

def obfuscate_forever(value: str) -> str:
    return value.replace(value, "****")


data = {
    "username": "johndoe",
    "password": "secretpassword"
}

class Policy(BaseModel):
    username: str
    password: str
    
    _irreversibly_obfuscate_password = validator('password', allow_reuse=True)(obfuscate_forever)  # Pydantic will call `obfuscate_forever` as soon as it traverses the tree and finds the password attribute

user = Policy(**data)
print(user)
# Output: {'username': 'johndoe', 'password': '******'} 

This means we can use the same mechanic to provide an actual Obfuscator who can encrypt data at any node in the tree, and customers can deobfuscate by using the same method in reverse (Obfuscator.unmark, etc.).

Using the sample example but with SecretStr:

from pydantic import BaseModel, SecretStr

data = {
    "username": "johndoe",
    "password": "secretpassword"
}

class Policy(BaseModel):
    username: str
    password: SecretStr  # Policy.password.get_secret_value() can get the real value


user = Policy(**data)
print(user)
# Output: {'username': 'johndoe', 'password': '******'} 

[seshubaws]

@heitorlessa Responding to the rest of your questions here!

Q: For AST, are you using stdlib ast or a 3rd party library? If the latter, what’s their license and package size?

I am using only the stdlib ast, but just to call out that python3.9 or later is needed for what we’re using from this library, though I don’t think that should be a problem.

Q: Could you include what the experience would look like to bring-your-own-masking provider?

I updated the original proposal above to include this, but had a few questions about the UX I wanted to get clarification on, since I was basing this draft on the one Heitor had proposed earlier in this comment:

• Are customers going to have to write the Policy class themselves? For a better experience, they should just be able to call a method and have it done for them right?
• Are we for sure going to use Pydantic and not ASTs?

Q: Based on your research with both Encryption SDK and ItsDangerous, what common nomenclature could we use for interfaces? e.g., sign, dumps, etc.

I suppose we could use encrypt and decrypt which translates directly for Encryption SDK, and would be dumps and loads for ItsDangerous.

[rubenfonseca]

@heitorlessa Responding to the rest of your questions here!

Q: For AST, are you using stdlib ast or a 3rd party library? If the latter, what’s their license and package size?

I am using only the stdlib ast, but just to call out that python3.9 or later is needed for what we’re using from this library, though I don’t think that should be a problem.

Great catch. We will have to support all the current supported Python Lambda runtimes, which at this moment means we have to support python 3.7. Do you know if there's a 3rd party library that we could use so we can cover all Python versions?

Q: Could you include what the experience would look like to bring-your-own-masking provider?

• Are customers going to have to write the Policy class themselves? For a better experience, they should just be able to call a method and have it done for them right?

One idea was to have common use cases implemented by us in Powertools, so we make it as easier as possible for customers to adapt.

• Are we for sure going to use Pydantic and not ASTs?

If we can't find a good 3rd party library to replace stdlib ast, Pydantic could be a good option since we already use it elsewhere.

Q: Based on your research with both Encryption SDK and ItsDangerous, what common nomenclature could we use for interfaces? e.g., sign, dumps, etc.

I suppose we could use encrypt and decrypt which translates directly for Encryption SDK, and would be dumps and loads for ItsDangerous.

I don't have any strong opinion on this.

[seshubaws]

@rubenfonseca

We will have to support all the current supported Python Lambda runtimes, which at this moment means we have to support python 3.7. Do you know if there's a 3rd party library that we could use so we can cover all Python versions?

astunparse is a library that also offers the same capabilities as the stdlib ast for all Python versions, and it has a BSD license.

[heitorlessa]

@seshubaws and I had a 1:1 sync as there were some confusing areas on scope (now vs later). I'd like to take a moment to recap on what we're trying to do and when.

There are two macro use cases:

1. Mask/Unmask primitive data types. This feature receives an input and return data masked - this will often be a string, a number, binary, or a sequence (list, tuple, set, range). In its simplest form, customers can use for use cases where they are passing the exact value they want it to be masked/unmasked irreversibly or not.
2. Traverse an arbitrary dictionary. This feature receives instructions as to where in the given dictionary it should mask/unmask the data - this means we have to work with Abstract Syntax Tree (AST) to efficiently recurse through arbitrary data.

For now, we should focus on the first macro use case - Mask/Unmask primitive data types.

Why?

Our primary focus is making sure this feature can use AWS Encryption SDK and it is easily swappable for another implementation (e.g., ItsDangerous, Bring-your-own). The second use case is also dependent on the first one.

For example, if a customer uses Pydantic (the most popular data validation library), they can start address the second use case by incorporating this feature with a validator. Pydantic will do the heavy lifting of traversing (Node visitor pattern) the model and call this feature to mask/unmask data upon the data validation phase.

This gives us time to focus on the UX to solve the actual problem (Mask/Unmask data), give us a window for customer feedback to get extensibility right (batteries included but swappable), and only then take the second use case for those not using Pydantic. This could mean a launch with the first use case completed, followed by the second use case in a separate release - we don't have do it all at once; it's a complex domain and UX is at utmost importance.

Please let me know if I can make any of these parts any clearer.

[heitorlessa]

Please let us know if you have any updates @seshubaws in the coming weeks.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[leandrodamascena]

Reopening this issue to update the future changes and the developer experience.

[heitorlessa]

Quick summary of the experience that's been merged.

Take the following sample data in mind:

{
     "id": 1,
     "name": "John Doe",
     "age": 30,
     "email": "[email protected]",
     "address": {
          "street": "123 Main St",
          "city": "Anytown",
          "state": "CA",
          "zip": "12345"
     }
}

We can now use .encrypt method along with email and address.street fields that should be encrypted:

from aws_lambda_powertools.utilities.data_masking import DataMasking
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import AwsEncryptionSdkProvider
import os

encryption_keys = os.getenv("ENCRYPTION_KEYS")

encryption_provider = AwsEncryptionSdkProvider(keys=[encryption_keys])
data_masker = DataMasking(provider=encryption_provider)


def lambda_handler(event: dict, context):
     encrypted: dict = data_masker.encrypt(data=event, fields=["email", "address.street"])
     # {
     #      "id": 1,
     #      "name": "John Doe",
     #      "age": 30,
     #      "email": "InRoaXMgaXMgYSBzdHJpbmciHsLZGx2na-XzP_TB5Bf2LNU1bLc",
     #      "address": {
     #           "street": "XMgYSB_KDddaDJYMb-JpbmGnagTklwQ-msdaDLP",
     #           "city": "Anytown",
     #           "state": "CA",
     #           "zip": "12345"
     #      },
     # }

Conversely, the decrypt operation could work in a separate function:

from aws_lambda_powertools.utilities.data_masking import DataMasking
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import AwsEncryptionSdkProvider
import os

encryption_keys = os.getenv("ENCRYPTION_KEYS")

encryption_provider = AwsEncryptionSdkProvider(keys=[encryption_keys])
data_masker = DataMasking(provider=encryption_provider)


def lambda_handler(event, context):
     decrypted = data_masker.decrypt(data=event, fields=["email", "address.street"])
     # {
     #          "id": 1,
     #           "name": "John Doe",
     #           "age": 30,
     #           "email": "[email protected]",
     #           "address": {
     #                "street": "123 Main St",
     #                "city": "Anytown",
     #                "state": "CA",
     #                "zip": "12345"
     #           },
     # }

[github-actions]

This is now released under 2.26.0 version!

[leandrodamascena]

Reopening

[heitorlessa]

Closing as code is merged; docs is in progress. that should better reflect what's pending on the board.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.