Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parser): implement safeParse option #2244

Merged
merged 18 commits into from
Mar 21, 2024
Merged

Conversation

am29d
Copy link
Contributor

@am29d am29d commented Mar 18, 2024

Description of your changes

This PR implements safeParse option for decorator and middy middleware. The safeParse is necessary to allow users to have a catch mechanism and recover from any parsing errors manually. The option improves the DX compared to manual parsing, even with provided envelopes and schemas. Here is the difference.

With the new options the example code will be:

const TestSchema = z.object({
   name: z.string(),
   age: z.string()
});

type TestEvent = z.infer<typeof TestSchema>;

@parser({ schema: TestSchema, envelope: EventBridgeEnvelope, safeParse: true })
public async handler(event: ParsedResult<EventBridgeEvent, TestEvent>,  _context: Context): Promise<unknown> {
  if(event.success) {
     event.data: TestEvent .... // event is parsed and is a TestEvent type derived from TestSchema
  } else {
    event.error // inspect error, log it, add metric
    event.originalEvent: EventBridgeEvent // this is the structure of the original event
  }
}

The based on the parse outcome the event object will be changed to ParsedResult with the structure:

export type ParsedResultSuccess<Output> = {
  success: true;
  data: Output;
};

export type ParsedResultError<Input> = {
  success: false;
  error: ZodError | Error;
  originalEvent: Input;
};

export type ParsedResult<Input = unknown, Output = unknown> =
  | ParsedResultSuccess<Output>
  | ParsedResultError<Input>

In case of success we return data, otherwise error and originalEvent, which allows user to anything they want for troubleshooting, observability and recovery. The Input and Output parameters are optional and should provide better DX during development, so TS understand what to expect.

The implementation became more complex than expected, which is a signal that Powertools takes care of complexity, but also a smell that there might be a better solution and we need to simplify in the future. I have explored one approach but hit few blockers.

You will notices that we often unwrap the inner part of the event, the actual payload we are interested in. It looks simple for API gateway events where we only have one level, the body. With more nested fields, arrays and additional transformations like decompression, json parse, base64 encoding, the chain becomes more complex and we have to do more manual work.

This leads to an assumption, that we could extend the built-in schema in the envelope and let the parse do the jobs, so we don't need to, for instance, here is an example of a potential ApiGatewayEnvelope

export class ApiGatewayEnvelope extends Envelope {
  public static parse<T extends ZodSchema>(
    data: unknown,
    schema: T
  ): z.infer<T> {
    const customisedParsedEnvelope = APIGatewayProxyEventSchema.extend({
      body: schema,
    });
    
    return customisedParsedEnvelope.parse(data);
}

It does look much simpler, especially if we apply it any other schema with arrays, we don't have to loop through the events and call the parser for each of them.

But there is a huge limitation with many cases, which does not allow us to uniformly apply this solution. In case of deeply nested events with more than one level we have to transform the payload and apply the parser afterwards, i.e. decompress bytes to string -> parse string to object -> apply parser schema. Zod does not provide a capability where we can apply transformation before parsing, only afterwards.

There might be an option to do it in zod, which I am not aware of yet.

While it might look like this could potentially impact performance, it does not change the underlying complexity and runtime overhead customers would have if they used zod on their own. We only provide built-in schemas and decorators to improve the DX.

Related issues, RFCs

Issue number: closes #2204

Checklist

  • My changes meet the tenets criteria
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my change is effective and works
  • The PR title follows the conventional commit semantics

Breaking change checklist

Is it a breaking change?: NO

  • I have documented the migration process
  • I have added, implemented necessary warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@am29d am29d requested review from a team as code owners March 18, 2024 09:11
@boring-cyborg boring-cyborg bot added parser This item relates to the Parser Utility tests PRs that add or change tests labels Mar 18, 2024
@pull-request-size pull-request-size bot added the size/XXL PRs with 1K+ LOC, largely documentation related label Mar 18, 2024
@github-actions github-actions bot added the feature PRs that introduce new features or minor changes label Mar 18, 2024
Copy link
Contributor

@dreamorosi dreamorosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some suggestions/comments almost exclusively on types/imports and documentation.

I really like the implementation and think this will add much value to users.

packages/parser/src/parserDecorator.ts Outdated Show resolved Hide resolved
packages/parser/src/parserDecorator.ts Outdated Show resolved Hide resolved
packages/parser/src/parserDecorator.ts Outdated Show resolved Hide resolved
packages/parser/src/parserDecorator.ts Outdated Show resolved Hide resolved
packages/parser/src/parserDecorator.ts Outdated Show resolved Hide resolved
packages/parser/src/types/envelope.ts Outdated Show resolved Hide resolved
packages/parser/src/types/parser.ts Outdated Show resolved Hide resolved
packages/parser/src/types/parser.ts Outdated Show resolved Hide resolved
packages/parser/src/types/parser.ts Show resolved Hide resolved
packages/parser/src/envelopes/apigw.ts Show resolved Hide resolved
@am29d am29d requested a review from dreamorosi March 21, 2024 12:59
Copy link

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
2.1% Duplication on New Code

See analysis details on SonarCloud

Copy link
Contributor

@dreamorosi dreamorosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this PR, really like how this is going!

@dreamorosi dreamorosi merged commit 0012f92 into feat/parser Mar 21, 2024
11 checks passed
@dreamorosi dreamorosi deleted the feat/parser-safeParse branch March 21, 2024 13:56
@dreamorosi dreamorosi linked an issue Mar 21, 2024 that may be closed by this pull request
2 tasks
dreamorosi added a commit that referenced this pull request Apr 15, 2024
* init parser package

* add init config

* feat(logger): Support for external observability providers (#1511)

* Updated formatAttributes for additional parameters and LogItem return type

* Updated the unit tests to pass with new formatter

* Updated Powertool named objects to Powertools

* Updated tests to match new naming consistency

* Updated for tests for new naming consistency

* Updated formatter for new design decisions

* Update Logger for ephemeral attributes

* Update bringYourOwnFormatter documentation to match new formatter

---------

Co-authored-by: erikayao93 <[email protected]>

* chore(logger): PowertoolsLogFormatter docstring and variable naming update (#1585)

* Updated formatAttributes for additional parameters and LogItem return type

* Updated the unit tests to pass with new formatter

* Updated Powertool named objects to Powertools

* Updated tests to match new naming consistency

* Updated for tests for new naming consistency

* Updated formatter for new design decisions

* Update Logger for ephemeral attributes

* Update bringYourOwnFormatter documentation to match new formatter

* Fixed incorrect return type, renamed variable for consistency

* feat(logger): Support for external observability providers (#1511)

* Updated formatAttributes for additional parameters and LogItem return type

* Updated the unit tests to pass with new formatter

* Updated Powertool named objects to Powertools

* Updated tests to match new naming consistency

* Updated for tests for new naming consistency

* Updated formatter for new design decisions

* Update Logger for ephemeral attributes

* Update bringYourOwnFormatter documentation to match new formatter

---------

Co-authored-by: erikayao93 <[email protected]>

* chore(logger): PowertoolsLogFormatter docstring and variable naming update (#1585)

* Updated formatAttributes for additional parameters and LogItem return type

* Updated the unit tests to pass with new formatter

* Updated Powertool named objects to Powertools

* Updated tests to match new naming consistency

* Updated for tests for new naming consistency

* Updated formatter for new design decisions

* Update Logger for ephemeral attributes

* Update bringYourOwnFormatter documentation to match new formatter

* Fixed incorrect return type, renamed variable for consistency

* chore(maintenance): bump dependencies & drop nodejs14x (#1687)

* chore: update release script to mark all utilities as alpha

* chore: restore version to ease conflicts

* chore: release version change

* chore: release version change

* chore(maintenance): remove `createLogger` and `createTracer` helpers (#1722)

* chore(maintenance): bump dependencies & drop nodejs14x (#1687)

* chore: add pre-release script

* chore: restore deps

* chore: added v2 shim

* chore(maintenance): remove logger and tracer helper function

* chore: remove imports

* chore: fix deps & versions

* tests: moved unit tests

* tests: move logger tests

* chore: added v2 shim

* chore: added v2 shim

* feat(logger): add esmodule support (#1734)

* feat(logger): add esm build output

* fix(Logger): Remove barrel files update references

* test(Logger): update jest/ts-jest to use ESM

* chore(Logger): remove unused lodash.merge

* fix(logger): reinstate lodash.merge

* chore(logger): revert TS assertion

* chore(logger): revert format changes

* chore(logger): update postbuild to remove incremental tsbuildinfo files

* fix(logger): correct reference to types output

* feat(logging): add middleware export

* chore(logger): replace postbuild script with echo statement

* feat(logger): add typesVersions property and barrel files to /types

* chore(logger): file not used, can be added back if needed

* chore(logger): add space back to README

* chore(logger): revert space in README

* feat(commons): add esmodule support (#1735)

* chore(logger): adapt logger to commons exports

* feat(commons): add esmodule support

* chore: address sonar findings

* chore(commons): exported version

* chore: fixed imports in examples

* chore(parameters): fixed imports

* chore(metrics): fixed imports

* chore(tracer): fixed imports

* chore(idempotency): fixed imports

* chore(commons): test coverage

* chore(batch): fix imports

* feat(parameters): add esmodule support (#1736)

* feat(batch): add esmodule support (#1737)

* feat(internal): add esmodule support (#1738)

* feat(testing): add esmodule support

* chore(all): update imports

* feat(metrics): add esmodule support (#1739)

* feat(tracer): add esmodule support (#1741)

* feat(tracer): add esmodule support

* chore(docs): update imports

* feat(idempotency): add esmodule support  (#1743)

* feat(idempotency): add esmodule support

* chore(metrics): fix import

* chore(ci): v2 release line

* chore(ci): fix alpha versioning pre-release

* docs(maintenance): add processes tab (#1747)

* docs(maintenance): update mkdocs to support tabs

* chore(ci): add parallel test npm script

* chore(ci): add jest command

* docs(maintenance): add testing page to navbar

* docs(maintenance): add contributing info

* chore: update roadmap

* chore: update release drafter workflow to allow for manual trigger

* fix formatting

* docs: maintainers handbook

* chore: link to new location

* fix links

* Update docs/maintainers.md

Co-authored-by: Alexander Schueren <[email protected]>

---------

Co-authored-by: Alexander Schueren <[email protected]>

* chore(tracer): update warning to better format segment name (#1750)

* chore(tracer): update warning in Tracer to better format segment name

* chore: linting

* chore(internal): remove outdated notice files (#1752)

* chore(maintenance): set `removeComments` to` false` in `tsconfig.json` (#1754)

* chore(docs): add invisible unicode char to decorator docs (#1755)

* chore: remove extra comma

* chore(docs): upgrade doc intro

* chore(ci): add workflow to publish v2 docs on merge (#1756)

* chore(docs): upgrade doc intro

* chore(ci): remove mike commands

* chore(ci): upgrade mkdocs

* feat(logger): align sampling debug logs feature implementation with the other runtimes (#1744)

* test(logger): remove logsSampled field, add default sampleRateValue

* test(logger): add tests for sampling debug logs feature

* feat(logger): change implementation to make sampling decision at per-function level

* refactor(logger): remove redundant createLogger method

* refactor(logger): remove getSampleRateValue method

* test(logger): improve tests

* refactor(logger): return createLogger() back with the detailed comment of the method importance

* test(logger): add constructor/custom config/env var priority tests for sampling rate feature, improve description

* refactor(logger): address review comments

* feat(logger): add refreshSampleRateCalculation method and tests

* test(logger): adjust end-to-end tests

* chore(logger): refactor types and interfaces (#1758)

* chore(logger): refactor types and interfaces

* chore: grouped type files

* chore: fix code smell

* chore: fix ci

* chore: fix ci

* chore(maintenance): bump Middy v4 & run tests (#1760)

* chore(parameters): fix esm bundling

* chore(parameters): refactor provider constructor to init client as needed (#1757)

* chore(parameters): refactor provider constructor to init client as needed

* chore(parameters): moved client instrumentation up in baseprovider

* chore(parameters): fix code smells

* chore(parameters): fix code smells

* chore(parameters): change declare client param

* chore(commons): update Powertools UA middleware detection (#1762)

* chore(commons): fix double ua detection

* chore(commons): fix unit test

* chore(layers) widen version check in e2e

* chore(maintenance): enable `isolatedModules` and isolate cache (#1765)

* chore(layers) widen version check in e2e

* chore(maintenance): enable isolatedModules

* chore: remove redundant comments from tsconfig

* chore: changed path of tsbuild cache

* fix: idempotency types

* chore(idempotency): refactor aws sdk init logic (#1768)

* build(tracer): bump aws-xray-sdk-core to latest

* build(maintenance): bump aws sdk dev dependencies

* chore(logger): set default UTC timezone (#1775)

* chore(parameters): add export types

* chore(logger): set default utc timezone

* chore(logger): pass down envvarsservice to log formatter

* feat(parser): add built-in schemas (#1788)

* add dynamodb schema

* add alb

* add parser to v2 build

* fix test

* add alb

* add built-in schema

* add more tests for schemas

* remove index export

* add cloudwatch with base64 zlip transform

* add throw test case

* formatting

* add kafka schema

* restructured tests

* add vpc lattice and lattice v2

* s3 event notification should extend eventbridge

* s3 sqs should extend from sqs

* simplify cloudwatch extract from string

* keep message as string, instead of empty object

* fix detail type of eb and field names

* remove duplicated entries

* fix homepage URL in readme

* improved test coverage

* key and value are always present

* cleanup unnecessary definitions, widen peerDep version req

* Update packages/parser/src/schemas/cloudwatch.ts

Co-authored-by: Andrea Amorosi <[email protected]>

* clean up events, some fields are imaginary

* fix api gw

* fix broken IP addresses in examples

* add more tests to api gw

* fix apigw2 add more tests

* add optional scopes to apigwv2

* add optional field back to api gw, stricter methods for vpc lattice

* add test for messageId refinement

* remove redundant entry

* fix sqs

* add dmarcPolicy for ses

* added tests

* moved cw function from kinesis, fix imports

* add parser to build step in ci

* use any safely here

* removed console logs

* name, add datetime to strings

* narrow string to datetime

* refine to url

* imports, remove try/catch

* add .js extension to imports

* moved comment, fixed path

* rename event filename to fix events

---------

Co-authored-by: Andrea Amorosi <[email protected]>

* feat(parser): add schema envelopes (#1815)

* first envelope

* add abstract class

* add tests

* add more tests

* fix tests

* add envelopes

* add middy parser

* minor schema changes

* add more envelopes and tests, refactored utils to autocomplete event files

* simplified check

* remove middleware from this branch

* refactored from class to function envelopes

* removed parser tests, should be in another branch

* add parser to pre push

* consistent naming

* feat(parser): implement middy parser middleware (#1823)

* add middy middleware

* add type to imports

* remove schema type, stick with unkown

* feat(parser): implement parser decorator (#1831)

* feat(parser): add types for built-in schemas (#1838)

* add types for built-in schemas

* fixed imports

* only use top level schema

* chore(parser): add parser subpath exports to package.json (#2179)

* add exports and type version to package json, including index.js

* use index.js as import for coverage

* use package lock from main

* fix envelope path and add types to exports

* use explicit exports instead of *

* import type

* make export types explicit

* adjust imports in tests for coverage, removed unused exports

* remove duplicate imports

* feat(parser): implement `safeParse` option (#2244)

* first draft on safeParse with major refactoring

* add safeParse

* fixed sns tests

* bump coverage

* remove throw error and return ParsedResult

* remove one level to reduce complexity score

* make static methods readonly

* simplified cryptic  ternary operation into something readble

* Update packages/parser/src/parserDecorator.ts

Co-authored-by: Andrea Amorosi <[email protected]>

* merged

* simplify export

* add invisible character for decorator rendering

* fix docs and tests

* Update packages/parser/src/parserDecorator.ts

Co-authored-by: Andrea Amorosi <[email protected]>

* add comment with description

* remove context

* remove unintentional safeParse export

* add examples to parse standalone function

---------

Co-authored-by: Andrea Amorosi <[email protected]>

* refresh package lock after merge

* docs(parser): add docs for parser utility (#1835)

* WIP: parser

* fix test imports

* remove unnecessary exports

* add custom validation

* remove unnecessary export

* add warning

* remove duplicate imports

* add types and error handlig

* remove comment from annotations

* minor changes

* revert merge changes

* merged package-lock

* Update docs/utilities/parser.md

Co-authored-by: Andrea Amorosi <[email protected]>

* Update docs/utilities/parser.md

Co-authored-by: Andrea Amorosi <[email protected]>

* adjust imports to new implementation

* add safeParse

* fixed line highlight

* typo

* revert index.md, add private scope to snippets packagef

* Update docs/utilities/parser.md

Co-authored-by: Andrea Amorosi <[email protected]>

* add parser to main, fixed zod install command

* fix callout indent

* fix tooltip

---------

Co-authored-by: Andrea Amorosi <[email protected]>

* feat(parser): add custom parse error (#2339)

* chore: remove rebase leftovers

* docs(parser): add utility readme

---------

Co-authored-by: Alexander Melnyk <[email protected]>
Co-authored-by: Alexander Melnyk <[email protected]>
Co-authored-by: Erika Yao <[email protected]>
Co-authored-by: erikayao93 <[email protected]>
Co-authored-by: Ant Stanley <[email protected]>
Co-authored-by: Sergei Cherniaev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature PRs that introduce new features or minor changes parser This item relates to the Parser Utility size/XXL PRs with 1K+ LOC, largely documentation related tests PRs that add or change tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature request: add safeParse option
2 participants