Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement batch support for Attio Actions #2057

Merged
merged 3 commits into from
May 28, 2024
Merged

Implement batch support for Attio Actions #2057

merged 3 commits into from
May 28, 2024

Conversation

nickcampbell18
Copy link
Contributor

This pull request adds batch support for the Attio Actions destination. We're looking to make this change because our customers are seeing high failure rates (rate limit exceeded, timeouts and 5xx errors) with the single-event mechanisms.

Our secret batching API supports up to 10,000 Segment events in a single batch that will allow us to process these asynchronously, in order, in our own infrastructure in a more controlled and reliable way.

Testing

  • Added unit tests for new functionality
  • Tested end-to-end using the local server
  • [Segmenters] Tested in the staging environment

@nickcampbell18 nickcampbell18 requested a review from a team as a code owner May 24, 2024 16:04
@nickcampbell18 nickcampbell18 requested a review from a team May 24, 2024 16:04
@joe-ayoub-segment joe-ayoub-segment self-assigned this May 27, 2024
joe-ayoub-segment
joe-ayoub-segment requested changes May 27, 2024
View reviewed changes
Copy link
Contributor

@joe-ayoub-segment joe-ayoub-segment left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @nickcampbell18 thanks for raising this PR.
Generally looks good.
1 major but hopefully simple to fix issue. These 3 new fields you are adding will resolve to undefined for all customers' instances which have already been set up prior to the deploy. So that needs to be handled in code defensively.

Also note that this means that batching will be disabled for these customers' instances. To enable batching the customer would need to edit the 'Action' , and save the Action. However because the enable_batching field is hidden it might not be possible.

What's your intention here?

Nick Campbell added 2 commits May 28, 2024 08:45
Make received_at parameter optional and gracefully handle it
fe3c78b
Make "enable_batching" parameter viewable, optional, default false
333f3e3 
There are tradeoffs associated with Segment event batching - there is no
realtime feedback if the request was invalid, for example - so we should
make it opt-in and controllable by the user.
@nickcampbell18
Copy link
Contributor Author

Thanks for the feedback! I've addressed both of those points.

What do you think of the wording here on the second commit? At the moment Attio won't tell the user if their mapping is causing validation failures on our side (although you can still use the event tester I guess to see if it would generally work). I can send a PR to the docs page as well, to clarify if that's easier...

@joe-ayoub-segment
Copy link
Contributor

Looks good. I might reword the batch label and description after I merge the code.

@joe-ayoub-segment joe-ayoub-segment merged commit 4b27451 into segmentio:main May 28, 2024
9 of 11 checks passed
@joe-ayoub-segment
Copy link
Contributor

hi @nickcampbell18 PR deployed. Please check it all looks good!

@nickcampbell18 nickcampbell18 mentioned this pull request May 30, 2024
Merged
harsh-joshi99 added a commit that referenced this pull request Jun 5, 2024
@harsh-joshi99
Squashed commit of the following:
74ec16c 
commit c37259c
Author: Harsh Joshi <[email protected]>
Date:   Wed Jun 5 16:05:46 2024 +0530

    Update test cases

commit 6ab6d57
Author: Harsh Joshi <[email protected]>
Date:   Tue Jun 4 15:15:20 2024 +0530

    Add profile properties in Add profilr to ost event

commit 524fc6c
Author: Varadarajan V <[email protected]>
Date:   Wed May 29 17:41:51 2024 +0530

    Remove build-experience-team from pr labeler as internal team (#2060)

commit 72adc64
Author: Joe Ayoub <[email protected]>
Date:   Tue May 28 11:04:54 2024 +0100

    Publish

     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]

commit e4bb620
Author: Prayansh Srivastava <[email protected]>
Date:   Tue May 28 02:25:29 2024 -0700

    add new excludeWhenNull directive and update tests (#2040)

    * add new excludeWhenNull directive and update tests

    * add readme docs

    * fix replace test case

commit e7b424a
Author: harsh-joshi99 <[email protected]>
Date:   Tue May 28 14:53:39 2024 +0530

    Add override_list_id mapping in upsert Profile (#2046)

    * Add override_list_id mapping in upsert Profile, add function to group profiles based on list ID

    * fix payload

    * Fix payload condition

    * Refactor profile filter

commit 6170334
Author: Joe Ayoub <[email protected]>
Date:   Tue May 28 10:18:32 2024 +0100

    updating field description and updating generated types

commit 1107970
Author: Alice Mackel <[email protected]>
Date:   Tue May 28 05:07:52 2024 -0400

    Change StackAdapt destination name (#2055)

commit 76c733d
Author: Leonel Sanches <[email protected]>
Date:   Tue May 28 02:07:21 2024 -0700

    Preventing double-hashing for TikTok Offline Conversions. (#2034)

commit a621179
Author: Nuno Sousa <[email protected]>
Date:   Tue May 28 10:07:09 2024 +0100

    Fix customer.io timestamp issue (#2054)

commit 1c71987
Author: Leonel Sanches <[email protected]>
Date:   Tue May 28 02:06:52 2024 -0700

    Preventing double-hashing for TikTok Conversions. (#2033)

commit 4b27451
Author: Nick Campbell <[email protected]>
Date:   Tue May 28 10:06:28 2024 +0100

    Implement batch support for Attio Actions (#2057)

    * Implement batch APIs for Attio

    * Make received_at parameter optional and gracefully handle it

    * Make "enable_batching" parameter viewable, optional, default false

    There are tradeoffs associated with Segment event batching - there is no
    realtime feedback if the request was invalid, for example - so we should
    make it opt-in and controllable by the user.

commit 0fd6c22
Author: Innovative-GauravKochar <[email protected]>
Date:   Tue May 28 14:33:36 2024 +0530

    [STRATCONN-3468] | Introduce 'Amazon AMC' Audience Destination (#2036)

    * Updated the code for amazon ads syncaudience action

    * Fix the build failing

    * Updated the getAudience and createAudience

    * Code to update the value to hashed

    * Added the advertiserId field in audienceSetting

    * Added check for advertiserid

    * Fixing External Id issue , replace with quoted string

    * Resolve Conflicts

    * Resolve Conflicts

    * Resolve Conflicts

    * Worked on unit test case for createAUdience and syncAudience

    * Code cleaning

    * removed stats

    * Updated Snapshots

    * added headers in refresh Access token

    * Fix refresh access token API

    * Worked on handling cpmCents and ttl

    * Fixed Pr Comments

    * Rename destination slug nad action name

    * removed amazon-ads as we are changing action destination slug

    * Validate Schema in createAudience

    * registered amazon-amc and updated destination id

    * will handle this in seperate PR

    ---------

    Co-authored-by: Harsh Vardhan <[email protected]>
    Co-authored-by: Gaurav Kochar <[email protected]>

commit 2fc41f9
Author: Varadarajan V <[email protected]>
Date:   Fri May 24 11:49:25 2024 +0530

    [snyk] bump jscodeshift from 0.13.0 to 0.14.0 (#2056)

commit e790949
Author: alfrimpong <[email protected]>
Date:   Wed May 22 00:59:22 2024 -0500

    feat: add script for action-cli shared (#2050)
harsh-joshi99 added a commit that referenced this pull request Jun 10, 2024
@harsh-joshi99
Squashed commit of the following:
bbcb77c 
commit f93c309
Author: Harsh Joshi <[email protected]>
Date:   Mon Jun 10 11:37:25 2024 +0530

    Update snapshot

commit e939c24
Author: Harsh Joshi <[email protected]>
Date:   Fri Jun 7 17:04:57 2024 +0530

    Update profile identifiers for track and order completed events.

commit 77b91d0
Author: Gaurav Kochar <[email protected]>
Date:   Thu Jun 6 15:17:54 2024 +0530

    Publish

     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]

commit 127c4c2
Author: Innovative-GauravKochar <[email protected]>
Date:   Thu Jun 6 15:10:11 2024 +0530

    Worked on fixing the refresh access token issue and made currency dropdown (#2079)

    Co-authored-by: Gaurav Kochar <[email protected]>

commit 6dfaf37
Author: Alice Mackel <[email protected]>
Date:   Wed Jun 5 10:42:27 2024 -0400

    Change name of browser destination (#2058)

commit 3510923
Author: Joe Ayoub <[email protected]>
Date:   Wed Jun 5 12:46:59 2024 +0100

    Publish

     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]

commit fec0383
Author: alfrimpong <[email protected]>
Date:   Wed Jun 5 05:30:50 2024 -0500

    Fcm payload update (#2073)

    * fix fcm payload

    * fix: removed FCM overrides

    * chore: rm log

commit 9515a35
Author: konoufo <[email protected]>
Date:   Tue Jun 4 17:39:16 2024 -0400

    [DC-789] adds subscription sync modes support in destination-kit (#2065)

    * adds subscription sync modes support in destination-kit

    * renames SyncModes type

    * remove test.only

    * adds syncMode support in performBatch handler + remove...

    ... internal field from payload

    ---------

    Co-authored-by: Mohamed K. Coulibali <[email protected]>

commit f83f691
Author: Joe Ayoub <[email protected]>
Date:   Tue Jun 4 13:53:06 2024 +0100

    Publish

     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]

commit c9d30b4
Author: hanoak20 <[email protected]>
Date:   Tue Jun 4 18:12:06 2024 +0530

    Contentstack's destination enhancement. (#2062)

    * feat: added Contentstack's destination-action.

    * chore: segregated the types of destination, and other improvements.

    * fix: Action schema changed to oauth-managed.

    * refactor: used 'extendRequest' function to add headers automatically.

    * refactor: removed yarn PackageManager config.

    * refactor: simplified the perform's data and its usage.

    * chore: personalize urls are now configurable.

commit e99e29f
Author: Matthew Saunders <[email protected]>
Date:   Tue Jun 4 08:40:32 2024 -0400

    Update MC contacts upsert to handle multiple identifiers (#2032)

    * Update MC contacts upsert to handle multiple identifiers

    * Update defaults for phone_number_id, anonymous_id

    * Update default for anonymous_id

    * Change anonymous_id to anonymousId

commit 807258a
Author: Innovative-GauravKochar <[email protected]>
Date:   Tue Jun 4 17:37:56 2024 +0530

    [AMAZON-AMC] | Fixed "415 Unsupported Media Type" issue while doing testAuthentication. (#2069)

    * [AMAZON-AMC] |Fixed content-Type issue in testAuthentication

    * Added unit test cases for content-type

    ---------

    Co-authored-by: Gaurav Kochar <[email protected]>

commit c149f44
Author: Leonel Sanches <[email protected]>
Date:   Tue Jun 4 05:02:58 2024 -0700

    Google Ads Enhanced Conversions: preventing double hashing (#2053)

    * - Preventing already hashed information to be hashed again;
    - Better typing for all actions.

    * Unit tests.

    * Checking whether the phone is hashed also for `uploadClickConversion` action.

commit 7f24d61
Author: maryamsharif <[email protected]>
Date:   Tue Jun 4 05:02:36 2024 -0700

    Add dropdown to region setting (#2044)

commit dbec70c
Author: Thomas Gilbert <[email protected]>
Date:   Tue Jun 4 08:02:05 2024 -0400

    Fix to show request headers in Action's Tester UI (#2067)

    * add test destination

    * add request body

    * convert req headers to js object

    * remove test dest

    * add filtering for sensitive headers

commit 5887a12
Author: Arijit Ray <[email protected]>
Date:   Tue Jun 4 17:31:20 2024 +0530

    [STRATCONN-3772] Add new operator "number_equals" for number (#2030)

    * Add == operator in destination subscription package

    * Update/add new unit tests for the operator

    * limit == to just numeric values

    * update test cases STRATCONN-3772

    * Add operator == in the AST level

    * Modified = to only store strings

    * Modified = to only store strings: update test cases

    * add == operator for traits AND CONTEXT

    * Add stringify on fqlExpression

    * Change operator name to number_equals

    * Change operator name to number_equals

    * Change operator name to number_equals

    * Refactor branching statements on parse-fq.ts

commit 524fc6c
Author: Varadarajan V <[email protected]>
Date:   Wed May 29 17:41:51 2024 +0530

    Remove build-experience-team from pr labeler as internal team (#2060)

commit 72adc64
Author: Joe Ayoub <[email protected]>
Date:   Tue May 28 11:04:54 2024 +0100

    Publish

     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]
     - @segment/[email protected]

commit e4bb620
Author: Prayansh Srivastava <[email protected]>
Date:   Tue May 28 02:25:29 2024 -0700

    add new excludeWhenNull directive and update tests (#2040)

    * add new excludeWhenNull directive and update tests

    * add readme docs

    * fix replace test case

commit e7b424a
Author: harsh-joshi99 <[email protected]>
Date:   Tue May 28 14:53:39 2024 +0530

    Add override_list_id mapping in upsert Profile (#2046)

    * Add override_list_id mapping in upsert Profile, add function to group profiles based on list ID

    * fix payload

    * Fix payload condition

    * Refactor profile filter

commit 6170334
Author: Joe Ayoub <[email protected]>
Date:   Tue May 28 10:18:32 2024 +0100

    updating field description and updating generated types

commit 1107970
Author: Alice Mackel <[email protected]>
Date:   Tue May 28 05:07:52 2024 -0400

    Change StackAdapt destination name (#2055)

commit 76c733d
Author: Leonel Sanches <[email protected]>
Date:   Tue May 28 02:07:21 2024 -0700

    Preventing double-hashing for TikTok Offline Conversions. (#2034)

commit a621179
Author: Nuno Sousa <[email protected]>
Date:   Tue May 28 10:07:09 2024 +0100

    Fix customer.io timestamp issue (#2054)

commit 1c71987
Author: Leonel Sanches <[email protected]>
Date:   Tue May 28 02:06:52 2024 -0700

    Preventing double-hashing for TikTok Conversions. (#2033)

commit 4b27451
Author: Nick Campbell <[email protected]>
Date:   Tue May 28 10:06:28 2024 +0100

    Implement batch support for Attio Actions (#2057)

    * Implement batch APIs for Attio

    * Make received_at parameter optional and gracefully handle it

    * Make "enable_batching" parameter viewable, optional, default false

    There are tradeoffs associated with Segment event batching - there is no
    realtime feedback if the request was invalid, for example - so we should
    make it opt-in and controllable by the user.

commit 0fd6c22
Author: Innovative-GauravKochar <[email protected]>
Date:   Tue May 28 14:33:36 2024 +0530

    [STRATCONN-3468] | Introduce 'Amazon AMC' Audience Destination (#2036)

    * Updated the code for amazon ads syncaudience action

    * Fix the build failing

    * Updated the getAudience and createAudience

    * Code to update the value to hashed

    * Added the advertiserId field in audienceSetting

    * Added check for advertiserid

    * Fixing External Id issue , replace with quoted string

    * Resolve Conflicts

    * Resolve Conflicts

    * Resolve Conflicts

    * Worked on unit test case for createAUdience and syncAudience

    * Code cleaning

    * removed stats

    * Updated Snapshots

    * added headers in refresh Access token

    * Fix refresh access token API

    * Worked on handling cpmCents and ttl

    * Fixed Pr Comments

    * Rename destination slug nad action name

    * removed amazon-ads as we are changing action destination slug

    * Validate Schema in createAudience

    * registered amazon-amc and updated destination id

    * will handle this in seperate PR

    ---------

    Co-authored-by: Harsh Vardhan <[email protected]>
    Co-authored-by: Gaurav Kochar <[email protected]>

commit 2fc41f9
Author: Varadarajan V <[email protected]>
Date:   Fri May 24 11:49:25 2024 +0530

    [snyk] bump jscodeshift from 0.13.0 to 0.14.0 (#2056)

commit e790949
Author: alfrimpong <[email protected]>
Date:   Wed May 22 00:59:22 2024 -0500

    feat: add script for action-cli shared (#2050)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joe-ayoub-segment joe-ayoub-segment joe-ayoub-segment approved these changes

Assignees

@joe-ayoub-segment joe-ayoub-segment

Labels
deploy:push mode:cloud partner-eng-review (cloud) Ready for Release team:external
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nickcampbell18 @joe-ayoub-segment