Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PDE-5817 Add support for profiles/events #2

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

Conversation

joelgardner
Copy link
Collaborator

@joelgardner joelgardner commented May 7, 2024

This adds two streams for events and profiles. This was taken hotgluexyz-klaviyo, which is a fork of Cody's original Klaviyo tap. The new profiles and events streams should allow us to migrate all existing streams by building views over the top of the raw profile and event data in Snowflake.

return metadata.to_list(mdata)

def stream_is_selected(mdata):
return mdata.get((), {}).get('selected', False)
Copy link

@louiszuckerman louiszuckerman May 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the first argument () to mdata.get?

a comment explaining that, or even better a comment showing the shape (or example) of mdata, would be good.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copied from this line of code and I have no idea what it means but it's keying into a structure that looks like:

"metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "table-key-properties": "uuid",
            "selected": true
          }
        }
]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some investigating, it's the Singer's python library's way of basically specifying key-paths into a dict. () appears to be equivalent to keying into the top-level of fields in the dictionary. See "breadcrumb" here.

'full'
)

FULL_STREAMS = [GLOBAL_EXCLUSIONS, LISTS, LIST_MEMBERS, EVENTS, PROFILES]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly, FULL_STREAMS refers to streams extracted in a full overwrite manner. Is that right? Or is it just the full list of streams? I don't think we should extract things like EVENTS full overwrite -- can we not do it incrementally?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This came from here and as far as I can tell, FULL_STREAMS is only used in the discover operation, so I don't think it actually drives anything related to full/incremental.

Yeah this code is really something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants