BUG - fivetran_log__connector_status.connector_health = broken for a connected connector

[ipear3]

Are you a current Fivetran customer?
Yes. My name is Ian Pearthree. I am a data engineer at Pendo.

Describe the bug
Our fivetran_log__connector status page says a connector is broken, but it is actually successfully syncing historical data.

Steps to reproduce
Configure a Pendo connector, then enable the connector. This will correctly start a priority-first sync. You'll also need to configure and enable a fivetran_log connector.
Then, install the fivetran_log dbt package to your DBT instance. Run the models contained in the package.
Then, check out the table built by the fivetran_log__connector_status model. The Pendo connector has a connector_health of "broken".
Then, check out the connector in you fivetran dashboard. The connector will have a green active status, and will indicate that the priority data sync is complete, and the historical sync may continue to run. All the UI elements indicate that the connector is healthy, and operating as intended.

Expected behavior
I would expect the Pendo connector in the fivetran_log__connector_status to have a "connected" connector_health value. If not "connected", then I'd expect a status like "priority first sync", so that it accurately reflects the status indicated by the fivetran UI.

Project variables configuration

name: 'analytics'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'default'

# These configurations specify where dbt should look for different types of files.
# The `source-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
    - "target"
    - "dbt_modules"


# Model Configuration
# Full documentation: https://docs.getdbt.com/docs/configuring-models
models:
    +persist_docs: # Create BigQuery descriptions for objects with resource descriptions.
        relation: True
        columns: True
    analytics:
     # Applies to all Files under models/temp
        materialized: table
        intermediate:
            dataset: staging
        final:
            dataset: final
        staging:
            schema: staging
        views:
            schema: views
        temp:
            schema: temp
    jira_source:
        +schema: staging
    jira:
        +schema: fivetran_jira
        intermediate:
            +schema: staging
    zendesk_source:
        +schema: staging
    zendesk:
        +schema: pendo_zendesk
        agent_work_time:
            +schema: staging
        intermediate:
            +schema: staging
        reply_times:
            +schema: staging
        sla_policy:
            +schema: staging
        ticket_history:
            +schema: staging
        utils:
            +schema: staging
    salesforce_source:
        +schema: staging
    salesforce:
        +schema: fivetran_salesforce
        intermediate:
            +schema: staging

vars:
    jira_schema: fivetran_jira
    jira:
        issue_field_history_columns: [
            "timespent",
            "assignee",
            "lastViewed",
            "summary",
            "resolutiondate",
            "creator",
            "resolution",
            "updated",
            "description",
            "timeoriginalestimate",
            "issuekey",
            "timeestimate",
            "security",
            "parent",
            "priority",
            "created",
            "workratio",
            "statuscategorychangedate",
            "statuscategory",
            "environment",
            "project",
            "issuetype",
            "reporter",
            "components",
            "labels",
            "versions",
            "fixVersions",
            "customfield_12602",  # The custom field for zendesk ticket count
            "customfield_12735", # one of the 'severity' fields
            "customfield_12100", #another 'severity' field
            "customfield_10114", # 'Team'
            "customfield_12801", # 'Team name'
            "customfield_11201", # 'Dev team'
            "customfield_12720", # 'Team'
            "customfield_12760", # 'Value proposition'
            "customfield_12508", # 'Buisness value'
            "customfield_12836" # 'Dev SOC Trust values'
        ]
    zendesk_schema: pendo_zendesk
    salesforce_schema: fivetran_salesforce
    fivetran_log:
        fivetran_log_using_transformations: false # this will disable all transformation + trigger_table logic
        fivetran_log_using_triggers: false # this will disable only trigger_table logic

Package Version

packages:
  - package: fivetran/jira_source
    version: 0.2.0
  - package: fivetran/jira
    version: 0.3.0
  - package: fivetran/zendesk_source
    version: 0.3.0
  - package: fivetran/zendesk
    version: 0.4.0
  - package: fivetran/salesforce_source
    version: 0.2.1
  - package: fivetran/salesforce
    version: 0.3.1
  - package: fivetran/fivetran_log
    version: 0.3.0

Warehouse
BigQuery

Additional context
I think the source of the issue is no end date in the fivetran_log.log table for the connector in question (connector_id = reclusive_reprimand)

Screenshots

Please indicate the level of urgency
Surfacing connector statuses is a quarterly objective for us, with a deadline of October 31st.

Are you interested in contributing to this package?
No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[fivetran-joemarkiewicz]

Hi @ipear3 thank you so much for raising this issue with our team!

Our team will take a look into this early next week and will let you know what we can do to help resolve the bug you are seeing. We will coordinate further here. Look forward to chatting more!

[fivetran-jamie]

hey @ipear3 thanks for providing so much detail - super helpful and i think i've figured out what's going on here!

so, the problem is that priority first syncs do not send sync_end log events when they complete. instead, a status event is sent, with a note that the rest of the sync has been rescheduled. the package's logic, however, depends on sync_end in determining the health of the connector, and because the connector encountered an error at some point, it's getting erroneously labeled as 'broken', since an error occurred more recently than a sync_end event.

so, ultimately it looks like the package needs some code updates to accommodate the way priority-first syncs work. we'll slot fixing this into our next sprint (which starts next Wednesday) and get working on it then if that still works for you 🙂

[ipear3]

Thank you @fivetran-jamie! I'm very delighted you honed in on the root cause, and even more delighted that it sounds feasible to work with the existing Fivetran Log connector and updated DBT package logic. Looking forward to the next release!

[fivetran-jamie]

FYI starting to work on this now! 🙂

[fivetran-jamie]

hey @ipear3 -- you mentioned that you would expect to see this connector categorized as either "connected" or something like "priority first sync". would it be more helpful to be provided the latter option? or is the value just knowing that the connector is working? thanks!

[ipear3]

Hey @fivetran-jamie thanks for asking for input.

In my opinion, it would be great if connector_health provided the most granular status, like "priority first sync".

That way we have the data necessary to fulfill two use cases: alerting when a connector is broken (where only "broken" matters), and visualizing the status of a connector (where it would be nice to know the difference between fully synced connectors and currently syncing connectors).

If we're brainstorming... even more ideal would be multiple fields. A connection state (connected or paused), a connection health (broken, healthy), and a connection status (initial sync underway, priority first sync underway, up-to-date, delayed).

[fivetran-jamie]

hey @ipear3 cool! after talking with the team I think it would be best to keep this a singular field, but i agree that a higher level of granularity for priority first syncs is ideal! i've implemented the fix in the feature/priority-first-syncs branch of this repo, if you'd like to try it out! however, know that if your connector has completed its priority-first sync at this point, it'll be labeled as "connected"

# your packages.yml
  - git: https://github.com/fivetran/dbt_fivetran_log.git
    revision: feature/priority-first-syncs
    warn-unpinned: false

[fivetran-jamie]

hey @ipear3 we're probably going to be releasing this fix tomorrow if you wanted to check it out before then! we've run it internally on our own priority-first syncing connectors and it looks like it's working, but fyi 😃

[fivetran-jamie]

hey just released this fix and it should be up on dbt hub shortly!