🐛 Source Facebook Marketing: discover is broke when using valid breakdown value in a custom insight

[alafanechere]

Environment

• Airbyte version: 0.32.8-alpha
• OS Version / Instance: example macOS, Windows 7/10, Ubuntu 18.04, GCP n2. , AWS EC2
• Deployment: example are Docker or Kubernetes deploy env
• Source Connector and version: Facebook Marketing 0.2.26
• Severity: Critical
• Step where error happened: Discover schema

Current Behavior

I reproduced a user's bug (slack conversation): when I set a custom_insights in the connection with a breakdown value such as product_id the connection test is sucessful but the schema discovery that follows crashes.

This is due to the fact that the list in the schemas dict

airbyte/airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/streams.py

Line 421 in 9f685b9

schemas = {

does not list all valid values for this field. The list used by the connection check (all the properties of ads_insights_breakdowns.json looks more accurate

Expected Behavior

• Successful discovery with all the possible breakdown value supported by Facebook marketing API.
• A single source of through for all the valid values the breakdown field can take.

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG

2021-12-02 14:58:41 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/0d25e941-5540-4141-82bf-d12d139558e7/0 --network host --log-driver none airbyte/source-facebook-marketing:0.2.26 discover --config source_config.json
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 - Traceback (most recent call last):
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -   File "/airbyte/integration_code/main.py", line 13, in <module>
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -     launch(source, sys.argv[1:])
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -   File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 108, in launch
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -     for message in source_entrypoint.run(parsed_args):
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -   File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 93, in run
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -     catalog = self.source.discover(self.logger, config)
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -   File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 68, in discover
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -     streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -   File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 68, in <listcomp>
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -     streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -   File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/streams/core.py", line 66, in as_airbyte_stream
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -     stream = AirbyteStream(name=self.name, json_schema=dict(self.get_json_schema()), supported_sync_modes=[SyncMode.full_refresh])
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -   File "/airbyte/integration_code/source_facebook_marketing/streams.py", line 408, in get_json_schema
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -     schema["properties"].update(self._schema_for_breakdowns())
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -   File "/airbyte/integration_code/source_facebook_marketing/streams.py", line 436, in _schema_for_breakdowns
2021-12-02 14:58:43 ERROR () LineGobbler(voidCall):82 -     return {breakdown: schemas[breakdown] for breakdown in self.breakdowns}

Steps to Reproduce

1. Set up a Facebook Marketing connection with custom insights using product_id in breakdown
2. Connection test passes
3. On a new connection setup between source and destination feching schema fails

Are you willing to submit a PR?

Yes