🐛 Source Facebook Marketing: fix unpack ValueError in remove_params_from_url

[vitaliizazmic]

What

According to comment in issue #5190 jobs are failing in streams.remove_params_from_url:

 ERROR () LineGobbler(voidCall):85 -     record["thumbnail_url"] = remove_params_from_url(thumbnail_url, ["_nc_hash", "d"])
2021-08-03 20:02:54 ERROR () LineGobbler(voidCall):85 -   File "/airbyte/integration_code/source_facebook_marketing/streams.py", line 53, in remove_params_from_url
2021-08-03 20:02:54 ERROR () LineGobbler(voidCall):85 -     key, value = q.split("=")
2021-08-03 20:02:54 ERROR () LineGobbler(voidCall):85 - ValueError: not enough values to unpack (expected 2, got 1)

remove_params_from_url method is used while syncing ad creatives streams. Its records contains thumbnail url, which have query parameters. Some URLs has random values, these values doesn't affect validity of URLs, but breaks SAT, therefore parameters with this values are removed. Error occurred when parameters didn't have values. Additional checking was added.

[vitaliizazmic]

/test connector=source-facebook-marketing

🕑 source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/1186442299
❌ source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/1186442299

[gaart]

Probably should use more generic solution

[gaart]

from urllib.parse import urlparse, parse_qs, urlunparse, urlencode

def remove_params_from_url(url, params):
    """
    >>> remove_params_from_url('https://foo.bar/?x=1&y=&z=3', params=['x'])
    'https://foo.bar/?y=&z=3'
    >>> remove_params_from_url('https://foo.bar/?x=1&y=&z=3', params=['y'])
    'https://foo.bar/?x=1&z=3'
    >>> remove_params_from_url('https://foo.bar/?x=1&y=&z=3', params=['x', 'z'])
    'https://foo.bar/?y='
    >>> remove_params_from_url('https://foo.bar/?x=1&y=&z=3&x=111', params=['z'])
    'https://foo.bar/?x=1&x=111&y='
    """
    parsed = urlparse(url)
    query = parse_qs(parsed.query, keep_blank_values=True)
    filtered = dict((k, v) for k, v in query.items() if k not in params)
    return urlunparse([
        parsed.scheme,
        parsed.netloc,
        parsed.path,
        parsed.params,
        urlencode(filtered, doseq=True),
        parsed.fragment
    ])

[CLAassistant]

All committers have signed the CLA.

[vitaliizazmic]

It was fixed in PR #5958