BigQueryInsertJobOperator will fail if the DagID contains the '.' character

[nathadfield]

Apache Airflow version: 1.10.*
BackPort Packages version: 2020.10.5rc1

What happened:

BigQueryInsertJobOperator tries to start a BigQuery job by specifying a job_id which is a combination of dag_id, task_id, execution_date and an additional uniqueness_suffix.

https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/bigquery.py#L2072

However, because BigQuery only accepts alphanumeric (in additon to dashes and underscore) characters then, if a DagID contains a version number whcih includes a . character then this will cause the task to fail.

https://cloud.google.com/bigquery/docs/running-jobs#generate-jobid

How to reproduce it:
Create a DAG with the name my-dag-v1.0.

[bradleydamato]

@nathadfield @potiuk what approach do we think should be taken here? Should we scrub the dag_id for the '.' char prior to submitting?

[saisiddhant12]

@nathadfield @potiuk
I would like to give it a shot, I can think introducing macro like ts_no_dash with a hyphen separation of each version should solve the problem, or
we can replace . to unixtimestamp will resolve it (if macro is not used)

[potiuk]

I think it we already do similar thing elsewhere in Airflow - we are replacing . with dot I believe. It is in "views.py" - and it is mostly to prevent some subdag matching. I do not know BigQuery part well enough , but something like that might be a good idea:

        resp = {
            r.dag_id.replace('.', '__dot__'): {
                'dag_id': r.dag_id,
                'last_run': r.last_run.isoformat(),
            } for r in query
        }
        return wwwutils.json_response(resp)

It's really typical "escaping" thing :).

[turbaszek]

Or should we build the string and then use the following code with added .?

airflow/airflow/providers/google/cloud/operators/bigquery.py

Line 2071 in a33a919

exec_date = re.sub(r"\:|-|\+", "_", context['execution_date'].isoformat())

[turbaszek]

Btw. I'm not sure if using . in dag_id is the best idea. The dot is special for subdags if I'm not mistaken

[saisiddhant12]

Btw. I'm not sure if using . in dag_id is the best idea. The dot is special for subdags if I'm not mistaken

yes, it creates a sub-dag if we use . between parent & child, I agree with you @turbaszek ,
@nathadfield were you explaining something else as a bug ?

[nathadfield]

Well, perhaps this is something that I shouldn't have done early on, but we've specified a DAG version as part of the DagID using a '.'. Not had any problems with this until now but then we don't use subdags.

I can change our DAG names quite easily to eliminate this but I would argue that it shouldn't be possible to create a DAG if the id contains a restricted set of characters.

Perhaps the dag_id should also be constrained to the same criteria as above?