example_gcp_bigquery_dts with 404 error for the gcp_bigquery_start_transfer task

[ludsh2000]

I tried to start a workflow by using the example_gcp_bigquery_dts.py. It can successfully create the bigquery data transfer task and get the right transfer_config_id. But when start the "gcp_bigquery_start_transfer" with the BigQueryDataTransferServiceStartTransferRunsOperator, it will return 404 error.

Is there any idea on how to resolve this problem? BTW, I could successfully manually execute the bigquery data transfer task within the console created by airflow.

Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 985, in _run_raw_tas
    result = task_copy.execute(context=context
  File "/usr/local/lib/airflow/airflow/providers/google/cloud/operators/bigquery_dts.py", line 290, in execut
    metadata=self.metadata
  File "/usr/local/lib/airflow/airflow/providers/google/common/hooks/base_google.py", line 425, in inner_wrappe
    return func(self, *args, **kwargs
  File "/usr/local/lib/airflow/airflow/providers/google/cloud/hooks/bigquery_dts.py", line 235, in start_manual_transfer_run
    metadata=metadata or ()
  File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/bigquery_datatransfer_v1/services/data_transfer_service/client.py", line 1110, in start_manual_transfer_run
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,
  File "/opt/python3.6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call_
    return wrapped_func(*args, **kwargs
  File "/opt/python3.6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callabl
    six.raise_from(exceptions.from_grpc_error(exc), exc
  File "<string>", line 3, in raise_fro
google.api_core.exceptions.NotFound: 404 Requested entity was not found
[2021-03-15 04:10:06,009] {taskinstance.py:1196} INFO - Marking task as FAILED. dag_id=example_gcp_bigquery_dts, task_id=gcp_bigquery_start_transfer, execution_date=20210315T040908, start_date=20210315T041004, end_date=20210315T041006
[2021-03-15 04:10:10,426] {local_task_job.py:102} INFO - Task exited with return code 1

[turbaszek]

Have you checked if the transfer_config_id which is passed to gcp_bigquery_start_transfer is valid?

[turbaszek]

@ludsh2000 do you still need help?