Getting "Invalid kube-config file" when trying to run SparkKubernetesOperator

[Coqueiro]

I'm running an Airflow cluster using CeleryExecutor inside a Kubernetes cluster, after installing cncf.kubernetes backport package. I already did some testing with the Spark Operator inside the cluster and I'm able to run SparkApplications smoothly by applying them with kubectl. I'm trying now to make an Airflow DAG execute them. When doing so, I ran into this message:

[2020-12-14 22:18:10,609] {taskinstance.py:1150} ERROR - Invalid kube-config file. No configuration found.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 67, in execute
    namespace=self.namespace,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 127, in create_custom_object
    api = client.CustomObjectsApi(self.api_client)
  File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 35, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 108, in api_client
    return self.get_conn()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 102, in get_conn
    config.load_kube_config(client_configuration=self.client_configuration)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/kube_config.py", line 739, in load_kube_config
    persist_config=persist_config)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/kube_config.py", line 701, in _get_kube_config_loader_for_yaml_file
    'Invalid kube-config file. '
kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found. 

I tried creating a connection with the following extra:

{"extra__kubernetes__in_cluster":True}

But it didn't work either. How can I correctly configure my airflow deployment so that I can make a task that can use the SparkKubernetesOperator, creating a SparkApplication within the same cluster?

I'm using Airflow 1.10.12 with Python 3.7.

[Coqueiro]

Please, close this discussion. I opened an issue:
#13076

[XD-DENG]

The folk who initiated the question has shared more details and the fix in the issue page above.