Can't configure Kubernetes and Celery workers in Helm Chart

[csp33]

Official Helm Chart version

1.7.0 (latest released)

Apache Airflow version

2.4.3

Kubernetes Version

1.25

Helm Chart configuration

No response

Docker Image customizations

No response

What happened

Current Helm chart uses the "workers" section to configure Kubernetes or Celery workers parameters (resources, affinity, etc.).
However, when using CeleryKubernetesExecutor, the "workers" section is used to configure the Celery ones, making Kubernetes workers settings available only via podTemplateFile, which can be difficult to manage.

I suggest splitting "workers" into "kubernetesWorkers" (used in pod_template_file) and "celeryWorkers" (used by the celery ones)

What you think should happen instead

Helm Chart should allow to configure both kind of workers in an easy way

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Why not - if someoene would like to pick it, i marked it as good first issue.

[csp33]

I could take this one. How about this proposal? values defined at the "workers" level will be taken by both celery & k8s.

workers:
  safeToEvict: false
  celery:
    resources: 
      limits:
        cpu: 1
        memory: 1Gi
      requests:
        cpu: 1
        memory: 1Gi
  kubernetes:
    resources:
      limits:
        cpu: 240m
        memory: 875Mi
      requests:
        cpu: 240m
        memory: 875Mi

@potiuk

[potiuk]

Just make a PR - we (or others) can easier discuss it there.

[amoghrajesh]

@potiuk if nobody is actively working on this, I would like to take it up.

[potiuk]

Feel free

[eladkal]

Is this problem still relevant? It's mostly about CeleryKubernetesExecutor which is no longer needed after AIP-61.

[rcheatham-q]

Yes this is still an issue because this is more of an issue with the helm chart than it is about the specific executor. If someone wants to configure both a Celery executor and a Kubernetes executor after 2.10 releases they will still encounter this issue.