UDF support: CREATE FUNCTION DDL with pipeline SQL integration

[ryannedolan]

Summary

End-to-end UDF support: from CREATE FUNCTION DDL through SqlJob CRD to Flink data plane execution.

• CREATE FUNCTION DDL with clean syntax supporting Java classes, Python modules, inline code ($$...$$ dollar-quoting), file paths, URLs, and JAR archives via the AS/IN/LANGUAGE clauses
• SqlJob as intermediate CRD: pipelines route through SqlJob instead of FlinkSessionJob directly, carrying both SQL statements and UDF files for dynamic delivery
• FlinkRunner fetches from K8s API: the data plane receives a --sqljob=namespace/name reference and pulls SQL + files directly from the SqlJob CR at runtime (matching the production Proteus pattern)
• SqlJob controller revived: hoptimator-flink-adapter module re-added to the build, reconciler rewritten to current K8sContext API, creates FlinkSessionJob from SqlJob
• Demo UDFs: Java (Greet, StringLength) and Python (reverse_string) implementations baked into the Flink runner image, with Dockerfile updated for PyFlink support
• USING JAR emission: functions with JAR archives emit Flink's USING JAR clause for dynamic classloading

DDL examples

  -- Java class (on classpath)
  CREATE FUNCTION greet AS 'com.linkedin.hoptimator.flink.runner.functions.Greet';

  -- Java class from JAR
  CREATE FUNCTION greet AS 'com.example.Greet' IN 'https://artifactory/.../udfs.jar';

  -- Python module reference
  CREATE FUNCTION foo LANGUAGE PYTHON AS 'func' IN 'module';

  -- Python inline code with dollar-quoting
  CREATE FUNCTION foo RETURNS VARCHAR LANGUAGE PYTHON AS 'foo' IN $$
  from pyflink.table.udf import udf
  from pyflink.table import DataTypes

  @udf(result_type=DataTypes.STRING())
  def foo(s):
      return s[::-1] if s else None
  $$;

  -- Python from file
  CREATE FUNCTION foo LANGUAGE PYTHON AS 'foo' IN '/path/to/my_udfs.py';

Test plan

• Unit tests for Java UDFs (GreetTest, StringLengthTest)
• Unit tests for FlinkRunner (FlinkRunnerTest)
• DDL parsing tests for new syntax including dollar-quoting and LANGUAGE-before-AS (create-function-ddl.id)
• Integration test for SqlJob generation with UDF files (k8s-ddl-udf-files.id)
• Integration test for demo UDFs with real class names (k8s-ddl-udf-demo.id)
• All existing integration tests updated for SqlJob output format
• Full build passes (checkstyle, spotbugs, all tests)
• End-to-end on cluster: SqlJob → reconciler → FlinkSessionJob → FlinkRunner fetches SqlJob → executes SQL with working UDFs

🤖 Generated with Claude Code