Add Spark DataSourceV2 Connector for Databricks Integration

[charphi]

Overview

Develop a Java-based Apache Spark DataSourceV2 connector that wraps sdmx-dl and enables seamless SDMX data querying within Apache Spark and Databricks environments.

Motivation

Currently, users who want to query SDMX data in Spark/Databricks must write custom code. A native Spark connector would provide:

• Direct SQL/DataFrame queries on SDMX data sources (ECB, IMF, OECD, Eurostat, etc.)
• Leverage sdmx-dl's existing functionality for fetching, parsing, and flattening hierarchical SDMX data
• Optimization support (filter pushdown, column pruning, partitioning)
• Easy distribution via JAR packaging for Databricks clusters

Proposed Solution

Architecture

Create a new module (sdmx-dl-spark) implementing Spark's DataSourceV2 API that wraps sdmx-dl's proven fetch data flattening logic:

Component	Purpose
SdmxTableProvider	Entry point; registers data source for .format("sdmx")
SdmxTable	Logical representation with schema and metadata
SdmxScan	Query planning and optimization
SdmxInputPartition	Parallel data chunks
SdmxPartitionReader	Fetches and converts SDMX data to Spark rows
SdmxSchemaMapper	Maps sdmx-dl's flattened output (Series key, ObsAttributes, ObsPeriod, ObsValue) to StructType

Data Format

Leverage sdmx-dl's existing fetch data command, which produces a flattened tabular format:

• Series:key – Dimension key (e.g., M.CHF.EUR.SP00.A)
• ObsAttributes:map – Observation-level metadata
• ObsPeriod:datetime – Time period
• ObsValue:number – Observation value

Usage Example

spark.read
  .format("sdmx")
  .option("provider", "ECB")
  .option("dataset", "EXR")
  .option("filter", "FREQ=D")
  .load()
  .createOrReplaceTempView("exchange_rates")

Key Decisions Needed

1. Series key handling:
   • Keep simple 4-column format as-is, or
   • Enhance by parsing Series keys into individual dimension columns based on flow metadata, or
   • Offer both as configuration options
2. Partitioning strategy: Partition by time period, region, or frequency
3. Filter pushdown: Map Spark filters to sdmx-dl query parameters
4. Dependency packaging: Fat JAR with shaded transitive dependencies for easy Databricks deployment

[charphi]

https://docs.databricks.com/aws/en/connect/spark-data-sources