Skip to content

Support the null value in bloom_filter_agg Spark aggregate function - #458

Open
weixiuli wants to merge 28 commits into
oap-project:updatefrom
weixiuli:fix_bloom_filter
Open

weixiuli wants to merge 28 commits into
oap-project:updatefrom
weixiuli:fix_bloom_filter

Conversation

@weixiuli

Copy link
Copy Markdown

Currently, the velox BloomFilterAggregate checks the input row and throws an exception if there are some null values in the row. So we need to be consistent with spark's behavior and ignore null values.

The spark BloomFilterAggregate will Ignore null values. https://github.com/apache/spark/blob/6cdca10f148433664b3e2be6f655b0ddba817537/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/BloomFilterAggregate.scala#L180-L188

 override def update(buffer: BloomFilter, inputRow: InternalRow): BloomFilter = {
    val value = child.eval(inputRow)
    // Ignore null values.
    if (value == null) {
      return buffer
    }
    updater.update(buffer, value)
    buffer
  }

weixiuli pushed a commit to weixiuli/gluten that referenced this pull request Dec 12, 2023
@zhztheplayer
zhztheplayer force-pushed the update branch 2 times, most recently from 13e79b6 to 8a6ef2b Compare December 13, 2023 07:11
@weixiuli weixiuli changed the title Support the null values in bloom_filter Spark aggregate Support the null value in bloom_filter_agg Spark aggregate function Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants