Skip to content

Support None comparison for nulls #1483

@ntjohnson1

Description

@ntjohnson1

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

>>> import datafusion as dfn
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1, 2, None]})
>>> df.filter(dfn.col("a") is None).count()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<path>/lib/python3.12/site-packages/datafusion/dataframe.py", line 499, in filter
    df = df.filter(ensure_expr(expr))
                   ^^^^^^^^^^^^^^^^^
  File "<path>/lib/python3.12/site-packages/datafusion/expr.py", line 250, in ensure_expr
    raise TypeError(EXPR_TYPE_ERROR)
TypeError: Use col()/column() or lit()/literal() to construct expressions
>>> df.filter(dfn.col("a") == None).count()
0
>>> df.filter(dfn.col("a").is_null()).count()
1

Describe the solution you'd like
I'd like to be able to compare against None like we can do with other scalar types. Ideally using both == and is semantics.

Describe alternatives you've considered
.is_null

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions