Skip to content

Cannot use now() or any TIMESTAMP WITH TIMEZONE type of constant in the filter. #70

@anlowee

Description

@anlowee

For the query like below:

select msg from data2 where t."$date" < now() limit 1;

The initial filter operation generated by Presto is LESS_THAN(CAST(DEREFERENCE(t, 0)), 7204114125742080), where the CAST(DEREFERENCE(t, 0)) is cast the first (0-th) nested field of t to the same type of the second operands, which is of TIMESTAMP WITH TIMEZONE data type. It is a packed 64-bit value used for TIMESTAMP WITH TIME ZONE, where the upper bits are epoch milliseconds (shifted left by 12 bits) and the lower 12 bits are the time-zone key.

In ClpFilterToKqlConverter.handleLogicalBinary(), the ClpExpression doesn't accept these two kinds of operands so they are just set to Optional.empty() and there is no pushdown generated.

        ClpExpression leftExpression = left.accept(this, null);
        ClpExpression rightExpression = right.accept(this, null);

Also, the result of this query is empty which is incorrect.

presto:default> select msg from data2 where t."$date" < now() limit 1;
 msg 
-----
(0 rows)

Query 20250925_174625_00005_n2nnx, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
[Latency: client-side: 0:01, server-side: 0:01] [683K rows, 0B] [1.24M rows/s, 0B/s]

I validated that the TIMESTAMP type can be converted to TIMESTAMP WITH TIMEZONE type, so this is not caused by we compare a TIMESTAMP column with a TIMESTAMP WITH TIMEZONE value, and it is our own bug (see here, the from_unixtime returns a TIMESTAMP type data):

presto:default> select (from_unixtime(1700000000) AT TIME ZONE 'America/Toronto');
                  _col0                  
-----------------------------------------
 2023-11-14 17:13:20.000 America/Toronto 
(1 row)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions