feat(spark): implement Spark width_bucket function#17331
Conversation
| } | ||
|
|
||
| fn output_ordering(&self, input: &[ExprProperties]) -> Result<SortProperties> { | ||
| // FIXME copy |
There was a problem hiding this comment.
this I am not sure if necesary output_ordering, for this I put this, only for these,
If its correct, drop de FIXME, thanks
There was a problem hiding this comment.
Hi, I drop the FIXME
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @davidlghellin for intensive testing, checked the slt tests are in sync with expected results in https://spark.apache.org/docs/latest/api/sql/#width_bucket
Waiting for CI
|
Thanks @comphead , I'm checking this because I don't understand the errors I'm getting on my machine. I'm not sure if it's something related to my setup or if I missed something in the configuration. |
| 5 | ||
|
|
||
| query I | ||
| SELECT width_bucket(INTERVAL '0' DAY, INTERVAL '0' DAY, INTERVAL '10' DAY, 10) |
There was a problem hiding this comment.
check interval
There was a problem hiding this comment.
I'll take a look to verify these errors
| return Ok(vec![Float64, Float64, Float64, Int32]); | ||
| } | ||
|
|
||
| let all_duration = matches!(v, Duration(_)) |
There was a problem hiding this comment.
its probably can be rewritten using pattern matching?
| 6 | ||
|
|
||
| query I | ||
| SELECT width_bucket(10.0, 0.0, 0.0, 5) |
There was a problem hiding this comment.
Can the test with null/nan be added? to use nan please check snippet
isnan(-'NaN'::FLOAT)
or
select greatest(arrow_cast('NAN','Float64'), arrow_cast('NAN','Float64'))
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @davidlghellin I think this PR is a good to go!
2 participants
Which issue does this PR close?
part of #15914
Rationale for this change
Migrate spark functions from https://github.com/lakehq/sail/ to datafusion engine to unify codebase
What changes are included in this PR?
implement spark udf width_bucket
https://spark.apache.org/docs/latest/api/sql/index.html#width_bucket
Are these changes tested?
unit-tests and sqllogictests added
Are there any user-facing changes?
now can be called in queries