Incorrect conversion of pyarrow interval value to datafusion literal about datafusion-python

[timsaucer]

TODO: If the PR #666 merges in before this issues is corrected, the following examples in the examples/tpch folder will need to be updated

• q01_pricing_summary_report.py
• q04_order_priority_checking.py
• q05_local_supplier_volume.py
• q06_forecasting_revenue_change.py
• q10_returned_item_reporting.py
• q12_ship_mode_order_priority.py
• q14_promotion_effect.py
• q15_top_supplier.py
• q20_potential_part_promotion.py

[Michael-J-Ward]

It's probably related to this issue in arrow-rs: Rust Interval definition is incorrect.

Here's a godbolt link demonstrating the "1 month becomes 1 nanosecond" example. (I based that on a comment in a similar thread in duckdb-wasm).

I would suspect that if all code paths use the same impl, then datafusion-python wouldn't notice it, but perhaps that's wrong, or maybe not all code-paths use arrow-rs?

The error occurs in the pyo3 magic as we cross the python -> rust bridge. Notice the python side assert's pyarrow.Scalar, but then the rust-side receives a datafusion::ScalarValue. (aside: is this magic type conversion intentional?)

python side:

datafusion-python/datafusion/__init__.py

Lines 182 to 185 in 67d4cfb

	def literal(value):
	if not isinstance(value, pa.Scalar):
	value = pa.scalar(value)
	return Expr.literal(value)

rust side:

datafusion-python/src/expr.rs

Lines 229 to 232 in 67d4cfb

	#[staticmethod]
	pub fn literal(value: ScalarValue) -> PyExpr {
	lit(value).into()
	}

The error is already present before the rust-method is invoked, adding print statements on both sides of the bridge:

converting: MonthDayNano(months=1, days=0, nanoseconds=0)
converting: IntervalMonthDayNano("1")