BUG: No kwargs in df.apply(raw=True, engine="numba") about pandas

[lithomas1]

Hm, this doesn't look too hard to fix, looking at the error (the nb_compat_func line).

It's been a while since I've looked at any numba stuff, though.

Are you interested in potentially contributing a PR?

[auderson]

Are you interested in potentially contributing a PR?

Sure, but I find the solution may not be that trivial:

Since numba generally requires the argument is known when function is defined, so any kind of varargs is not well supported.
It dosen't support **kwargs, and IIRC it also had some bugs with *args.

This means we can't simply add **kwargs to nb_looper:

pandas/pandas/core/_numba/executor.py

Lines 26 to 52 in 58461fe

	@numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)
	def nb_looper(values, axis):
	# Operate on the first row/col in order to get
	# the output shape
	if axis == 0:
	first_elem = values[:, 0]
	dim0 = values.shape[1]
	else:
	first_elem = values[0]
	dim0 = values.shape[0]
	res0 = nb_compat_func(first_elem)
	# Use np.asarray to get shape for
	# https://github.com/numba/numba/issues/4202#issuecomment-1185981507
	buf_shape = (dim0,) + np.atleast_1d(np.asarray(res0)).shape
	if axis == 0:
	buf_shape = buf_shape[::-1]
	buff = np.empty(buf_shape)
	if axis == 1:
	buff[0] = res0
	for i in numba.prange(1, values.shape[0]):
	buff[i] = nb_compat_func(values[i])
	else:
	buff[:, 0] = res0
	for j in numba.prange(1, values.shape[1]):
	buff[:, j] = nb_compat_func(values[:, j])
	return buff

A possible solution is to dynamically create an intermediate function for each case. Use the example above:
df.apply(np.nanquantile, q=0.5, raw=True, engine="numba")

In this case, we dynamically defines:

@register_jitable
def nb_compat_func(values, q=0.5):
    return user_func(values, q=q)

which holds the **kwargs and the only required argument is values and this nb_compat_func will be used by nb_looper just as before.

However, I felt this solution is somewhat too hacky and not clean. Do you have any other ideas?

[lithomas1]

I think *args might be the best way to deal with this, if it's possible to fix this.
It's how we pass the kwargs to the other numba stuff like the numba groupby stuff.

Do you remember the issue you had with *args?