marcogorelli / polars-plugins-tutorial

How you (yes, you!) can write a Polars Plugin

How you (yes, you!) can write a Polars Plugin

• ✅ Unlock super-high performance
• ✅ Have a tonne of fun
• ✅ Impress everybody with your superpowers

This repository is meant to accompany the tutorial from https://marcogorelli.github.io/polars-plugins-tutorial/.

Logo

Thanks to Olha Urdeichuk for the illustration.

polars-plugins-tutorial's Issues

Add example on how to do rolling calculations with time strings

It would be awesome to add some examples on how to efficiently do something like polars does with rolling_mean_by where you can take in a string, for example 30d and a datetime column and do the rolling calculation on a time window instead of simply a certain number of rows.

add example of struct output

Add example for plugin with state

Hello,

I am not sure if it is in the scope of the tutrial, but I think example of stateful plugin will be nice.
For instance, StringIndexer object that has two methods:

• fit: works on a dataframe with a string column, and save a mapping between every unique string to index.
• transform: works on a dataframe string cloumn, and transform every string value in the column to its corresponding index.

Similar to Spark's API.

Thanks!

add example of rolling_skew or rolling_kurtosis ?

Thanks MarcoGorelli !

The tutorial is very excellent. Could you give us more about rolling examples?

Thanks again! 😊

simpler crate example

this is way too complex and to most people probably uninteresting. perhaps try snowball stemmer?

add example with output_func_with_kwargs

add example with PlMutableString

add more types to the list example

it currently reads

binary_amortized_elementwise(values, weights, |values_inner, weights_inner| {

can we add types to values_inner and weights_inner to make it easier to understand? they should just be Series

Add an overview of optimisation tips (such as `apply_to_buffer`)

I presume a lot of people will be coming from the Python side of Polars. Creating a list of best practices and often used optimisations could help improve the performance of plug-ins written by people not too familiar with Rust yet. (Me included!)

Add example of generic List as input and output

For instance,

Suppose I want to take a DataFrame that contains column with a List of numbers, and returns a new List with all the non-zero elements indices.

Input example:

┌───────────┐
│ dense     │
│ ---       │
│ list[i64] │
╞═══════════╡
│ [0, 9]    │
│ [8, 6]    │
│ [3, 3]    │
└───────────┘

Output example:

┌───────────┬────────────────┐
│ dense     ┆ sparse_indices │
│ ---       ┆ ---            │
│ list[i64] ┆ list[i64]      │
╞═══════════╪════════════════╡
│ [0, 9]    ┆ [1]            │
│ [8, 6]    ┆ [0, 1]         │
│ [3, 3]    ┆ [0, 1]         │
└───────────┴────────────────┘

Produced with this code:

df = df.with_columns(
    pl.col('dense').map_elements(lambda l: [i for i, v in enumerate(l) if v != 0]).alias('sparse_indices')
)

I know it can be solved with an index and explode, but it's pretty non-trivial code.

Thanks.

How to write a UDF for polars that run concurrently?

I am writing a UDF with rust for polars, I want to know if polars will automatically run UDF concurrently or will I need to implement the concurrency myself in the rust code ?

add tips on avoiding vectors of `Option<T>`

Typo in `aggregate.md`?

Should this line in docs/aggregate.md change from [0] to [1]?

#[polars_expr(output_type=Float64)]
fn vertical_weighted_mean(inputs: &[Series]) -> PolarsResult<Series> {
    let values = &inputs[0].f64()?;
-    let weights = &inputs[0].f64()?;
+    let weights = &inputs[1].f64()?;

use `broadcast_try_binary_elementwise`

Add specialised Array chapter

commented out code in cookie cutter utils.rs references verstion 37.0

I just noticed that the plugin cookiecutter has commented out code for a function that accepts two columns, and it references an older version than the rest of the polars dependencies in the Cargo.toml (0.41.2). Figured it was worth mentioning.

// and make sure you have
//
// polars-arrow = { version = "0.37.0", default-features = false }
// polars-core = { version = "0.37.0", default-features = false }
//
// in your Cargo.toml file.

get rid of `downcast_iter`, can we teach this using only apply / apply_values / apply_generic / apply_values_generic / ?

Add unary_elementwise_values example

In the List at last, it was shown how to use binary_amortized_elementwise function.

I will be glad to see an equivalent unary usage example.

Thanks.

Set Polars>=1.3.0 as minimum version

Since we're using the pyo3_polars allocator, we need to make Polars>=1.3.0 the minimum required python-Polars version

So, the task is:

• in pyproject.toml, set Polars 1.3.0 as the minimum version
• Get rid of minimal_plugin/utils.py, and just use polars.plugins.register_plugin_function: see https://github.com/MarcoGorelli/polars-reverse-geocode/pull/5/files and MarcoGorelli/cookiecutter-polars-plugins#19 for the changes which need doing. The gist of it is that if Polars 1.3.0 is the minimum required version, then we no longer need to do version checks on Polars and we can get rid of the old way of registering plugins

simplify cumulative aggs

pola-rs/polars#14409

add essential Rust background

• printing 'hello world'
• variable assignment, mutability
• options and results
• vectors and iteration
• strings, allocations, ownership? too hard to teach exhaustively, but can show basics and link?

Add to Polars Plugin Introduction - Definition, Outcome

Having a brief introductory outline could be helpful for readers who are entirely new to Polars and plugins to know what to expect.

It can look something like:

What's a Polars plugin?

Plugins let you extend Polars with your own features to create a custom dataframe experience.

What you'll learn 

After you've completed this tutorial, you'll be able to:

• Boom
• Boom
• Boom