DynamicOut downstream op using PolarsParquetIOManager fails with ComputeError: RecursionError about dagster

[ritchie46]

I just see this now. I will ensure we will have a glob flag upstream in Polars which users can toggle off.

[CrispyCrafter]

It seems that polars is interpreting square brackets [ ] as valid shell-glob syntax.
I would recommend that this pattern is not used for DynamicOutput persistent object paths as it is inevitable that a third party package that uses glob would fail.

This is from the glob man page:

       Character classes

       An expression "[...]" where the first character after the leading
       '[' is not an '!' matches a single character, namely any of the
       characters enclosed by the brackets.  The string enclosed by the
       brackets cannot be empty; therefore ']' can be allowed between
       the brackets, provided that it is the first character.  (Thus,
       "[][!]" matches the three characters '[', ']', and '!'.)

       Ranges

       There is one special convention: two characters separated by '-'
       denote a range.  (Thus, "[A-Fa-f0-9]" is equivalent to
       "[ABCDEFabcdef0123456789]".)  One may include '-' in its literal
       meaning by making it the first or last character between the
       brackets.  (Thus, "[]-]" matches just the two characters ']' and
       '-', and "[--0]" matches the three characters '-', '.', and '0',
       since '/' cannot be matched.)

https://man7.org/linux/man-pages/man7/glob.7.html

Here is the core bug being tracked in polars

pola-rs/polars#12876

[CrispyCrafter]

I just see this now. I will ensure we will have a glob flag upstream in Polars which users can toggle off.
@ritchie46

Although I agree that a glob flag would be useful in polars, it seems that in the case of Dagster, the convention of using square brackets is really the key issue here.

For instance, say an operation created as set out partitioned outputs with the pattern:
output/[partition_A]/[partition_B]/*.parquet one would like the ability to read all files in lowest folder root.

Perhaps it should be considered invalid syntax to write files containing square brackets and reserve [] to act as correct glob functionality for reads

[CrispyCrafter]

Compiled and tested against polars master branch. Can confirm the issue has been fixed.
It should be resolved with the latest polars release

[CrispyCrafter]

Scrap that latest got released two hours ago.