kaiko-ai / typedspark Goto Github PK

This project looks great, and might be what I'm missing on my projects, what would be great, to speed up adoption and integration, would be an interface we can use to generate stubs for schemas, given a spark dataframe as an input e.g:

from typedspark.helpers import generate_stubs


my_person_df = get_person()
print(generate_stubs(my_person_df))

This would really help on big projects with lots of data sources that need schemas (including mine)

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: File contents are invalid JSON but parse using JSON5. Support for this will be removed in a future release so please change to a support .json5 file name or ensure correct JSON syntax.

This repository currently has no open or pending branches.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: Invalid JSON (parsing failed)
Message: Syntax error: expecting end of expression or separator near ] "auto

Currently there is no way to set the nullability of a field and when calling the get_structtype function to get the schema the nullability is hardcoded as true.

Hi,

Thank you for this project, really helpful for people using type hints!
I'd like to know if there's a way to annotate a Struct column that can have a varying schema?
All examples I've seen in docs indicate a Struct column needs to have a particular schema.

For example, I'd like to have a Resource[T] dataframe, with a struct column resource_properties of type T, such that T is a Python TypeVar, or at the very least be able to have a column with type Any so that the type linter ignores it and the developers will know how to treat the values.

I wanted to have a function accept a Union[] of Schema, but when I try that I hit this error:

TypeError: issubclass() arg 1 must be a class

Is there a way to express this?

The unit tests currently don't pass for Python 3.11.9. As a temporary fix, the ci/cd is constrained to use 3.11.8 for now.

Interestingly, the other supported versions (3.9, 3.10, 3.12) work without problems.

I'll debug the problem later. Currently, I can't install Python 3.11.9 with pyenv.

It would be nice for the DataSet to support the Spark cache operations to support this use-case:

cached_ds: DataSet[A] = original_ds.cache()

These cache operations return a PySpark DataFrame:

• DataFrame.cache()
• DataFrame.persist()
• DataFrame.unpersist()

I would like to contribute on this organisation,
If you have any bugs or want the implementation of the feature then I'm available.

Thank you