Raising errors for pyspark dataframe validation about kedro-pandera

[felipemonroy]

Hi @michal-mmm, could you make the PR with that change? And see what @Galileo-Galilei thinks about it. I am happy to help if you can't

[Galileo-Galilei]

Hi, sorry for not responding earlier. I think we should go forward. I suggest that we implement in general some kwargs to be passed to the validate function :

my_dataset: 
    type: ...
    filepath: ...
    metadata: 
        pandera: 
            schema: ...            
            validate_kwargs: 
                lazy: true

and then in the hook:

metadata["pandera"]["schema"].validate(data, **metadata["pandera"]["validate_kwargs"])

Feel free to open to a PR, and eventually suggest a different design.

[felipemonroy]

Hi @michal-mmm, I like the idea of adding lazy=False when calling the validation method. We should also evaluate including tests with a PySpark dataset (and even others like Polars) in order to check that errors are raised.

In the future, we should evaluate how to handle validations with lazy=True, for instance, with an after-pipeline-run hook.

[Galileo-Galilei]

Closed by #78