Store functions in YAML file about intake-spark

[martindurant]

The reason behind this, is to prevent the execution of arbitrary code upon loading a catalog. I don't know how you could include actual serialised functions in YAML and load safely. You could, however, include strings to code that lives in modules, like "package.module.function", and it would be reasonable for intake-spark to acquire the ability to parse these at run-time. Intake already does something similar for dereferencing driver classes.

(note that, as far as I know, spark-sql queries are truly equivalent to their functional counterparts except for user-defined functions)

[jecolvin]

After looking through the source files and reading up more on PyYAML, I've discovered the solution. Your YAML file could look something like this:

sources:
  spark_dataframe:
    args:
      args:
      - - read
      - - jdbc
        - - jbdc:blah/
          - "TBL"
        - column: TBL_KEY
          lowerBound: 0
          numPartitions: 4
          properties:
            password: pwd
            user: user
          upperBound: 11000000
      - - withColumn
        - - foo
          - !!python/object/apply:pyspark.sql.functions.lit [1]
    description: ''
    driver: intake_spark.spark_sources.SparkDataFrame
    metadata: {}

You have to put the list with the function args in the same line using the square bracket syntax.

A decent amount of our queries look like sdf.groupby("foo").pivot("bar").agg([F.sum("col1"), F.countDistinct("col2")]) as we're having to create various higher-level groupings from the original data and then pivot off of that to create features.

Also now that I know this capability is possible I'll probably look into saving more complex transforms as custom functions bundled with our catalog that get called (for example for our utilization dataset I get the range of years the user is requesting and run a for loop to pivot out certain features by year, as typically our data scientists are using one year's utilization features to predict some utilization-based label for the next year).