For discussion: adding Pydantic for metadata models about pyairtable

[xl0]

@mesozoic , could you please clarify if there will also be a provision for generating the Pydantic schema for the table metadata?

[mesozoic]

@mesozoic , could you please clarify if there will also be a provision for generating the Pydantic schema for the table metadata?

My first thought here is just to use Pydantic to represent all the complex metadata we get back from the API (schemas, webhooks, etc). Building a Pydantic model to reflect a table's data is an interesting idea, but I'm not sure how useful it will be (since the ORM module does not use Pydantic under the hood).

If I'm not quite getting your meaning, perhaps you could clarify your use case?

[xl0]

The way I'm using pyairtable with Pydantic at the moment, and I'm pretty new to Pydantic:

Base class for any AirTabel data:

Note that the record_id and record_created_time are excluded from serialization. This way we can get a record, modify it, and call table.update() on the same object.

from pydantic import BaseModel, Field

class AirTableRow(BaseModel):
    record_id: str = Field(None, alias=str("id"), exclude=True)
    record_created_time: str = Field(None, alias=str("createdTime"), exclude=True)

    class Config:
        extra = "forbid" # Catch typos and field name changes
        allow_population_by_field_name = True

    @classmethod
    def from_dict(cls, d):
        return cls(record_id=d["id"], record_created_time=d["createdTime"], **d["fields"])

Now for each table, we need to define the schema.

Note that the calculated field, again, is excluded from serialization, as it can't be part of an update.

class Data(AirTableRow):
    field1: str = Field(None, alias=str("Field 1"))
    field2: int = Field(None, alias=str("Field 2"))

    calculated_field: int = Field(None, alias=str("Calculated Field"), exclude=True)

Now using the class with pyairtable:

table = pyairtable.Table(AT_API_KEY, "appDsQdcFsh1bJlGE", "Test")

data = [ Data.from_dict(d) for d in table.all() ]
data

[Data(record_id='rec4yN9Jr6cjH6zbW', record_created_time='2023-07-09T07:39:40.000Z', field1='Hello there.', field2=345, calculated_field=357),
 Data(record_id='recJSJdoiBdNOqeTP', record_created_time='2023-07-09T07:39:40.000Z', field1='General Kenobi!', field2=123, calculated_field=138)]

Here is what happens if we serialize the data:

list(map(lambda x: x.dict(by_alias=True, exclude_unset=True), data))

[{'Field 1': 'Hello there.', 'Field 2': 345},
 {'Field 1': 'General Kenobi!', 'Field 2': 123}]

So to update a record, we can do

data[1].field2 = 4321
table.update( data[1].record_id, data[1].dict(by_alias=True, exclude_unset=True) )

{'id': 'rec4yN9Jr6cjH6zbW',
 'createdTime': '2023-07-09T07:39:40.000Z',
 'fields': {'Field 1': 'Lalalala', 'Field 2': 345, 'Calculated Field': 353}}

Or with a new Data object (I split it into 2 lines for better readability):

update = Data(field1="Lalalala")
table.update(data[0].record_id, update.dict(by_alias=True, exclude_unset=True))

{'id': 'recJSJdoiBdNOqeTP',
 'createdTime': '2023-07-09T07:39:40.000Z',
 'fields': {'Field 2': 4321,
  'Field 1': 'General Kenobi!',
  'Calculated Field': 4336}}

This provides at least some type of safety and working IntelliSense. There is definitely space for improvement, for examples we could have a way to

data = Data.all()

return a List[Data].

What I'm suggesting is, let's have an official tool that takes the table schema from airtable, and generates the pydantic boilerplate. Would it make sense? Is there enough information provided by the API to generate it automatically?

[mesozoic]

@xl0 What you're describing seems like an interesting approach to consider when we get around to autogenerating ORM classes from table schemas (probably 3.0; see roadmap in #249). I think we'll need to weigh whatever advantages or new features it provides against whatever ways it might break backwards-compatibility with the current ORM module.

This thread was intended solely to suggest using Pydantic (vs. plain old dicts) for metadata like schemas, webhooks, etc. Seems like that's probably acceptable, since this is not the only thread where I've heard general enthusiasm for using Pydantic in more places :)