piccolo-orm / piccolo_api Goto Github PK
View Code? Open in Web Editor NEWASGI middleware for authentication, rate limiting, and building REST endpoints.
Home Page: https://piccolo-api.readthedocs.io/en/latest/
License: MIT License
ASGI middleware for authentication, rate limiting, and building REST endpoints.
Home Page: https://piccolo-api.readthedocs.io/en/latest/
License: MIT License
I am struggling a bit with Piccolo's authentication systems. I have a FastAPI app and am wrapping it with Starlette's AuthenticationMiddleware, as hinted in the docs, with the joint SessionAuth and SecretTokenAuth providers. The secret token seems to be working alright; my API client won't get results without the correct header-cum-token. However whatever I do on the browser gives me "Auth failed for all backends". I can't get to the login endpoint, though this appears properly configured according to the docs. I tried 'allow unauthenticated' to see if this would loosen up the permissions, but even the FastAPI docs give me this error. Is there any robust example app with SessionAuth to see how everything should be organized?
Following this:
scripts
folderWhen using the session_login
endpoint, it would be nice to have hooks (basically callback functions) which are called at different parts of the verification process.
There are many situations where this would be valuable. Here are some examples:
There will have to be a minimum of three hook types:
The session_login
function will have a new hooks
argument:
def session_login(
...,
hooks: t.List[SessionLoginHook] = []
):
...
The hook will accept a request argument, and additional data (such as email, BaserUser
instance etc).
If the hook raises an Exception
, then the login is stopped, and an error is shown to the user.
It would be great if this functionality was available for all of the auth types that we provide (token auth etc), but session auth is the most important.
The code coverage was increased dramatically in this issue: #63
One missing area is a test for the session logout endpoint - would be nice to add this at some point.
There are some outstanding MyPy errors in the codebase, meaning MyPy can't be enabled as part of the build pipeline.
We have out of of the box login / logout endpoints, which render nice templates:
The current approach to customising it is to create an entirely new template using template_path
(see docs).
Most of the time providing an entirely new template is overkill - we just need to customise some minor things like the background colour, and button colour, to make it more consistent with whichever app we're building.
A nice solution would be for session_login
/ session_logout
to accept a css_variables
argument. For example:
session_login(css_variables={'button-color': 'red'})
We now have the powerful visible_fields
filter in PiccoloCRUD
.
It would be nice if a consumer of the API could visit the schema endpoint, and see which fields are available (this would include fields in related tables).
For example:
{
...
"fields": ["id", "name", "director.name", "director.id"],
...
}
So they can know which fields they can include in the GET param:
/?__visible_fields=name,director.name
See: https://www.cockroachlabs.com/docs/stable/timestamp.html
Cockroach always uses Timestamptz
(set to +00:00) when Timestamp
is specified. This is confusing to piccolo migrations.
gnat@gnat:~/Desktop/piccolo_examples-master/headless_blog_fastapi$ piccolo migrations forwards all
BLOG
----------------------------------------------------------------
๐ 1 migration already complete
๐ No migrations need to be run
SESSION_AUTH
----------------------------------------------------------------
๐ 1 migration already complete
โฉ 1 migration not yet run
๐ Running 1 migration:
- 2019-11-12T20:47:17 [forwards]... The command failed.
expected DEFAULT expression to have type timestamp, but 'current_timestamp() + '01:00:00'' has type timestamptz
For a full stack trace, use --trace
Suggestions? Too new to Piccolo to know what course of action to take in order to make this work smoothly.
Otherwise Cockroach seems to work fairly flawlessly with Piccolo, because of asyncpg.
Based on this discussion:
In some frameworks, such as Django, views are wrapped with transaction handling logic. If an exception is raised, then the transaction is rolled back.
To use transactions is more explicit currently in Piccolo - a context manager has to be used within the view itself.
@app.get("/")
async def my_endpoint():
try:
async with MyTable._meta.db.transaction():
# a bunch of queries
except Exception:
return JSONResponse({"message": "something went wrong"})
return JSONResponse({"message": "success"})
It would be nice to have a decorator which wraps a view with this logic:
@app.get("/")
@piccolo_transaction
async def my_endpoint():
...
Now that Piccolo API has a fairly broad set of features, it's important to improve upon the documentation.
I think the following needs adding:
Related to piccolo-orm/piccolo_admin#167
Example table:
class Recipe(Table):
name: str = Varchar(length=100)
content: str = Text()
author: BaseUser = ForeignKey(references=BaseUser)
category: str = Varchar(length=50)
ingredients: List = JSONB()
extras: Dict = JSON()
Currently, any JSON and JSONB fields from a Piccolo Table is being serialized as strings. It technically works, but parsing it as a string outputs this:
Setting value_type = pydantic.Json
for JSON and JSONB fields makes it look a lot better:
Now we have codecov integrated, it's easier to track code coverage.
There are some areas of the code base which require additional testing.
Some of the endpoints would benefit from CAPTCHAs for bot protection - e.g. login and register.
Currently, only offset based pagination is supported in PiccoloCRUD (using the __page
and __page_size
query params). There are issues with this, as outlined in this article.
Modify PiccoloCRUD so it accepts a __cursor
GET parameter. The value will be the ID of a row for now, but encoded as string. Encoding it as a string gives more flexibility (for example, we may want to return a UUID instead of an integer in the future).
The response should then be something like:
{
"rows": [...],
"next_cursor: "1234"
}
If both the __page
and __cursor
GET params are passed to an endpoint, an error should be returned.
There's a limitation at the moment with filtering via PiccoloCRUD
- you can't specify 'null' for some column types.
Take this as an example:
The user can't type in 'null'. In Piccolo Admin, a solution is to add 'null' as an option in the operator dropdown (similar to 'Equals' etc).
When PiccoloCRUD receives an operator value of 'null' it should ignore the value it receives, and treat it as null instead.
Now that Python 3.10 has been released, we need to check if Piccolo API works with it, and update the GitHub actions accordingly.
This PR from the main Piccolo repo is a good example of what's required:
Originally posted by sumitsharansatsangi August 18, 2022
await MOVIE_POSTER_MEDIA.delete_unused_files()
doesn't support for Array
column type.
Received following error :
File ~/.cache/pypoetry/virtualenvs/proj-haD7zQkN-py3.10/lib/python3.10/site-packages/piccolo_api/media/base.py:297, in MediaStorage.delete_unused_files(self, number_shown, auto)
274 async def delete_unused_files(
275 self, number_shown: int = 10, auto: bool = False
276 ):
277 """
278 Over time, you will end up with files stored which are no longer
279 needed. For example, if a row is deleted in the database, which
(...)
295
296 """
--> 297 unused_file_keys = await self.get_unused_file_keys()
299 number_unused = len(unused_file_keys)
301 print(f"There are {number_unused} unused files.")
File ~/.cache/pypoetry/virtualenvs/ducofastapi-haD7zQkN-py3.10/lib/python3.10/site-packages/piccolo_api/media/base.py:272, in MediaStorage.get_unused_file_keys(self)
266 """
267 Compares the file keys we have stored, vs what's in the database.
268 """
269 db_keys, disk_keys = await asyncio.gather(
270 self.get_file_keys_from_db(), self.get_file_keys()
271 )
--> 272 return list(set(disk_keys) - set(db_keys))
TypeError: unhashable type: 'list'
```</div>
Piccolo API has great Pydantic support, but the docs are a bit hidden away under the 'CRUD' section.
It would be good to move them, so 'Serialisation' is a top level topic in the docs, and contains much more examples about how to use create_pydantic_model
.
Currently, the __page
and __page_size
query params aren't documented.
Also, PiccoloCRUD
should have a max_page_size
argument, to limit abuse of an endpoint.
If the max_page_size
is exceeded, an error should be returned. A 403 feels most appropriate, with a body such as The page size limit has been exceeded
.
It would be useful to show an app which has all of the session auth components working together (session_login
, session_logout
and SessionsAuthBackend
).
Here's an example:
import datetime
from fastapi import FastAPI
from piccolo_api.csrf.middleware import CSRFMiddleware
from piccolo_api.openapi.endpoints import swagger_ui
from piccolo_api.session_auth.endpoints import session_login, session_logout
from piccolo_api.session_auth.middleware import SessionsAuthBackend
from starlette.middleware import Middleware
from starlette.middleware.authentication import AuthenticationMiddleware
from starlette.routing import Route
app = FastAPI()
app.mount(
"/login/",
session_login(),
)
private_app = FastAPI(
routes=[
Route("/logout/", session_logout()),
],
middleware=[
Middleware(
AuthenticationMiddleware,
backend=SessionsAuthBackend(
increase_expiry=datetime.timedelta(minutes=30)
),
),
Middleware(CSRFMiddleware, allow_form_param=True),
],
docs_url=None,
redoc_url=None,
)
# The Swagger docs which come with FastAPI don't support CSRF middleware, so we mount
# a custom one which Piccolo provides (accessible at /private/docs):
private_app.mount("/docs/", swagger_ui(schema_url="/private/openapi.json"))
@private_app.get('/my-secret-endpoint/')
def my_endpoint():
# This is just a normal FastAPI endpoint, and is protected by Session Auth
pass
app.mount("/private/", private_app)
if __name__ == "__main__":
import uvicorn
uvicorn.run(app)
Would you please be able to add a license to this repo? Although piccolo-orm/piccolo specifies an MIT license, there is no such licensing on this repo.
Currently broken by pydantic - waiting for a fix.
../../../virtualenv/python3.9-dev/lib/python3.9/site-packages/fastapi/dependencies/utils.py:265: in get_typed_annotation
annotation = evaluate_forwardref(annotation, globalns, globalns)
../../../virtualenv/python3.9-dev/lib/python3.9/site-packages/pydantic/typing.py:50: in evaluate_forwardref
return type_._evaluate(globalns, localns)
E TypeError: _evaluate() missing 1 required positional argument: 'recursive_guard'
We're missing a blank link before :param
.
piccolo_api/piccolo_api/media/base.py
Lines 203 to 205 in 9d1aa35
The approach is basically the same as this:
Hello, sorry if this is a naive assertion, but shouldn't SessionsAuthBackend.authenticate return (AuthCredentials(scopes=[]), UnauthenticatedUser()) if not token but self.allow_unauthenticated == True?
Just like we've done with other endpoints. There are many potential use cases:
PiccoloCRUD
currently has hooks like pre_save
, which is used to modify the data before insertion into the database. We should add a post_save
too, which is run after insertion into the database. It can be used for logging, sending an email etc.
@dantownsend I've just upgraded my infrastructure to the latest version of FastAPI and I'm now getting a strange error when trying to view the Swagger UI. In the first instance FastAPI is giving me a "failed to load API definition" which is a problem with retrieving the OpenAPI JSON spec. Digging a bit more into the problem however reveals that something is going on when creating the Pydantic response models.
Fetch error
Response status is 500 /api/openapi.json
Digging a bit more into the problem however reveals that something is going on when creating the Pydantic response models. I assume this is soemthing to do with create_pydantic_model
. I'm getting an error like the one here, which is the closest I can find to any explanation of the issue:
The relevant bit in my case is:
File "/usr/local/lib/python3.7/site-packages/fastapi/openapi/utils.py", line 364, in get_openapi
flat_models=flat_models, model_name_map=model_name_map
File "/usr/local/lib/python3.7/site-packages/fastapi/utils.py", line 28, in get_model_definitions
model_name = model_name_map[model]
KeyError: <class 'pydantic.main.Lemma'>
I'm stumped, as nothing has changed at all in my model / table definitions since the update. Is this likely something due to the interaction of Piccolo - Pydantic - FastAPI or out of Piccolo's scope?
Having a screen grab showing the generated Swagger docs would be nice on this page:
https://piccolo-api.readthedocs.io/en/latest/fastapi/index.html
The FastAPIWrapper / PiccoloCRUD combo is one of the most powerful things in the Piccolo ecosystem, and a screen grab will make it clearer the functionality it gives you.
That page is also unusual in that it uses Task
as an example - we should change it to use Movie
, as we can get example code / screen grabs from Piccolo Admin.
As reported here:
PiccoloCRUD
is what powers Piccolo Admin. When someone wants to create a new row, we call the 'new' endpoint. For example:
GET /api/tables/movie/new/
{
"id": null,
"name": "",
"rating": 0.0,
"duration": 0.0,
"director": null,
"oscar_nominations": 0,
"won_oscar": false,
"description": "",
"release_date": "2022-09-01T19:14:42.268809",
"box_office": 0,
"tags": [],
"barcode": 0,
"genre": 0,
"studio": null
}
This returns the default values for the new row. The problem is with email columns - if the default value is ''
, then it's not a valid email, so Pydantic won't serialise it, and we get a 500 error.
You can see the relevant code here:
piccolo_api/piccolo_api/crud/endpoints.py
Lines 848 to 859 in 8dc04f2
Noticing that the response from PATH seems to be a text-type instead of json-type response. I'm noticing in the code that the PATH response uses JSONResponse
instead of the CustomJSONResponse
used by many other endpoints.
As far as I can see, functions either use json response with a dict input:
JSONResponse(self.pydantic_model.schema()
or CustomJSONResponse
with a string input:
json = self.pydantic_model_plural(
include_readable=include_readable,
include_columns=tuple(visible_columns),
nested=nested,
)(rows=rows).json()
return CustomJSONResponse(json, headers=headers)
the PATCH response is the only one using JSONResponse
with a string param.
I'm guessing this is just a bug (and not a critical one at that) but before I submit a PR to fix I just wanted to make sure.
PiccoloCRUD
has an option called read_only
.
I think it's worth phasing this out in favour of an argument called methods
.
It could be something like:
PiccoloCRUD(methods=[HTTP.get])
If read_only
is provided in kwargs, then we can just map that to methods=[HTTP.get]
and provide a deprecation warning.
It means you have more fine grained control over which methods to expose:
PiccoloCRUD(methods=[HTTP.get, HTTP.post])
When moving to GitHub Actions from Travis, coveralls had to be disabled as I couldn't get it to work.
It needs adding back.
We have two great new features for create_pydantic_model
(exclude_columns
and nested
).
I'd like to make sure that they work together. For example:
create_pydantic_model(Band, nested=True, exclude_columns=(Band.manager.id,))
In which case the Manager
sub model would omit it's id
column.
At version 0.29.0 there seems to be no support for ARRAY types in the CRUD generation.
Code is generated, but an ARRAY of VARCHAR comes out as a single Varchar, and the field cannot be properly used.
In the Piccolo Admin the Arrays of Varchar are handled ok. Are there already plans to support Array types in CRUD?
I find a small bug. After the last changes in Piccolo ORM Table.__init__
ignore_missing
argument is now _ignore_missing
and we have to change this line to row = self.table(_ignore_missing=True)
otherwise we get 500 Internal Server Error from get_new
method. If you want I can change it.
We added the __visible_fields
filter to PiccoloCRUD
- but it only works on the bulk GET endpoint, not the detail endpoint.
# Works:
GET /?__visible_fields=name,director.name
# Doesn't currently work:
GET /1/?__visible_fields=name,director.name
It would be good if we modified PiccoloCRUD
, so it could accept a media_storage
argument (like create_admin
does in Piccolo Admin).
PiccoloCRUD(
Movie,
media_columns=[
LocalMediaStorage(Movie.poster, media_path='/srv/media/movie_posters/')
]
)
We could then add a new GET parameter called something like __media_urls
, which then auto adds the URL to the response for accessing the media.
GET /1/?__media_urls
{
'id': 1,
'name': 'Star Wars',
'poster': 'some-file-abc-123.jpg',
'poster_url': '/media/some-file-abc-123.jpg',
}
Once this is in place, we can refactor Piccolo Admin slightly, so it passes the media_storage
argument to PiccoloCRUD
, and lets PiccoloCRUD
do all of the heavy lifting.
Related to:
If asyncpg isn't installed, Piccolo API won't work:
As asyncpg is now an optional requirement in Piccolo, we need to work out a way of making Piccolo API handle it gracefully if asyncpg isn't installed.
Piccolo Admin has a read only mode, which is useful when demoing it online.
This PR updates the register
and create_user
endpoints to have a read only mode, because we're going to use them in Piccolo Admin.
I want to create blockchain backend app, based on piccolo. So I need custom User
model (generally, without password field). As suggested in docs, I must implement custom user app. But I also want to use session mechanism. How can I achieve that case?
BTW, Great project! I've struggled with fastapi-sqlalchemy stack and it's back and forth model transitions. I've looked other ORMs, compatible with pydantic and FastAPI (Tortoise, SQLModel, Databases, GINO), but those projects looks too young or unmaintained. And only piccolo had my heart from first look). Thank you and keep doing your great work!
Currently, PiccoloCRUD
(and hence FastAPIWrapper
) allow you to filter by the row ID, but you can't filter by a list of IDs.
For example:
# Currently supported:
DELETE /api/tables/movies?id=1
# Proposed:
DELETE /api/tables/movies?id=1,2,3,4,5
This would be a very useful feature, and would make bulk deletes more efficient.
As suggested here:
It would be great if create_pydantic_model
could generate nested models.
create_pydantic_model(Band, nested_models=[Band.manager])
This would output something equivalent to this:
from pydantic import BaseModel
class ManagerModel:
name: str
class BandModel:
name: str
manager: ManagerModel
Based on this discussion:
The session_logout
endpoint only works when you send a POST request.
We can modify this endpoint to return a simple logout button when a GET request is received (in a similar way to how session_login
works). It will make session_logout
easier to use out of the box.
Similar to what we did for Session Auth, maybe we can add a full example app?
Given a piccolo model containing an enum:
class CustomerActivationStatus(str, enum.Enum):
Signup = "Signup"
Active = "Active"
Disabled = "Disabled"
FreeSubscriptionActive = "FreeSubscriptionActive"
class Customer(Table):
signup_guid = UUID()
email: str = Varchar()
created_at: datetime = Timestamptz()
activation_status = Varchar(
choices=CustomerActivationStatus,
default=CustomerActivationStatus.Signup.value
)
...and using create_pydantic_model
to generate a dynamic pydantic class:
CustomerPatchResponseModel = create_pydantic_model(Customer)
..and then using that as a fastapi response model:
@api_v1_router.patch(
"/customer/guid/{signup_guid}",
response_model=CustomerPatchResponseModel,
responses={409: {"model": None}}
)
async def patch_customer_by_guid()
<the code>
Seems to break fastapi's openapi spec auto-generation:
Traceback (most recent call last):
File "/projects/customer-api/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
return await self.app(scope, receive, send)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
await super().__call__(scope, receive, send)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
raise exc
File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
raise exc
File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/routing.py", line 656, in __call__
await route.handle(scope, receive, send)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle
await self.app(scope, receive, send)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/routing.py", line 61, in app
response = await func(request)
File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 161, in openapi
return JSONResponse(self.openapi())
File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 136, in openapi
self.openapi_schema = get_openapi(
File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/openapi/utils.py", line 387, in get_openapi
definitions = get_model_definitions(
File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/utils.py", line 28, in get_model_definitions
model_name = model_name_map[model]
KeyError: <class 'pydantic.main.Customer'>
If I exclude the enum field, the problem goes away:
CustomerPatchResponseModel = create_pydantic_model(
Customer,
exclude_columns=(Customer.activation_status, Customer.email,)
)
I'm pretty sure fastapi is able to deal with regular pydantic models including enums, so I suspect there's something about how create_pydantic_model
generates these dynamic models.
A simple dictionary which the user can define, which maps the English word / phrase to their local language.
For example:
# For Portuguese users:
register(translations={'register': 'registro'})
The only option at the moment is to create your own template. This is definitely easier.
We could probably include a few translations out of the box for some language (just pre-defined dictionaries):
from piccolo_api.shared.auth.translations import register_portuguese
# For Portuguese users:
register(translations=register_portuguese)
For example:
/api/tables/movie?__fields=id,name
Would return:
{
"rows": [
{
"id": 17,
"name": "The Hobbit: The Battle of the Five Armies",
},
]
}
This is generally useful, but would be great for Piccolo Admin too, as when a user uses a TableConfig
to specify that the list page should only show a subset of columns, we aren't over fetching data.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.