Comments (10)
@Fingel I think those issues are all solvable, but will require some changes to Piccolo.
- Custom column types could be registered in
piccolo_conf.py
, which solves the migration issue. - In terms of adding custom methods like
ST_Dwithin
, the best analogy so far is theJSONB
column type, which has anarrow
method. Something similar to this could be implemented. - With raw SQL in where clauses - this also shouldn't be too tricky, and I can add this.
I should be able to get 1 and 3 done in the next day or so. Otherwise you might want to fork it for now.
from piccolo.
@Fingel Thanks for the kind words.
You're right - it should be possible to create custom column types without too much hassle by subclassing Column
. I should mention this in the docs.
I don't know much about the Geography column type, but you'll probably need something like this:
class Geography(Column):
value_type = str
def __init__(
self,
shape: str = 'POINT',
number: t.Optional[int] = None,
default: t.Union[str, t.Callable[[], str], None] = "",
**kwargs,
) -> None:
self._validate_default(default, (str, None))
self.shape = shape
self.number = number
kwargs.update({"shape": shape, "number": number})
super().__init__(**kwargs)
@property
def column_type(self):
if self.number:
return f"GEOGRAPHY({self.shape}, {self.number})"
else:
return f"GEOGRAPHY({self.shape})"
I'm interested to know how you get along. If you're able to implement PostGIS functionality, I'd like to merge it into the main piccolo repo, or have a separate piccolo_gis repo.
from piccolo.
Hi @dantownsend ,
I did start our writing something that looks like your example. I wanted to make it easy on myself to I defined a Point(Column)
which looked a lot like your example, with plans to generalize into full Geography type later.
The first issue I ran into when trying to create a migration was this:
module 'piccolo.columns.column_types' has no attribute 'Point'
It looks like the migration manager assumes all classes are defined within piccolo.columns.column_types
. I could fork the project and work in there, with the assumption that maybe support for external columns could be added later?
The next hurdle will be querying. PostGIS uses special functions like so:
SELECT * FROM source WHERE ST_Dwithin(source.location, 'SRID=4035;POINT(1 1)', 100)
for example.
Here is where it looks a little more difficult. I'm still going through the code, but I'm not quite sure yet how a custom method like this could be implemented. So that you could so something like this:
MyTable.objects().where(MyTable.location.ST_Dwithin('SRID=4035;POINT(1 1)', 100)).run_sync()
Any ideas on how to approach that?
Alternatively (which could go a long way to help other niche use cases) would be to support raw sql inside of a Where. So you could do something like this:
MyTable.objects().where(
(MyTable.name=='foo') &
(raw("ST_Dwithin(location, 'SRID=4035;POINT(1 1)', 100)"))
).run_sync()
But I have no idea of the feasibility of that.
from piccolo.
Hi @dantownsend
I was able to implement geometry/geography columns in my fork here:
https://github.com/piccolo-orm/piccolo/compare/master...Fingel:feature/postgis_funcs?expand=1
Some questions:
- geography fields in particular really need to use gist indexes instead of btree. I noticed that gist is referenced here: but I do not believe anything other than btree is implemented. What do you think about adding the ability for columns to specify which kind of index they should use?
- Database extensions. In order for these columns to work, the PostGIS extension needs to be installed. If you want to test it out, I'd recommend spinning up an instance of the Postgis Docker image here. It's kept up to date with PostgreSQL, but comes with the extension installed. People not using this image will have to install postgis manually. One thing that is nice (which Geodjango does) is automatically run "CREATE EXTENSION 'postgis' IF NOT EXISTS;" so that any new database created will have the extension installed automatically. This is really helpful for test databases, for example. How about a way to specify custom sql to be run at certain hooks, or something like that?
Looking forward to continuing on this project!
from piccolo.
@Fingel You've made some great progress.
With the index types, it would be easy enough to expose this in the Column
constructor as an index_type
arg. What complicates matters is making it work with migrations, if someone was to change the index_type
. It's not impossible to do, but anything which touches migrations usually takes a bit longer to implement.
The PostgresEngine
has a prep_database
method, which currently just sets up the uuid extension.
piccolo/piccolo/engine/postgres.py
Line 270 in c0d34ea
Extending this to also setup other extensions would be pretty straightforward. PostgresEngine
would just need to accept a list of extensions names.
I've added the ability to run raw SQL in where clauses: #57
If this is what you were expecting, I'll release it tomorrow. Unfortunately Travis CI is painfully slow now, and I really need to switch to Github Actions.
from piccolo.
Hi,
I think the raw where PR is great, I'm glad it was merged.
I still think a PostGIS extension for Piccolo would be great. For example, It would allow us to make working with GIS fields a little more user friendly by being able to leverage tools like Shapely to covert the textual representation of geometries into useful python objects. This would require installing additional dependencies you probably wouldn't want in Piccolo core.
I'm not really sure how to proceed at this point. I could keep working in my branch, but it would be good to know if creating an extension would be possible, or if keeping it super simple so that the fields can be included in core. Thoughts?
from piccolo.
@Fingel I've made a couple of updates, which should make using custom column types easier.
- You can now specify an
extensions
argument inPostgresEngine
, and it will try and create that extension when the engine starts. This is currently in master. - I've just created a pull request, which should fix the issue of custom column types in migrations #69
There are still some limitations though. PostGIS may support custom DDL statements, which the Piccolo migrations don't support. This could mean some more refactoring of Piccolo.
Merging the PostGIS stuff into core is still an option - shapely could be an optional dependency.
Can you think of any other blockers for integrating GIS with Piccolo, either as part of the main library, or a separate package?
from piccolo.
@dantownsend This is good stuff! I will try factoring the GIS column types out of my fork and test it with #69. If it goes well and I can develop the columns as a separate package, that would be great. Why don't we see how that goes and if the package looks good enough we can consider merging it back into core?
from piccolo.
@Fingel Cool - makes sense.
from piccolo.
Going to close this for now - custom column types should now be possible.
from piccolo.
Related Issues (20)
- Be able to extract parts of timestamps / dates
- Update Litestar asgi template HOT 1
- [Litestar] nested not working with create_pydantic_model response HOT 8
- Something like «makemigrations» HOT 5
- name 'Serial' is not defined HOT 11
- Atomic transaction? HOT 6
- `ModelBuilder` doesn't support `LazyTableReference`
- [Enhancement] Exclude `graphlib` from test coverage HOT 1
- Litestar asgi template currently does not work properly
- `Ruff` as a linter for Piccolo HOT 3
- Tweak how `ModelBuilder` creates JSON values
- auto migrations fails when table in schema HOT 3
- Intermittent TypeError in get_sql_value: isinstance() arg 2 must be a type, a tuple of types, or a union HOT 2
- How to use join? HOT 2
- Implement support for Pydantic v2 HOT 2
- Allow user-defined alternatives to piccolo_app and piccolo_conf files HOT 1
- instance.delete() not work as expected HOT 2
- Properly fix `slotscheck` for Python 3.12
- Type of "create_pydantic_model" is partially unknown (Python 3.11) HOT 3
- Column definition not parsed in class Table HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from piccolo.