Comments (31)
Super Graph is a community driven project and not just my project which is why we have these open discussions. As the only current core maintainer I'm more than happy to review and guide developers with any PR's they are working on. Yes it helps to come to some kind of consensus here through a discussion before a PR is worked on.
In short making Super Graph work as a library was always on the books this discussion thread was created as a way to gather more inputs to help with the design process. The sample code you submitted above to show how Super Graph as a module would work is really helpful so thanks for taking the trouble. It helps conceptualize how the API would look.
I agree there is no reason to force people to only use it as a standalone service when we can easily do both. As for auto-migrations I've also wanted this feature but personally have never seen it as yet used in real production systems. But I'm all for doing it as I rather maintain a GraphQL schema file instead of SQL migrations.
I'll start doing some initial looking into the Super Graph as a library thing sometime next week.
from graphjin.
It would be great If we can have an initial version without auto migration via schema file. We can later provide a cli to interscope a database to create a schema file for existing projects.
Yes, these are two very separate things.
I think that now we immediately need to use super-graph as an importable library in any Go project (maybe as a simple http
endpoint) because we would also increase the number of people who use it and therefore its testability on the battlefield.
After this it would be nice to open another thread on plugins and other long-term maintainability techniques.
from graphjin.
Great news to share, Super Graph is now a GO library that you can include in your own code. Fetch data in your own code with GraphQL instead of struggling with ORMs or complex SQL.
Checkout this example:
package main
import (
"database/sql"
"fmt"
"time"
"github.com/dosco/super-graph/core"
_ "github.com/jackc/pgx/v4/stdlib"
)
func main() {
db, err := sql.Open("pgx", "postgres://postgrs:@localhost:5432/example_db")
if err != nil {
log.Fatalf(err)
}
conf, err := core.ReadInConfig("./config/dev.yml")
if err != nil {
log.Fatalf(err)
}
sg, err = core.NewSuperGraph(conf, db)
if err != nil {
log.Fatalf(err)
}
query := `
query {
posts {
id
title
}
}`
res, err := sg.GraphQL(context.Background(), query, nil)
if err != nil {
log.Fatalf(err)
}
fmt.Println(string(res.Data))
}
from graphjin.
Currently Super Graph is using Chai, Would it be possible to use any other router/framework like Fiber https://fiber.wiki/ or Echo https://echo.labstack.com/ ? Both frameworks are faster than Chai.
Here is how I would like to use amazing Super Graph;
Define database schema and access control in schema file
Super Graph can use this schema to create/update database tables without requiring users to create migrations
# database.schema
# Define schema with Graphql SDL (https://github.com/genie-team/graphql-genie)
enum Role {
# Open to all requests
ANY
# Must be logged in
USER
# User must have created/be the type
OWNER
ADMIN
}
# Only users can create posts, anybody can read posts, only the person who created the post can update/delete it
type Post @partition(type: list, value: author) @auth(create: USER, read: ANY, update: OWNER, delete: OWNER) {
id: ID! @unique
title: String!
text: String
# Set a rule of "SELF" here that we can look for in our authenticate function so users aren't allowed to change it to other users
author: User @relation(name: "posts") @auth(create: USER, read: ANY, update: OWNER, delete: OWNER, rules: "SELF")
}
# Anyone can create users (aka signup), be able to see user info by default, can only update yourself, and only admins can delete
type User @auth(create: ANY, read: ANY, update: OWNER, delete: ADMIN) {
id: ID! @unique
username: String! @unique
# only users can see their own email, it's not public
email: String! @unique @auth(create: ANY, read: OWNER, update: OWNER, delete: ADMIN)
# only admins can read password
password: String! @auth(create: ANY, read: ADMIN, update: OWNER, delete: ADMIN)
posts: [Post] @relation(name: "posts")
# Only admins can alter roles, will need additional logic in authenticate function so users can only set themself to USER role
# So we set only:USER in the rules so we can find that later in our authenticate function
roles: [Role] @default(value: "USER") @auth(create: ANY, read: ADMIN, update: ADMIN, delete: ADMIN, rules: "only:USER")
}
Use Super Graph as a module
package main
import "github.com/gofiber/fiber"
import "github.com/dosco/super-graph"
import "github.com/jackc/pgx/v4"
func main() {
// Create new Fiber instance:
app := fiber.New()
db, _ := pgx.Connect(context.Background(), os.Getenv("DATABASE_URL"))
defer db.Close(context.Background())
graphql = super_graph.New()
graphql.config.db = db
graphql.schema_path = "./schema"
graphql.config.xxx = // Other configurations
// Extend queries
graphql.extend.queries(// query definition)
// Extend mutations
graphql.extend.mutations(// Mutation definition)
// Allowed named queries
graphql.whitelist.add( `query getUserById($userId: Int) {
user(id: 4) {
id
username
email
}
}`)
// Create route for graphql:
app.post("/api", graphql.handler)
// Custom endpoint
app.Get("/", func(c *fiber.Ctx) {
// Query on server
result , _ = graphql.query(// Graphql named query)
c.Send(result)
})
// Start server on "localhost" with port "8080":
app.Listen(8080)
}
from graphjin.
Thanks for those links I'll take a deeper look. Having read up on some of this stuff it seems if you poll using a join query where the original subscribed query gets its variables (user id, etc) from the joined table then you are not executing 100k queries but just 1 with 100k rows one per client. In theory this sounds more efficient to me on small and large subscription counts.
from graphjin.
while obvious off-topic I think polling is a very valid first approach and quick to implement.
Other implementations can always be added later on.
from graphjin.
OMG!
from graphjin.
I tried very quickly to integrate it into a project I'm working on (I'll continue later).
It would be great to be able to make it become a middleware (https://www.alexedwards.net/blog/making-and-using-middleware) in a middleware chain of any Go HTTP server.
Example
Now my server has this structure:
main () {
setup()
router := newRouter() // Gin, Echo, Chi, whatever
router.Use(...middlewares...)
// endpoints for GraphQL
router.Group(... {
router.Post(... handler for POST GraphQL calls)
})
}
How do you think I can use SuperGraph in this scenario?
Considering also that the endpoint for users must always be only one (e.g. localhost / graphql
).
Perhaps a solution may be to analyze the incoming query and see if it is among "SuperGraph's", then use middleware before the line:
router.Post(... handler for POST GraphQL calls)
or let it go to the next middleware that can handle it.
What do you think about it?
Did you have other use cases in mind that I didn't think of?
from graphjin.
@dosco Thanks for this amazing project. I would like to encode query response with https://github.com/fxamacker/cbor Will it be possible?
from graphjin.
Super Graph generates JSON using Postgres so if this cbor library can parse JSON then it would be possible. To switch out the encoding entirely to cbor would be very hard and introduce additional allocations and complexity.
from graphjin.
I want to use Supper Graph with Dart/Flutter and Its difficult to decode graphql query response in json to Dart types. this Dart package https://pub.dev/packages/cbor can decode cbor to dart types. It would be great If there is an option to send query response in cbor.
Super Graph is great and we can further make developers life easy by providing a data modeling/Auto migration and query builder feature like Prisma2 https://github.com/prisma/prisma2/blob/master/docs/data-modeling.md , Edgedb https://edgedb.com/roadmap or Facebook Ent https://edgedb.com/roadmap to make it a complete data access solution.
from graphjin.
I'll look more info cbor
when I get some time. It's surprising that Dart/Flutter has a hard time with JSON which is pretty much the standard.
Super Graph just uses the data model defined in the database I'm unclear on the value of duplicating that again on another file. Built-in support for DB migrations allows you to update and manage this data model having one more file to update can get out of sync and slow you down.
from graphjin.
Super Graph just uses the data model defined in the database I'm unclear on the value of duplicating that again on another file. Built-in support for DB migrations allows you to update and manage this data model having one more file to update can get out of sync and slow you down.
I wanted to say, It would be great If we can define data model in a schema file and not in the database and then Super Graph auto generate DDL and apply it to database. We can version control this file with our project code repo. Here is an example of a schema file from Prisma2 docs.
// schema.prisma
datasource sqlite {
url = "file:data.db"
provider = "sqlite"
}
model User {
id Int @id @default(autoincrement())
email String @unique
name String?
role Role @default(USER)
posts Post[]
profile Profile?
}
model Profile {
id Int @id @default(autoincrement())
user User
bio String
}
model Post {
id Int @id @default(autoincrement())
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
author User
title String
published Boolean @default(false)
categories Category[]
}
model Category {
id Int @id @default(autoincrement())
name String
posts Post[]
}
enum Role {
USER
ADMIN
}
Super Graph has made developers life very easy by providing database access from client via Graphql but we also need to access database from server. Go is great but relational database access with SQL is really hard. I have yet to find a good solution. Super Graph can fill this gap If it can provide Prisma2 like query and migration functionally in addition to Graphql API.
I would request you to please check Prisma2 https://github.com/prisma/prisma2/blob/master/docs/data-modeling.md and EdgeDB https://edgedb.com/roadmap when you get time.
from graphjin.
I can't wait for this!
from graphjin.
I have added a partition schema directive for declarative partitioning and query white listing to my proposal.
from graphjin.
@ansarizafar is there an easy way to integrate it today in a Golang project?
from graphjin.
@ansarizafar this is technically not hard to do expect for the reading schema stuff that does not exist so will have to be built, however I don't think we need it initially Super Graph does a good job automatically resolving schema and relationships directly from the database.
As for the part about adding and using named queries technically this is exactly how the Super Graph seed.js works. Not sure if you have seen this https://supergraph.dev/guide.html#seed-js
In the seed file you have a build-in function called graphql
so you can use graphql queries to seed your database with say test data or whatever. This function even through you call it in Javascript is executed internally the Super Graph GO code and supports all of it's syntax query
,mutations
, batch inserts-updates, whatever. Look at serv/cmd_seed.go
for the code there's a function called graphQLFunc(query string, data interface{} ...)
all you have to do is expose this. Yes it won't do prepared statements but thats ok for now the compiler is very fast and adding that in is pretty easy once this initial stuff is done. Adding to an allow.list
is not required we can just generate prepared statements on the fly.
var res = graphql("
mutation {
user(insert: $data) {
id
}
}", { data: data })
Note 1 - Super Graph uses a GO JS interpreter to run the seed.js file as the graphql
function returns JSON which is really easy to mold in Javascript.
Note 2 - For now only 1 instance of Super Graph can be created in your app since it makes use of some globals serv/cmd.go
I don't think this is an issue but if it is then the solution would be to package those globals into a struct and make it the Super Graph context.
from graphjin.
this is technically not hard to do expect for the reading schema stuff that does not exist so will have to be built.
Let's Do It.
I don't think we need it initially Super Graph does a good job automatically resolving schema and relationships directly from the database.
Super Graph's goal is to help developers to "Build web products faster" Auto migrations will help achieve this goal. We can define complete data structure with data access rules on tables/field easily in one schema file without using SQL and creating a migration file after every change in data structure.
The suggested schema is just using Graphql SDL so there is no need to interscope database. The SDL is easily extensible with directives, we can build schema features like declarative partitioning etc gradually.
As for the part about adding and using named queries technically this is exactly how the Super Graph seed.js works.
Super Graph complies graphql queries to single SQL statement and that's the main advantage, no other ORM (Prisma,GORM, Facebook ant etc) except Micosoft Entity framework has the ability to generate single SQL statement for eager loading. I have recommended an API to use queries/mutations on the server so that we don't have to use SQL and database driver directly.
// Query on server
result , _ = graphql.query(// Graphql named query)
Adding to an allow.list is not required we can just generate prepared statements on the fly.
The suggested featured allow developers to add queries to a white list. The white listed queries should be complied to single prepared SQL statement in advance and only white listed queries should be allowed from Graphql endpoint in production.
// Allowed named queries
graphql.whitelist.add( `query getUserById($userId: Int) {
user(id: 4) {
id
username
email
}
}`)
Super Graph has a great potential to become a complete and de facto database access solution for developers all around the world but I would not like to use it in its current form as I don't like to use a separate standalone system with a separate config file (Prisma1 failed exactly of this reason).
I don't like to define actions/remote joins in a configuration file to extend the Graphql Api. The suggested Query/Mutation extend and query on server are far better features than the current solution.
In the end Super Graph is your project, you have done the hard work and its your right to decide what better suits your needs.
from graphjin.
It would be great If we can have an initial version without auto migration via schema file. We can later provide a cli to interscope a database to create a schema file for existing projects.
from graphjin.
I'm testing it... I'll let you know ASAP!
Amazing work! Amazing work!
from graphjin.
Your example here: https://supergraph.dev/guide.html#stripe-api-example
having SuperGraph inside my project is a game changer!
I can call my own stripe()
client internally, not using HTTP calls!
Am I wrong?
from graphjin.
It should be relatively easy to build a middleware depending on the GOLang HTTP framework you are using. Below is an example I found if you're using standard lib. Thanks or the feedback I'm equally excited launching this and using it myself. You don't have to make it a middleware just use it in the handlers for you're existing REST endpoints in place of whatever ORM you might have used.
func supergraphMiddleware(next http.Handler) http.Handler {
<initial super graph here>
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
<call the GraphQL function here>
next.ServeHTTP(w, r)
})
}
from graphjin.
I can call my own
stripe()
client internally, not using HTTP calls!
Yes you should be able to,
from graphjin.
That is great. Is there a way to use core.NewSuperGraph() directly with a *pgxpool.Pool instance instead of *sql.DB? One would use pgx's stdlib.AcquireConn(), but on PGX4 they've made pgxpool.Pool the default pool and stidlib.AcquireConn() works only with *pgx.Pool and not *pgxpool.Pool.
Thanks.
Howe
from graphjin.
@howesteve funny story originally the Super Graph service used *pgxpool.Pool internally everywhere. I changed it to the more generic *sql.DB when I extracted the core API out. So now I use pgx/stdlib. Honestly their docs are very unclear on what the difference is as far as I know stdlib has a built in pool.
Maybe you could write your own sql.DB driver that wraps pgx.Pool?
from graphjin.
Sure, I'll work on it. But I was thinking that when (and if) subscriptions support arrive, LISTEN/NOTIFY support (i.e. some kind of conn.Listen()) method will be needed anyway, and that is not supported by sql.DB...
from graphjin.
I'm working on some early design ideas for subscriptions focus is on making it scale. Early experiments tell me that listen notify is does not scale as well and polling seems be the way to go here.
from graphjin.
Nice. Actually that seems to be the same conclusion of the hasura team when then implemented it two years ago:
https://github.com/hasura/graphql-engine/blob/master/architecture/live-queries.md
But I think listen/notify could scale better on some heavier loads. Ex:
50000 clients checking if a message arrived for their own user. That will require comparing a database field to a session variable, so one query will have to be issued for each client each time. If the polling timer is 500ms, that is 100000 queries per second? On heavier loads, that doesn't seem to end up well. It would be super expensive on servers that charge per database read.
Postgraphile guys chose a plugin-based approach and support LISTEN/NOTIFY or WAL monitoring:
https://www.graphile.org/postgraphile/live-queries/
Each approach comes with its pros and cons; polling always give perfect results but could easily overload the server, WAL requires extra client side programming and can only inspect physical columns, and LISTEN/NOTIFY require extra setup steps in the connection and configuring triggers.
Not a clear win here...
from graphjin.
I think we can close this, @dosco.
from graphjin.
I was keeping it open just till we land the public API for the service too,
from graphjin.
Closing this since the Serv package is also a library.
from graphjin.
Related Issues (20)
- Support `returns table (like <tablename>)` in stored procedure record returns. HOT 1
- Columns prefixed with "upper_" or "lower_" incorrectly parsed
- Support aggregation result as order field
- list query & id query having same args HOT 1
- count query field is not accepting correct arguments HOT 1
- webui: Error fetching schema HOT 1
- Perhaps there is a bug in the SetContextValues function within auth0.go. HOT 1
- Sorry, there was a problem running the beginner example and I don't know how to solve it HOT 4
- I have doubts about how to insert it password HOT 1
- I have a problem required variable 'user_id' of type 'integer' must be set HOT 4
- Does it not support mysql? HOT 1
- variable limit not work? HOT 1
- Introspection result missing inputFields HOT 1
- wasm / deno updates HOT 2
- Webshop example schema error with graphql-code-generator HOT 6
- How to use pgx.Pool
- Unable to connect postgres
- Dynamic configuration for big number of schemas
- wrong relation resolving in multiple schemas environment HOT 1
- Incorrect SDL types generated for insert/update HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from graphjin.