mikestefanello / pagoda Goto Github PK

How about we use s3 and local ?

The directory info can be in the db.

this is a very standard approach.

so after a file is transacted into the file store and event updates the directory info on the db.

I love pagoda. It's an excellent piece of software.

However, I do feel like the default template engine from Golang std lib is quite limited.
Specially since, HTMX could benefit from a lot of features not found in go's official templating library.

Hence, I wanted to use Pongo2 (Similar to Django's template engine). How do I go about that?

I would like to know your opinion on using sqlc instead of Ent as the database layer. Do you think there is a good way to add it as an optional or interchangeable option alongside Ent, or should we consider creating a separate fork for it

It would nice to have an explaination on the command line to get the project to work quickely.

1. go get didn t work on my Ubuntu 20.04 I was asked to use go isntall
   "
   go: downloading github.com/google/go-cmp v0.5.6
   go get: installing executables with 'go get' in module mode is deprecated.
   Use 'go install pkg@version' instead.
   For more information, see https://golang.org/doc/go-get-install-deprecation
   or run 'go help get' or 'go help install'.
   "
2. A little explaination on how to install docker.
   In the readme it is assumed that we know docker... Some persons like me don t.
   On ubuntu do :
   snap install docker
   cd ~/go/pkg/mod/github.com/mikestefanello/[email protected]
   $ make up
   "
   Status: Downloaded newer image for postgres:alpine
   Creating pagodav040_cache_1 ...
   Creating pagodav040_db_1 ...
   Creating pagodav040_db_1 ... error

Creating pagodav040_cache_1 ... done
5e2b0267327e1f48384ad): Error starting userland proxy: listen tcp4 0.0.0.0:5432: bind: address already in use

ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint pagodav040_db_1 (b1f6e840f3529cf0eb014fb1428118e3b55481e8f5d5e2b0267327e1f48384ad): Error starting userland proxy: listen tcp4 0.0.0.0:5432: bind: address already in use
ERROR: Encountered errors while bringing up the project.
make: *** [Makefile:39 : up] Erreur 1

"

I am a golang dev. thanks for making this nice and clean golang base.

htmx really looks like an answer to so many gui issues.

would be great if you added a htmx example, so i can see how to join up the front and back.

Apologies if this does not belong in issues, but when I clone the project I don't see any syntax highlighitng on the gohtml files. Is there a step I am missing that would give me proper syntax highlighting? I checked VSCode extensions and didn't see anything. Should I just use default html highlighting?

Thanks!

The database port from the configuration is ignored when creating the postgres connect string. This defaults to the standard port 5432 which in my case is not where the postgres server is.

I updated in container.go to the following to be able to spin this up.

	getAddr := func(dbName string) string {
		return fmt.Sprintf("postgresql://%s:%s@%s:%s/%s",
			c.Config.Database.User,
			c.Config.Database.Password,
			c.Config.Database.Hostname,
			strconv.Itoa(int(c.Config.Database.Port)),
			dbName,
		)
	}

Can this project be deployed as a lambda function like so https://github.com/kaihendry/ltabus/blob/master/template.yml ?

Or does it require to be long running?

Setting project to go 1.21 then running go get -u all we encounter breakage with pegasus and mergo

Updating mergo was fixed with these commands:

go mod edit -replace "github.com/imdario/mergo=github.com/imdario/[email protected]"
go get -u all
go mod tidy

But when running it make run there's a compile error in XiaoMi/pegasus:

go run cmd/web/main.go
# github.com/XiaoMi/pegasus-go-client/idl/cmd
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:42:15: not enough arguments in call to iprot.ReadStructBegin
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:47:35: not enough arguments in call to iprot.ReadFieldBegin
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:61:26: not enough arguments in call to iprot.Skip
        have (thrift.TType)
        want (context.Context, thrift.TType)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:71:26: not enough arguments in call to iprot.Skip
        have (thrift.TType)
        want (context.Context, thrift.TType)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:76:25: not enough arguments in call to iprot.Skip
        have (thrift.TType)
        want (context.Context, thrift.TType)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:80:13: not enough arguments in call to iprot.ReadFieldEnd
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:84:12: not enough arguments in call to iprot.ReadStructEnd
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:91:15: not enough arguments in call to iprot.ReadString
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:100:18: not enough arguments in call to iprot.ReadListBegin
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:108:16: not enough arguments in call to iprot.ReadString
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/cmd/cmd.go:108:16: too many errors
# github.com/XiaoMi/pegasus-go-client/idl/base
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/base/blob.go:18:15: not enough arguments in call to iprot.ReadBinary
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/base/blob.go:27:27: not enough arguments in call to oprot.WriteBinary
        have ([]byte)
        want (context.Context, []byte)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/base/error_code.go:105:18: not enough arguments in call to iprot.ReadString
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/base/error_code.go:110:27: not enough arguments in call to oprot.WriteString
        have (string)
        want (context.Context, string)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/base/gpid.go:18:12: not enough arguments in call to iprot.ReadI64
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/base/gpid.go:30:24: not enough arguments in call to oprot.WriteI64
        have (int64)
        want (context.Context, int64)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/base/rpc_address.go:26:18: not enough arguments in call to iprot.ReadI64
        have ()
        want (context.Context)
../../go/pkg/mod/github.com/!xiao!mi/[email protected]/idl/base/rpc_address.go:35:24: not enough arguments in call to oprot.WriteI64
        have (int64)
        want (context.Context, int64)
make: *** [run] Error 1

I tried linking XiaoMi/pegasus to the apache hosted version but that did not work. Maybe I did something wrong

file: go.mod

replace github.com/imdario/mergo => github.com/imdario/mergo v0.3.16
replace github.com/apache/incubator-pegasus/tree/master/go-client => github.com/XiaoMi/pegasus-go-client v0.0.0-20220519103347-ba0e68465cd5

go mod graph:

github.com/mikestefanello/pagoda github.com/XiaoMi/[email protected]
github.com/mikestefanello/pagoda github.com/pegasus-kv/[email protected]
github.com/XiaoMi/[email protected] github.com/BurntSushi/[email protected]
github.com/XiaoMi/[email protected] github.com/agiledragon/[email protected]+incompatible
github.com/XiaoMi/[email protected] github.com/cenkalti/backoff/[email protected]
github.com/XiaoMi/[email protected] github.com/fortytw2/[email protected]
github.com/XiaoMi/[email protected] github.com/pegasus-kv/[email protected]
github.com/XiaoMi/[email protected] github.com/sirupsen/[email protected]
github.com/XiaoMi/[email protected] github.com/stretchr/[email protected]
github.com/XiaoMi/[email protected] golang.org/x/[email protected]
github.com/XiaoMi/[email protected] golang.org/x/[email protected]
github.com/XiaoMi/[email protected] gopkg.in/natefinch/[email protected]
github.com/XiaoMi/[email protected] gopkg.in/[email protected]
github.com/XiaoMi/[email protected] k8s.io/[email protected]
github.com/eko/gocache/[email protected] github.com/XiaoMi/[email protected]
github.com/eko/gocache/[email protected] github.com/pegasus-kv/[email protected]

I am trying to implement the functionality to bulk update as set out in https://htmx.org/examples/bulk-update/
It seems that the swap of the tbody element isn't being done properly since the checkboxes are not visible but they do appear in the html, and the button elements are repeated. (which seems to indicate that it is not just the tbody which is being swapped but other elements on the page (or maybe the whole page)
In your experience with htmx, do you have any tips about where to start troubleshooting this?

A really common pattern is where you need to allow users to upload images and i need to save its existence in the DB for later use.

For example you might have a web page with a form, and as part of that they need to upload a screenshot or a list of images.

Then you want to save the image in a local FS or S3, and record the URL ( or RID - resource ID ) in the DB, so that you have a record of the form data and also the image that they also uploaded with the Form.

So you need a Web template for image uploads also that can be easily included in other golang templates. This is because you need image upload GUI that can be used with different form golang templates.

is there a reusable pattern here ?

The above example if a very simple one. A generic pattern of having a "blobs" table in the DB that is just for holding a mapping of files to the db can also be useful because it can be reused for lots of use cases. You can then reference that "blobs" table from other tables in your DB.

In terms of transactional integrity, you save the file first and if no errors then save the RID into the "blobs" table.

There is also a side effect with this pattern that can occur if you allow mutation access to S3. An image is changed, and you need to tell the DB that it has changed. File systems and S3 have a Notification API for this. Here is the minio docs on this to illustrate: https://docs.min.io/docs/minio-bucket-notification-guide.html
In this case, you woudl want a webhook to call back into the Golang API.

Hope that i explained myself ok and that you think this is generic enough and crosses enough use case that its worth including.

For anyone reading this, please let me know what you think. Is this something you need ? Is it better to do this in a different way ?

feel free to ask me if i have not explained this very well.

There is a rust example here: https://www.reddit.com/r/htmx/comments/y64ian/wasmservice_proof_of_concept_for_htmx_webassembly/

The concept I like is to render off the server to ensure fast initial load and then boot strap the Service worker.

The Service worker is just a version of the golang backend that acts as a proxy.

Use cases ?

Quasi db and template rendering . In the same way that the cache uses tags and they are invalidated by the DAL , we could invalidate data and hence rendered templates in the service worker.

Materialised views are its data. The Bsckend DB produces data , and we cache it in Redis. We could extend it to the Service worker .

both the above examples require the Server to fire events to the Service worker to tell it that data is invalidate.

Posts from the Browser DOM to the Service Worker could do validation , and if passed get fed further to the backend where further validation is sometimes required.

Building a service worker just requires the service.js and the wasm ( originally golang ) will load.

The htmx Ajax would route through to the Service Worker. Just like it does now to the backend.

after successful post URL stays /user/login even if we are on home page.

there are two ways to fix this
in handler add HX-Push header https://htmx.org/docs/#response-headers
ctx.Response().Header().Set("HX-Push", url)

OR
dont do a 302 redirect and send html content back from handler ( in case of login 302 is better)

any other better way to fix this ?

Sorry for the unnecessary issue, but I had to say this starter project is really really cool. I love the decisions you guys have made on it, the initial project functionality is a really great demo/example, really nice touches with the docker-compose, hooking up the redis cache, and make scripts and everything. The documentation on the README is great too. Very well done thanks everyone!

https://github.com/kataras/blocks is a fully working competent wrapper for html/template, it would likely be an upgrade to use it's API instead of the current one due to blocks' full support of multi-layout multi-page and partials with an easy to use interface. It also supports markdown content ootb using blackfriday.

Your code is a little bit low code type of architecture.

I was thinking about also using Cuelang with it. This would mean that you dont have to have types everywhere.
Types can be used at runtime. But you can also generate JSON, YMAL, OPENAPI, and a ton more things with Cue.

https://cuelang.org/

https://cuelang.org/docs/usecases/ is a good place to start.

https://github.com/cue-lang/cue is golang ... Its even has the equivalent go.mod and go.sum.. Its really cool.

There are a ton of golang coders using it around github etc.

https://github.com/cueblox/blox

https://github.com/eonpatapon/tree-sitter-cue

When using HTMX the full template is not rendered on the backend (which means that flash messages are not processed). How should we use flash messages when using htmx powered forms ?

It is possible to include the "messages" but I was wondering if these are in the base template and you would like to show flash messages not in the form but in the base template. I'm not sure if this can even work, given the semantics of what is actually rendered (just the html which should be changed and not the whole template)

So any tips on patterns would be greatly appreciated!

Hi Mikestefanello,
In routes, I attempts to Query n latest data that saved in db before.
Assume, I want to Query 14 newest posts(latest created_at).
I coded this code:

// ./routes/posts.go
...
	smt, err := c.Container.ORM.Posts.
		Query().
		Order(ent.Desc("created_at")).
		Limit(14).
		Only(ctx.Request().Context())
...

but it returned error:

ent: posts not singular

Could you tell me what is correct way to use .Order()?

Thanks

The architecture is fast thanks for the smart layering of caches.

I was thinking that deployment examples might be useful for devs .

Fly.io can scale out Postresql and Redis automatically for example. Its requires slight changes to the Docker as fly only allow a Docker and not a Docker compose. This might sound like a problem but it is not and just requires a Dockerfile that can boot Redis and Postresql.

Anyway, there is no Dockerfile in the repo for the main golang code anyway, so we could make one that is designed for fly and scaling out Redis and Postresql if we want.

It can also be used for non fly too btw, but shaping the docker appropriately.

https://fly.io/docs/reference/redis/ for edit but you can also just boot your own.

https://fly.io/docs/postgres/ but you can also boot your own and scale out your own.

Fly makes all databases sync into all regions automatically.

The golang code scales to zero so is server less. So costs you close to nothing if you users hitting your system.

fly cli is golang and is insanely easy.

Fly anycast / bop automatically does nearest server LB, so its seamless scale out at that level too. Its pretty no brainer stuff

Most people use Cloudflare but any DNS works. SSL is done for you.

Unfortunately, Gorilla seems like it will no longer be maintained
https://github.com/gorilla/

Firstly I'd like to say thanks for putting together this "framework" - it's a great alternative to something like Buffalo or Bud. I plan on experimenting with it and adapting to my own base default template, trying a different template engine templ and adding an assets pipeline for js/css.

Here are a few observations / questions I noticed from using the template for the first time (some of which could become a PR based on feedback):

1. Pages are cached, but is it intended for Routes to be cached as well? For example, if you start the application and visit /about, change the /about route to /about-us (and rerun), then visit /about, the page renders. I would expect a change in the router would no longer route to any page, cached or not.
2. config.GetConfig() appears to be guessing where the config directory is based on what sub-package file is loading the config. I think a better approach would be something like:

	// Get the basepath / project root
	_, b, _, _ := runtime.Caller(0)
	basepath := filepath.Dir(b)
	...
	viper.AddConfigPath(basepath)

3. A make down command would be useful, e.g.: $(DCO_BIN) down

I am struggling to implement a redirect with querystring parameters
It seems the controller.Redirect can receive optional parameters which are empty interfaces.
I've tried a few variations on this theme but the parameters never find their way into the URL.
In this example the route defaults to "itemselect" when I would expect "itemselect?id=13&name=aaa"
Anybody know how to implement this correctly?

var queryParam1 interface{} = "id=13"
var queryParam2 interface{} = "name=aaa"
return c.Redirect(ctx, "itemselect", queryParam1, queryParam2)

Hi Mike, I find this project extremely helpful and with the trend that HTMX is going through right now I can only assume that a modern stack like this would serve a lot of community members. Was wondering whether it was worthwhile creating a Slack/Discord for like minded people working on this project to share their experience, help with any issues, recommendations and future roadmap. Happy to setup something and create a PR for the readme if that helps.

The structure and patterns are pretty nice, and really help with not reinventing the wheel again.

I was wondering if you have or are planning or wanting example repos that use this template.

Also i assume that this template approach allows golang developers to work in their real repo, and when the architecture of the template repo changes, they can do a git merge into their example repo ?

Anyways nice code

Thanks for this great project! I issue ctrl-C and make run every time I edit a file to rebuild and restart the server. Is this the correct approach or is there a filesystem watcher that I am not using properly?

Thank you for putting Pagoda together and for sharing it!

I could not find any examples for implementing API endpoints. What's the best way to do that in Pagoda? And can that be added to the docs?

Also, would it be possible to use the same controller or function to implement a json response and an HTML (template) response, and respond with either based on the request (content-type request header, for example)?

Great project!

Just noticed it looks like this
https://github.com/XiaoMi/pegasus-go-client

Has been migrated to https://github.com/apache/incubator-pegasus/tree/master/go-client

Any chance of including a websocket example using HTMX? I haven't been able to get the official docs example to work, but I'm probably overlooking something. Anyway, would be nice to establish a pattern for it here.

Thanks for making the Repo.

I'm having issues getting the ORM setup correctly. On a fresh clone of the repo following the README and running make up and then make run i'm getting the following error.

go run cmd/web/main.go
panic: failed to create database schema: postgres: unexpected number of rows: 1

goroutine 1 [running]:
github.com/mikestefanello/pagoda/pkg/services.(*Container).initORM(0xc000192370)
	/~/~/~/pagoda-template/pkg/services/container.go:175 +0x1a5
github.com/mikestefanello/pagoda/pkg/services.NewContainer()
	/~/~/~/pagoda-template/pkg/services/container.go:66 +0xdb
main.main()
	/~/~/~/pagoda-template/cmd/web/main.go:18 +0x25
exit status 2
make: *** [run] Error 1

Adding debug to Schema.Create() shows

2023/09/17 15:33:23 driver.Query: query=SHOW server_version_num args=[]
2023/09/17 15:33:23 driver.Tx(ce7868ac-32fc-483e-805f-df638139aab3): started
2023/09/17 15:33:23 Tx(ce7868ac-32fc-483e-805f-df638139aab3).Query: query=SELECT setting FROM pg_settings WHERE name IN ('lc_collate', 'lc_ctype', 'server_version_num', 'crdb_version') ORDER BY name DESC args=[]
panic: failed to create database schema: postgres: unexpected number of rows: 1
...

I've checked this query against the postgres container and it returns

make db
docker exec -it pagoda_db psql postgresql://admin:admin@localhost:5432/app
psql (16.0)
Type "help" for help.

app=# SELECT setting FROM pg_settings WHERE name IN ('lc_collate', 'lc_ctype', 'server_version_num', 'crdb_version') ORDER BY name DESC;
 setting
---------
 160000
(1 row)

Unsure where to go from here, specifically the unexpected number of rows: 1 error and how many rows is it expecting?

Hi Mikestefanello,

Your kit is awesome for starter like me.
I'm researching how to utilizing your kit to build an Web App.
My purpose is that:

• A task runs in background to fetch data from a website. Then save obtained data into Postgres db utilizing Ent
• When a client fetches router(GET), the data is load from Postgres db to show on HTML

My problem is:

• When saving data, like /routes/*.go, it needs 2 variables c(*Container) and ctx(echo.Context) to manipulate c.Container.ORM. But ctx of a task handler(in /worker/tasks/*.go) is context.Context(using reflect I see that: *context.timerCtx). Therefore I couldn't save data to db.
  Could you tell me how to save data to db in worker?(If there is any sample code, really appreciate your help)

Thanks

Hi thank you for building/maintaining this project. Quick question, Have we considered replacing the way we are currently styling the project with the popular Tailwind utility library? We could perhaps include this in the makefile as a build step for production or run a script for development.

Thank you for a great project! This project is perfect for me as a new go developer.

I am having hard time working with migrations, especially making some seed data. May I know what are the steps on how you guys create migrations? Thank you!

So far what I have tried:

1. atlas migrate new add-default-roles --dir "file://ent/migrate/migrations"
2. atlas migrate hash --dir "file://ent/migrate/migrations"
3. atlas migrate status --dir "file://ent/migrate/migrations" -u "sqlite://mydb.db"

On third, step I'm having an error:
Error: sql/migrate: connected database is not clean: found multiple tables: 4. baseline version or allow-dirty is required

On a mac its "docker-compose", not "docker compose".

I changed the makefile like this:

# mac name
DCO_BIN_NAME=docker-compose
# linux name
#DCO_BIN_NAME=docker compose

....

# Start the Docker containers
.PHONY: up
up:
	$(DCO_BIN_NAME) up -d
	sleep 3

Hi mikestefanello
Do you have some pointers about how to render the data returned by an ent orm query? For example do you create a struct for the data returned and range through the data returned by the query to populate the struct or what is your suggested approach or how you have used it in the past? I am just not so clear on how you feed the query data into Page.Data

After this commit, hot-reloading of templates is no longer working since the files are now embedded rather than being read on-demand.