S3 without peers about gun

[amark]

"Kinda".

S3 does have a feature that allows you to upload directly from the browser, however for it to work you have to generate an authorization token from a server. As far as reads, you can set S3 to be a website bucket which will allow for reads.

So the answer is " yes" theoretically.

[speier]

Great! We are only concerned about reads, at the moment. So, if we set the bucket to be a website, how can we configure Gun to work with that bucket? Thanks!

[amark]

Question: are you wanting to use this for a single user? Or are you wanting this to work with multiple users?

If you are wanting it for multiple users then doing reads from S3 would be bad because you won't have any updates or realtime sync - this requires a server or WebRTC (which isn't implemented yet, and even WebRTC requires a stun/ice server).

If you're just making some single user hack for yourself, then you should know that gun already saves to local storage which persists across reloads on the same device (no sync). As far as S3 reads from a public bucket website, I haven't thought of this before - it is a good idea (although it comes with a lot of draw backs and loss of primary features), but there isn't code to handle this. If you know how to make public S3 bucket ajax requests from the client, I could probably hack it into a forked version of gun for you (although warning this probably shouldn't be done as it would lead to a lot of dead end code from feature loss of realtime sync), but I don't currently know how to do S3 ajax requests... especially given all the browser restrictions on cross origin and content-type problems (this was one of the reasons I dropped ajax and went with websockets and jsonp).

If I can GUESS right though, is that what you want is to use gun with all the realtime functionality yet not have to manage any server. Is that correct? That is my dream too! If so, I can probably let you use one of my existing servers or get one set up for you. That way you don't have to worry about managing a server. Is that what you want?

[speier]

Yes, basically we want to avoid managing servers, or at least having an option like S3 as a fallback solution to fetch data, possible with "realtime" sync (ajax w/ long polling could be fine).

At the moment we have our own WebSocket API (with http fallback) running on our own servers, but we'd like to introduce an option where our client-side application would be able functional even when our servers are down for whatever reason, so we are experimenting in this area...

[amark]

Right, agreed. The only problem is S3 acts like a file storage system, so it is incapable of certain features (or would be very slow and costly). You either need WebRTC between gun browser peers with S3 write access as backup storage (S3 cannot be used as a "middle man" to sync data between peers becaue S3 is a file storage system that does NOT run gun's conflict resolution algorithm, aka your data would get lost/overwritten) or you need some gun server(s) that browsers can realtime connect through as a proxy (even in WebRTC you need a stun/ice server unfortunately).

Unfortunately the state of the web is that we are still a ways off from having the type of apps that you (and I) envision, where things are fully peer to peer needing no servers, simply because the browser vendor's concerns with security exploits of such systems. However, GUN is making a step towards that direction, but it unfortunately requires some work around which I am trying to make easier for people (WebRTC is one of the biggest steps but has nasty coordination problems which still require some sort of server, which I think is dumb and why I currently haven't worked on it yet since it seems similar to just having websockets which is currently more reliable and stable).

So I don't advise going the S3 route cause that will lead to worse problems. I do definitely recommend the goal you guys are going for, but would suggest accomplishing it by still running some gun server(s) (which I can do for you), and the beauty of gun is that if those servers go down the browser will still have a copy (of some, maybe not all) of that data locally which can still be updated and saved to. When the other peers come back online again, when the browsers reconnect they'll sync the offline updates just fine.

There should be noted these two comments:

1. For your app's static files to still be loaded and work even in offline mode, you'll need to use app cache manifest. This has nothing to do with gun but will be needed, substack has some cool work on this called hyperboot.
2. I need to add better controls for telling gun in the browser what data it should store long term and what data it should expire or replace with more important data.

I want to make sure we are building gun in a direction that is more helpful for people like you, I want a database built for the people and by the people. If you would be willing to share your project in more detail I would love to learn.

[speier]

We only need one-way "sync" so basically just getting the data from S3 in case when our WebSocket API is unavailable.

[amark]

Sorry I got busy. So, idea? If the one-way (read only, not write) data is put in a public S3 website bucket, we could pull that into gun browser side. Sure you might not get the realtime features of GUN, but at least you could use the graph traversal stuff. Yeah?