maxpert / marmot Goto Github PK
View Code? Open in Web Editor NEWA distributed SQLite replicator built on top of NATS
Home Page: https://maxpert.github.io/marmot/
License: MIT License
A distributed SQLite replicator built on top of NATS
Home Page: https://maxpert.github.io/marmot/
License: MIT License
I need to adjust the nats Jetstream code so it supports gateways and super clusters. This will allow replication between all you data centers globally without by data sync clashes.
https://docs.nats.io/running-a-nats-service/configuration/gateways
then , it’s just a matter of changing the config
https://docs.nats.io/running-a-nats-service/configuration/gateways/gateway
Users will tend to hit their closest data centre, and so we will not have issues with eventually consistency of mutations hitting Sqlite DB. So all good.
Adjust the Pocketbase example to use this, and then test using HTTP calls from many POPS ( points of presence, and then check DB consistency globally.
Logging to NATS would make doing this type of work way way easier, and we have nats embedded anyway. Then from the NATS CLI you can see exactly what is going on.
https://github.com/synadia-io/rethink_connectivity/blob/main/17-microservices-architectures/slog.go has a great example and video showing how easy this is.
Need to add NATS Cli to the setup script too
PB also has a global logging feature too.
Every-time there is a new release for Marmot the button should just update itself without requiring new deploy, and take people to appropriate GH release page.
Just kicking the tires on latest tagged release. It almost works...
The embedded NATS servers are not finding each other with errors like
my Makefile:
# https://github.com/maxpert/marmot
BIN=$(PWD)/.bin
export PATH:=$(PATH):$(BIN)
print:
init:
rm -rf $(BIN)
mkdir -p $(BIN)
dep: init
curl -L https://github.com/maxpert/marmot/releases/download/v0.8.8-alpha.4/marmot-v0.8.8-alpha.4-darwin-arm64.tar.gz | tar -zx -C $(BIN)
start:
chmod +x $(BIN)/marmot
chmod +x $(BIN)/examples/run-cluster.sh
cp $(BIN)/marmot $(BIN)/examples
cd $(BIN)/examples && ./run-cluster.sh
And the output:
chmod +x /Users/apple/workspace/go/src/github.com/gedw99/kanka-cloudflare/modules/nats/examples/marmot/.bin/marmot
chmod +x /Users/apple/workspace/go/src/github.com/gedw99/kanka-cloudflare/modules/nats/examples/marmot/.bin/examples/run-cluster.sh
cp /Users/apple/workspace/go/src/github.com/gedw99/kanka-cloudflare/modules/nats/examples/marmot/.bin/marmot /Users/apple/workspace/go/src/github.com/gedw99/kanka-cloudflare/modules/nats/examples/marmot/.bin/examples
cd /Users/apple/workspace/go/src/github.com/gedw99/kanka-cloudflare/modules/nats/examples/marmot/.bin/examples && ./run-cluster.sh
Created /tmp/marmot-1.db
Created /tmp/marmot-2.db
Created /tmp/marmot-3.db
2:06PM INF Starting nats-server from=nats node_id=5973743519734446439
2:06PM INF Version: 2.10.4 from=nats node_id=5973743519734446439
2:06PM INF Git: [not set] from=nats node_id=5973743519734446439
2:06PM INF Cluster: e-marmot from=nats node_id=5973743519734446439
2:06PM INF Name: marmot-node-5973743519734446439 from=nats node_id=5973743519734446439
2:06PM INF Node: 16AHgXE3 from=nats node_id=5973743519734446439
2:06PM INF ID: NB4WCSC7ADZLDR4LNCZU6XGUSEFKZL5UFTEA3SBYL3WUMNTA2L3UMPLN from=nats node_id=5973743519734446439
2:06PM INF Starting JetStream from=nats node_id=5973743519734446439
2:06PM INF _ ___ _____ ___ _____ ___ ___ _ __ __ from=nats node_id=5973743519734446439
2:06PM INF _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ | from=nats node_id=5973743519734446439
2:06PM INF | || | _| | | \__ \ | | | / _| / _ \| |\/| | from=nats node_id=5973743519734446439
2:06PM INF \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_| from=nats node_id=5973743519734446439
2:06PM INF from=nats node_id=5973743519734446439
2:06PM INF https://docs.nats.io/jetstream from=nats node_id=5973743519734446439
2:06PM INF from=nats node_id=5973743519734446439
2:06PM INF ---------------- JETSTREAM ---------------- from=nats node_id=5973743519734446439
2:06PM INF Max Memory: 12.00 GB from=nats node_id=5973743519734446439
2:06PM INF Max Storage: 137.21 GB from=nats node_id=5973743519734446439
2:06PM INF Store Directory: "/var/folders/pj/n3sth0z55md7lydld97r8mmh0000gn/T/nats/marmot-node-5973743519734446439/jetstream" from=nats node_id=5973743519734446439
2:06PM INF ------------------------------------------- from=nats node_id=5973743519734446439
2:06PM INF Starting JetStream cluster from=nats node_id=5973743519734446439
2:06PM INF Creating JetStream metadata controller from=nats node_id=5973743519734446439
2:06PM INF JetStream cluster recovering state from=nats node_id=5973743519734446439
2:06PM INF Listening for client connections on 0.0.0.0:56401 from=nats node_id=5973743519734446439
2:06PM INF Server is ready from=nats node_id=5973743519734446439
2:06PM INF Cluster name is e-marmot from=nats node_id=5973743519734446439
2:06PM INF Listening for route connections on localhost:4221 from=nats node_id=5973743519734446439
2:06PM WRN JetStream has not established contact with a meta leader from=nats node_id=5973743519734446439
2:06PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4222: connect: connection refused from=nats node_id=5973743519734446439
2:06PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4222: connect: connection refused from=nats node_id=5973743519734446439
2:06PM WRN Waiting for routing to be established... from=nats node_id=5973743519734446439
2:06PM INF Starting nats-server from=nats node_id=5973743519734446439
2:06PM INF Version: 2.10.4 from=nats node_id=5973743519734446439
2:06PM INF Git: [not set] from=nats node_id=5973743519734446439
2:06PM INF Cluster: e-marmot from=nats node_id=5973743519734446439
2:06PM INF Name: marmot-node-5973743519734446439 from=nats node_id=5973743519734446439
2:06PM INF Node: 16AHgXE3 from=nats node_id=5973743519734446439
2:06PM INF ID: NCAAUCM4WY7EYU2E2F3YQGOKESPRDE2D65663G3PAQEOYKUG2SG6TVLU from=nats node_id=5973743519734446439
2:06PM INF Starting JetStream from=nats node_id=5973743519734446439
2:06PM INF _ ___ _____ ___ _____ ___ ___ _ __ __ from=nats node_id=5973743519734446439
2:06PM INF _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ | from=nats node_id=5973743519734446439
2:06PM INF | || | _| | | \__ \ | | | / _| / _ \| |\/| | from=nats node_id=5973743519734446439
2:06PM INF \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_| from=nats node_id=5973743519734446439
2:06PM INF from=nats node_id=5973743519734446439
2:06PM INF https://docs.nats.io/jetstream from=nats node_id=5973743519734446439
2:06PM INF from=nats node_id=5973743519734446439
2:06PM INF ---------------- JETSTREAM ---------------- from=nats node_id=5973743519734446439
2:06PM INF Max Memory: 12.00 GB from=nats node_id=5973743519734446439
2:06PM INF Max Storage: 137.21 GB from=nats node_id=5973743519734446439
2:06PM INF Store Directory: "/var/folders/pj/n3sth0z55md7lydld97r8mmh0000gn/T/nats/marmot-node-5973743519734446439/jetstream" from=nats node_id=5973743519734446439
2:06PM INF ------------------------------------------- from=nats node_id=5973743519734446439
2:06PM INF Starting JetStream cluster from=nats node_id=5973743519734446439
2:06PM INF Creating JetStream metadata controller from=nats node_id=5973743519734446439
2:06PM INF JetStream cluster recovering state from=nats node_id=5973743519734446439
2:06PM INF Listening for client connections on 0.0.0.0:56406 from=nats node_id=5973743519734446439
2:06PM INF Server is ready from=nats node_id=5973743519734446439
2:06PM INF Cluster name is e-marmot from=nats node_id=5973743519734446439
2:06PM INF Listening for route connections on localhost:4222 from=nats node_id=5973743519734446439
2:06PM WRN JetStream has not established contact with a meta leader from=nats node_id=5973743519734446439
2:06PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4221: connect: connection refused from=nats node_id=5973743519734446439
2:06PM INF 127.0.0.1:4221 - rid:7 - Route connection created from=nats node_id=5973743519734446439
2:06PM INF 127.0.0.1:56407 - rid:8 - Route connection created from=nats node_id=5973743519734446439
2:06PM ERR 127.0.0.1:56407 - rid:8 - Remote server has a duplicate name: "marmot-node-5973743519734446439" from=nats node_id=5973743519734446439
2:06PM INF 127.0.0.1:56407 - rid:8 - Router connection closed: Duplicate Server Name from=nats node_id=5973743519734446439
2:06PM INF 127.0.0.1:4221 - rid:7 - Router connection closed: Client Closed from=nats node_id=5973743519734446439
2:06PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4221: connect: connection refused from=nats node_id=5973743519734446439
2:06PM WRN Waiting for routing to be established... from=nats node_id=5973743519734446439
2:06PM ERR NATS client disconnected node_id=5973743519734446439
2:06PM ERR NATS client exiting node_id=5973743519734446439
^C./run-cluster.sh: line 32: kill: `': not a pid or valid job spec
Regarding "synced out of order" in your last point in the readme:
Marmot is eventually consistent. This simply means rows can get synced out of order, and SERIALIZABLE assumptions on transactions might not hold true anymore.
Does this mean nodes could become inconsistent and stay inconsistent?
Graphjin does almost exactly the same .
it’s golang
It’s postresql which I know is not appropriate for marmot but I think with a pg listener it can work with marmot if you’re interested.
Pocketbase supports files. Store the existence in SQLite and the actual on the File system.
We can use NATS object store with chunking to replicate massive files.
https://github.com/mprimi/nasefa for inspiration of the concept.
Pocketbase also does file uploads to local and S3.
I would like to brain storm the idea of using Marmot to make file uploads distributed without S3.
suggested Logic:
PB upload to local , just like it does to SQLite.
Fike system watcher sees the new file and then proceeds to replicate it to any other system instances , just like It does to SQLite.
advanced upload logic:
A user uploads from their device to their browser .
The browser then uploads in async chunks to the server.
I've been getting consistent crashes on the master server (in a master node + 1 replica setup). Both are on the latest version from the realeases page. Uptime is inconsistent. Sometimes its up for a day then crashes and sometimes it crashes within a few hours.
They're using basic configs so I might be missing an important thing that I do not know about.
The replica server has been running fine with no crashes.
Master (
config-main.toml
)
db_path="/home/tik/redis/videos-replica.v2.db"
seq_map_path="/tmp/videos-main.cbor"
node_id=1
publish=true
replicate=false
Replica (
config-replica.toml
)
db_path="/home/rep/tik/videos.v2.db"
seq_map_path="/tmp/videos-replica-1.cbor"
node_id=2
publish=false
replicate=true
Each instance is ran through the command line like
# Master
./marmot -config config-main.toml -cluster-addr 10.1.0.12:4223 -cluster-peers 'nats://10.1.0.1:14222/'
# Replica
./marmot -config config-replica.toml -cluster-addr 10.1.0.1:14222 -cluster-peers 'nats://10.1.0.12:4223/'
The database that it's using is 1.8GB with 4 tables of which only 1 (videos_clean
) is being updated frequently. The master database is a replica itself to keep it separate from the production one, a script pushes changes to it every minute.
Below is the output from the most recent crash.
because marmot is a CDC based pattern, it would probably be a pretty good match with benthos.
https://github.com/benthosdev/benthos
benthos already is used in CDC patterns with cockroach for example.
so the idea is that a change to the DB gets sent to benthos, and then benthos can react.
Benthos has a NATS source / sink btw.
https://www.benthos.dev/docs/components/inputs/nats_jetstream/
Here is a simple example: https://github.com/davidandradeduarte/benthos-nats-jetstream-output-loop-bug
Another cool example: https://github.com/amirhnajafiz/jetstream-mirroring/
The use cases people would use this for a huge IMHO.
Maybe you want to initiate certain logic or secondary data transform to update your business analytics db or whatever you want.
Following are required to be done:
Marmot can undo a locally committed write if it conflicts with a write that was already replicated to the other nodes.
An application might prefer to only respond - possibly only for some of it's writes - once the write has been successfully replicated. For that Marmot would need to provide some mechanism for the application to detect when the write in a particular transaction has been successfully replicated/rejected.
I imagine the application choosing to do that would have to be prepared to handle at least the following cases:
I setup polling due to inability to detect changes (running inside docker; probably can fix this).
With two marmot services, one leader and one follower. Polling (scan_changes) only starts after an insert/update is performed
Expected behavior: polling/scan_changes
should start on bootup
In reading over some of the Nats docs I ran across a note re the two headers Nats uses to "enforce optimistic concurrency" on a stream level or a per-subject level. nats-io/nats-server#3772
Not sure if you're currently making use of these but perhaps they might help ensure that operations at least always replay in the proper order since those headers ensure you can't write to the stream if you're out of sequence.
Just a thought.
Love what you're doing here!
Hi All,
Is it possible to have the HA setup with Master and slave only for marmot?
If yes, Do you have any conf file to configure it. I have tried configure it. But I am unable to setup master and slave setup
Thanks,
Kannan V
Hey @maxpert
Was thinking that a HTMX based Web GUi might be a nice match for Marmot ? All golang of course..
what sort of things would be useful to expose ?
seems pretty inactive but considering what it does (i can't find other projects like this where any given node can become master/slave with writes)
corrosion is interesting but doesn't support joins....so its a tough ask
just wish there was more activity here but maybe its already quite mature?
trying to learn more about this comment on HN by maxpert:
"- Easy to start, yet hard to master - You can get up and running pretty quickly, but make no mistake this tool is not for rookie who doesn't understand how incremental primary keys are bad, and how to they can keep things conflict free."
can we get some more guide/tips on what not to do and what to do for beginners?
Currently the triggers are only attached during the start phase of marmot. Individual schema changes (new tables / columns) are not reflected at runtime and require either a manual way to apply them on all DBs in the cluster + restarting each marmot node to update triggers and changelog tables.
Are there some recommendations how to approach it in a straightforward and maintainable way?
Best,
Roman
Hey @maxpert
Would basic CRDT help with reconciliation ?
https://github.com/superfly/corrosion uses https://github.com/vlcn-io/cr-sqlite
Marmot could do this I think too I think. I need to see what cr-sqlite is doing
I am using https://github.com/ncruces/go-sqlite3 and looking into if I can use Marmot with it.
Then I realised that a CRDT aspect would help with the Master Master reconciliation aspects of Marmot
In the README is says 138.3 writes/sec for the latest.
I really like Marmot but am the Perf in the README is alarming..
Can you qualify why it's so insanely slow maybe.
Was it doing cross DC replication or something else that slows it down so much ?
Doesn't seem to work with ipv6. That would be nice to have, and Go networking has pretty good support for it, in my experience. You have a lot of dependencies though, so hopefully there are no blockers.
Thanks for the useful project.
I've had a good look at what you've done with this project and I like it a lot. I've been thinking of ways to add CDC to SQLite3 and this abstraction is really nice and simple.
I've managed to "hack" in a CDC event pusher to a http endpoint as an example, but I don't know a lot about replication protocol and found that in multi nodes the CDC events are publish for N number of nodes.
Wondering if you are open to implementing CDC replicating functionality similar to Debezium. Namely the event dispatching to different sinks.
Right now RestoreFrom copies over database file but does not modify the -wal
and -shm
to ensure same reboot state. This will cause corruption of DB and wal files need to be deleted after restore snapshot for a cluster recovering from bootstate.
I am sure I am just missing something, but I cannot figure out how to user marmot as my SQLIte DB. Example code with a connection string, creating a table, insert, ... would be great!
Earlier versions of marmot didn't require JetStream. As far as I can tell, I can still connect to a regular nats server - I can see the connections - but it seems that sync isn't working.
Another question is:
If I understand this properly, writes from one database trigger inserts to a dedicated table which will then be read by marmot
and sent to nats. The question is - how often does marmot read changes, and is this configurable?
Hello, I searched your issues for a clue.
I couldn't find a reference to the working versions of sqlite.
If it helps I do know since 3.38 and 3.44 at some point they got stricter with certain things.
For instance here is a chat I had about ".dump" and single quotes and double quotes.
https://sqlite.org/forum/forumpost/2eeab88391e744e1
Example failed
Parse error: unsafe use of virtual table "pragma_function_list"
sqlite3 /tmp/marmot-1.db
SQLite version 3.44.2 2023-11-24 11:41:44
Enter ".help" for usage hints.
sqlite> INSERT INTO Books (title, author, publication_year) VALUES ('Pride and Prejudice', 'Jane Austen', 1813);
Parse error: unsafe use of virtual table "pragma_function_list"
./examples/run-cluster.sh
Created /tmp/marmot-1.db
Created /tmp/marmot-2.db
Created /tmp/marmot-3.db
2:40PM INF Starting nats-server from=nats node_id=1
2:40PM INF Version: 2.10.4 from=nats node_id=1
2:40PM INF Git: [not set] from=nats node_id=1
2:40PM INF Cluster: e-marmot from=nats node_id=1
2:40PM INF Name: marmot-node-1 from=nats node_id=1
2:40PM INF Node: OWL7P9aV from=nats node_id=1
2:40PM INF ID: NA6IGSANNMR6TLA6GTW6NPEEQR5K5HFCTQWVLB6WGLHCXP24ZORPP77X from=nats node_id=1
2:40PM INF Starting JetStream from=nats node_id=1
2:40PM INF _ ___ _____ ___ _____ ___ ___ _ __ __ from=nats node_id=1
2:40PM INF _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ | from=nats node_id=1
2:40PM INF | || | _| | | \__ \ | | | / _| / _ \| |\/| | from=nats node_id=1
2:40PM INF \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_| from=nats node_id=1
2:40PM INF from=nats node_id=1
2:40PM INF https://docs.nats.io/jetstream from=nats node_id=1
2:40PM INF from=nats node_id=1
2:40PM INF ---------------- JETSTREAM ---------------- from=nats node_id=1
2:40PM INF Max Memory: 24.00 GB from=nats node_id=1
2:40PM INF Max Storage: 133.71 GB from=nats node_id=1
2:40PM INF Store Directory: "/tmp/nats/marmot-node-1/jetstream" from=nats node_id=1
2:40PM INF ------------------------------------------- from=nats node_id=1
2:40PM INF Starting JetStream cluster from=nats node_id=1
2:40PM INF Creating JetStream metadata controller from=nats node_id=1
2:40PM INF JetStream cluster bootstrapping from=nats node_id=1
2:40PM INF Listening for client connections on 0.0.0.0:56533 from=nats node_id=1
2:40PM INF Server is ready from=nats node_id=1
2:40PM INF Cluster name is e-marmot from=nats node_id=1
2:40PM INF Listening for route connections on localhost:4221 from=nats node_id=1
2:40PM WRN JetStream has not established contact with a meta leader from=nats node_id=1
2:40PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4222: connect: connection refused from=nats node_id=1
2:40PM ERR Error trying to connect to route (attempt 1): dial tcp 127.0.0.1:4222: connect: connection refused from=nats node_id=1
2:40PM WRN Waiting for routing to be established... from=nats node_id=1
2:40PM INF Starting nats-server from=nats node_id=2
2:40PM INF Version: 2.10.4 from=nats node_id=2
2:40PM INF Git: [not set] from=nats node_id=2
2:40PM INF Cluster: e-marmot from=nats node_id=2
2:40PM INF Name: marmot-node-2 from=nats node_id=2
2:40PM INF Node: aThtkTV0 from=nats node_id=2
2:40PM INF ID: NAH3JJIZ7WT4JVQN7TNACQR6E7BZG2SX675GWXNSYOUFE7ZGCCTDMH6F from=nats node_id=2
2:40PM INF Starting JetStream from=nats node_id=2
2:40PM INF _ ___ _____ ___ _____ ___ ___ _ __ __ from=nats node_id=2
2:40PM INF _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ | from=nats node_id=2
2:40PM INF | || | _| | | \__ \ | | | / _| / _ \| |\/| | from=nats node_id=2
2:40PM INF \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_| from=nats node_id=2
2:40PM INF from=nats node_id=2
2:40PM INF https://docs.nats.io/jetstream from=nats node_id=2
2:40PM INF from=nats node_id=2
2:40PM INF ---------------- JETSTREAM ---------------- from=nats node_id=2
2:40PM INF Max Memory: 24.00 GB from=nats node_id=2
2:40PM INF Max Storage: 133.71 GB from=nats node_id=2
2:40PM INF Store Directory: "/tmp/nats/marmot-node-2/jetstream" from=nats node_id=2
2:40PM INF ------------------------------------------- from=nats node_id=2
2:40PM INF Starting JetStream cluster from=nats node_id=2
2:40PM INF Creating JetStream metadata controller from=nats node_id=2
2:40PM INF JetStream cluster bootstrapping from=nats node_id=2
2:40PM INF Listening for client connections on 0.0.0.0:56538 from=nats node_id=2
2:40PM INF Server is ready from=nats node_id=2
2:40PM INF Cluster name is e-marmot from=nats node_id=2
2:40PM INF Listening for route connections on localhost:4222 from=nats node_id=2
2:40PM WRN JetStream has not established contact with a meta leader from=nats node_id=2
2:40PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4221: connect: connection refused from=nats node_id=2
2:40PM INF 127.0.0.1:4221 - rid:7 - Route connection created from=nats node_id=2
2:40PM INF 127.0.0.1:56540 - rid:8 - Route connection created from=nats node_id=1
2:40PM WRN Waiting for routing to be established... from=nats node_id=2
2:40PM ERR NATS client disconnected node_id=1
2:40PM ERR NATS client exiting node_id=1
2:40PM INF Starting nats-server from=nats node_id=3
2:40PM INF Version: 2.10.4 from=nats node_id=3
2:40PM INF Git: [not set] from=nats node_id=3
2:40PM INF Cluster: e-marmot from=nats node_id=3
2:40PM INF Name: marmot-node-3 from=nats node_id=3
2:40PM INF Node: 1p7dAfNG from=nats node_id=3
2:40PM INF ID: NBOHQIIF2S4VEC4DHGXB3YPVL3BYBK6DEI3YW4ISXSZEAZIOJ4S3ZBJJ from=nats node_id=3
2:40PM INF Starting JetStream from=nats node_id=3
2:40PM INF _ ___ _____ ___ _____ ___ ___ _ __ __ from=nats node_id=3
2:40PM INF _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ | from=nats node_id=3
2:40PM INF | || | _| | | \__ \ | | | / _| / _ \| |\/| | from=nats node_id=3
2:40PM INF \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_| from=nats node_id=3
2:40PM INF from=nats node_id=3
2:40PM INF https://docs.nats.io/jetstream from=nats node_id=3
2:40PM INF from=nats node_id=3
2:40PM INF ---------------- JETSTREAM ---------------- from=nats node_id=3
2:40PM INF Max Memory: 24.00 GB from=nats node_id=3
2:40PM INF Max Storage: 133.71 GB from=nats node_id=3
2:40PM INF Store Directory: "/tmp/nats/marmot-node-3/jetstream" from=nats node_id=3
2:40PM INF ------------------------------------------- from=nats node_id=3
2:40PM INF Starting JetStream cluster from=nats node_id=3
2:40PM INF Creating JetStream metadata controller from=nats node_id=3
2:40PM INF JetStream cluster bootstrapping from=nats node_id=3
2:40PM INF Listening for client connections on 0.0.0.0:56546 from=nats node_id=3
2:40PM INF Server is ready from=nats node_id=3
2:40PM INF Cluster name is e-marmot from=nats node_id=3
2:40PM INF Listening for route connections on localhost:4223 from=nats node_id=3
2:40PM WRN JetStream has not established contact with a meta leader from=nats node_id=3
2:40PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4221: connect: connection refused from=nats node_id=3
2:40PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4222: connect: connection refused from=nats node_id=3
2:40PM INF 127.0.0.1:56549 - rid:9 - Route connection created from=nats node_id=1
2:40PM INF 127.0.0.1:4221 - rid:7 - Route connection created from=nats node_id=3
2:40PM INF 127.0.0.1:56550 - rid:9 - Route connection created from=nats node_id=2
2:40PM INF 127.0.0.1:4222 - rid:8 - Route connection created from=nats node_id=3
2:40PM ERR NATS client disconnected node_id=2
2:40PM ERR NATS client exiting node_id=2
2:40PM INF 127.0.0.1:4222 - rid:10 - Route connection created from=nats node_id=3
2:40PM INF 127.0.0.1:56551 - rid:10 - Route connection created from=nats node_id=2
2:40PM WRN Waiting for routing to be established... from=nats node_id=3
2:40PM INF 127.0.0.1:4222 - rid:11 - Route connection created from=nats node_id=3
2:40PM INF 127.0.0.1:56552 - rid:11 - Route connection created from=nats node_id=2
2:40PM INF 127.0.0.1:4222 - rid:10 - Route connection created from=nats node_id=1
2:40PM INF 127.0.0.1:56553 - rid:12 - Route connection created from=nats node_id=2
2:40PM INF 127.0.0.1:4222 - rid:11 - Route connection created from=nats node_id=1
2:40PM INF 127.0.0.1:56554 - rid:13 - Route connection created from=nats node_id=2
2:40PM INF 127.0.0.1:4222 - rid:10 - Router connection closed: Duplicate Route from=nats node_id=1
2:40PM INF 127.0.0.1:56553 - rid:12 - Router connection closed: Duplicate Route from=nats node_id=2
2:40PM WRN Waiting for routing to be established... from=nats node_id=1
2:40PM INF Self is new JetStream cluster metadata leader from=nats node_id=2
2:40PM INF JetStream cluster new metadata leader: marmot-node-2/e-marmot from=nats node_id=1
2:40PM INF Streaming ready... node_id=1
2:40PM INF Streaming ready... node_id=1
2:40PM INF JetStream cluster new stream leader for '$G > marmot-changes-c-1' from=nats node_id=1
2:40PM INF JetStream cluster new stream leader for '$G > KV_e-marmot' from=nats node_id=1
2:40PM INF Listing tables to watch... node_id=1
2:40PM INF Starting change data capture pipeline... node_id=1
2:40PM INF Creating global change log table node_id=1
2:40PM INF Creating trigger for Books node_id=1
2:40PM INF JetStream cluster new consumer leader for '$G > marmot-changes-c-1 > AF430TAJ' from=nats node_id=1
2:40PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4222: connect: connection refused from=nats node_id=1
2:40PM INF 127.0.0.1:4221 - rid:14 - Route connection created from=nats node_id=2
2:40PM INF 127.0.0.1:56556 - rid:22 - Route connection created from=nats node_id=1
2:40PM ERR Error trying to connect to route (attempt 1): dial tcp [::1]:4221: connect: connection refused from=nats node_id=2
2:40PM INF Streaming ready... node_id=2
2:40PM INF Streaming ready... node_id=2
2:40PM INF Listing tables to watch... node_id=2
2:40PM INF Starting change data capture pipeline... node_id=2
2:40PM INF Creating global change log table node_id=2
2:40PM INF Creating trigger for Books node_id=2
2:40PM ERR NATS client disconnected node_id=3
2:40PM ERR NATS client exiting node_id=3
2:40PM INF JetStream cluster new consumer leader for '$G > marmot-changes-c-1 > 3GqGDqy2' from=nats node_id=1
2:40PM INF 127.0.0.1:4222 - rid:25 - Route connection created from=nats node_id=1
2:40PM INF 127.0.0.1:56560 - rid:17 - Route connection created from=nats node_id=2
2:40PM INF 127.0.0.1:4221 - rid:18 - Route connection created from=nats node_id=2
2:40PM INF 127.0.0.1:56563 - rid:26 - Route connection created from=nats node_id=1
2:40PM INF 127.0.0.1:4221 - rid:18 - Router connection closed: Duplicate Route from=nats node_id=2
2:40PM INF 127.0.0.1:56563 - rid:26 - Router connection closed: Client Closed from=nats node_id=1
2:40PM INF 127.0.0.1:4221 - rid:14 - Route connection created from=nats node_id=3
2:40PM INF 127.0.0.1:4222 - rid:13 - Route connection created from=nats node_id=3
2:40PM INF 127.0.0.1:56564 - rid:27 - Route connection created from=nats node_id=1
2:40PM INF 127.0.0.1:56565 - rid:19 - Route connection created from=nats node_id=2
2:40PM ERR NATS client disconnected node_id=3
2:40PM ERR NATS client exiting node_id=3
2:40PM INF 127.0.0.1:4221 - rid:15 - Route connection created from=nats node_id=3
2:40PM INF 127.0.0.1:56566 - rid:28 - Route connection created from=nats node_id=1
2:40PM INF 127.0.0.1:4221 - rid:16 - Route connection created from=nats node_id=3
2:40PM INF 127.0.0.1:56567 - rid:29 - Route connection created from=nats node_id=1
2:40PM INF Streaming ready... node_id=3
2:40PM INF Streaming ready... node_id=3
2:40PM INF Listing tables to watch... node_id=3
2:40PM INF Starting change data capture pipeline... node_id=3
2:40PM INF Creating global change log table node_id=3
2:40PM INF Creating trigger for Books node_id=3
2:40PM INF JetStream cluster new consumer leader for '$G > marmot-changes-c-1 > m0DR7kkv' from=nats node_id=1
Hi, I am trying to set up an SQLite cluster using marmot, while writing to node1 with the rate of 500 records in 5 seconds, continuously
I am getting a database-locked error, am I missing something in the configuration, I have configured nodes with default configurations, that are mentioned in the repo.
If I increase the rate from 1000 per second then it's crashing also.
Insert Users Create Users Err &{0xc0000f2d00 INSERT INTO ip_usernames (endpoint, username, domain_name, last_updated_timestamp) VALUES (?, ?, ?, ?) <nil> {{0 0} 0 0 0 0} 0xc0004b4080 <nil> <nil> {0 0} true [] 0} Error is database is locked (5) (SQLITE_BUSY)
even after ingesting at a low rate, the database is getting locked.
Are you open for support for TLS authentication in NATS being added?
This will require a SIGHUP (or any way of triggering a reload) handler that can re-read the certificate and key from the provided path.
Also it seems to be impossible to specify a custom CA certificate for the connection.
This could look something like this in the config:
[nats]
urls=["nats://localhost:4222"]
ca_file="/tmp/nats/ca.crt"
cert_file="/tmp/nats/client.crt"
key_file="/tmp/nats/client.key"
I tried the binaries at AMD64 0.7.4 and 0.7.5
4:03PM DBG Opening database path=???????????????????????????????????????/scavenger-beta.db
4:03PM DBG Forcing WAL checkpoint
4:03PM DBG Stream ready... leader=lily_nats name=eros-changes-c-1 replicas=1 shard=1
4:03PM INF Listing tables to watch...
4:03PM INF Starting change data capture pipeline...
4:03PM DBG Listening stream shard=1
4:03PM DBG duration=0.394929 name=scan_changes
4:03PM ERR Unable to scan global changes error="no such table: __marmot___change_log_global"
The db isn't open and no other errs are posted.
Since this is a bit more involved in sqlite than most other projects, this might very well not be an option, but have you considered/tested using https://pkg.go.dev/modernc.org/sqlite instead of the standard github.com/mattn/go-sqlite3
?
It is reportedly a bit slower since it's transpiled from c to non-optimized go but it has worked for my use-cases well. And building is a breeze :) (I actually could not build marmot without using a docker image on NixOS).
Are there any other cgo dependencies that I'm missing?
As a test, I have two machines with public IPs on the internet, both running pocketbase
, nats-server
, and marmot
. If I create my admin user and collections on the first machine, how can I get the second machine to migrate over that initial set of data and then start replicating things in both directions after the initial snapshot migration?
This is a courtesy note to say that this project is now tracked at https://github.com/benallfree/awesome-pocketbase. If you are able to add any PocketBase-specific instructions somewhere, that would be great too!
Could a basic roadmap be established ?
Would help with seeing where myself and others can dovetail on to help
Just like rest of good open source projects we need a nice clean well designed static website.
It would be useful to support configuration options for NATS connection retries in the event that a marmot follower is initialized slightly before its leader. In our use case, two hosts are provisioned simultaneously and the follower host occasionally lags the leader by as much as a few minutes.
Marmot follower error when attempting to connect prior to leader initialization:
Oct 17 00:40:04 two-node-two marmot[1693]: 12:40AM DBG Opening database node_id=2197861447266130575 path=/var/lib/rancher/k3s/server/db/state.db
Oct 17 00:40:04 two-node-two marmot[1693]: 12:40AM DBG Forcing WAL checkpoint node_id=2197861447266130575
Oct 17 00:40:07 two-node-two marmot[1693]: 12:40AM PNC Unable to initialize snapshot storage error="dial tcp X.X.X.X:4222: connect: no route to host" node_id=2197861447266130575
Oct 17 00:40:07 two-node-two marmot[1693]: panic: Unable to initialize snapshot storage
Oct 17 00:40:07 two-node-two marmot[1693]: goroutine 1 [running]:
Oct 17 00:40:07 two-node-two marmot[1693]: github.com/rs/zerolog/log.Panic.(*Logger).Panic.func1({0x1158d4c?, 0x0?})
Oct 17 00:40:07 two-node-two marmot[1693]: /home/runner/go/pkg/mod/github.com/rs/[email protected]/log.go:376 +0x27
Oct 17 00:40:07 two-node-two marmot[1693]: github.com/rs/zerolog.(*Event).msg(0xc000282300, {0x1158d4c, 0x25})
Oct 17 00:40:07 two-node-two marmot[1693]: /home/runner/go/pkg/mod/github.com/rs/[email protected]/event.go:156 +0x2c2
Oct 17 00:40:07 two-node-two marmot[1693]: github.com/rs/zerolog.(*Event).Msg(...)
Oct 17 00:40:07 two-node-two marmot[1693]: /home/runner/go/pkg/mod/github.com/rs/[email protected]/event.go:108
Oct 17 00:40:07 two-node-two marmot[1693]: main.main()
Oct 17 00:40:07 two-node-two marmot[1693]: /home/runner/work/marmot/marmot/marmot.go:66 +0x70a
Oct 17 00:40:07 two-node-two systemd[1]: marmot.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 17 00:40:07 two-node-two systemd[1]: marmot.service: Failed with result 'exit-code'.
Oct 17 00:40:07 two-node-two systemd[1]: marmot.service: Scheduled restart job, restart counter is at 2.
Oct 17 00:40:07 two-node-two systemd[1]: Stopped Marmot synchronizes the k8s state in SQLite between nodes in a two node topology.
Oct 17 00:40:07 two-node-two systemd[1]: Started Marmot synchronizes the k8s state in SQLite between nodes in a two node topology.
Oct 17 00:40:07 two-node-two marmot[1699]: 12:40AM DBG Opening database node_id=2197861447266130575 path=/var/lib/rancher/k3s/server/db/state.db
Oct 17 00:40:07 two-node-two marmot[1699]: 12:40AM DBG Forcing WAL checkpoint node_id=2197861447266130575
Oct 17 00:40:10 two-node-two marmot[1699]: 12:40AM PNC Unable to initialize snapshot storage error="dial tcp X.X.X.X:4222: connect: no route to host" node_id=2197861447266130575
Thank you for creating Marmot. The built-in NATS functionality, in particular, has been a significant factor in adopting Marmot. However, it seems challenging to discern whether the settings are for built-in NATS, external NATS, or the configurations required by Marmot to use NATS. It would be beneficial if the names of the settings were more explicit.
First of all, thanks for the cool tool.
Do you consider build the executable file for mac OSX (amd64 and arm64)?
Would like a clearer explanation of the strategy to prevent split brain.
I use marmot alongside a python application to replicate my data to secondary/tertiary servers.
Within my database, I've got several sqlite datetime fields, such as: received DATETIME NOT NULL
. My application stores dates in this field using ISO 8601 format: 2023‐08‐15T17:01:43+00:00
. When marmot copies this field and sends it to a replica, the field is delivered as a unix epoch timestamp: 1692122503
.
This wouldn't be an issue as python datetime can correctly parse this into the correct datetime, except that my sqlite queries which implement conditionals on this field do not work against epoch timestamps.
For example, this query does not return any results against fields who's timestamps have been transformed:
query = "SELECT * FROM table WHERE received BETWEEN :from_date AND :to_date"
params= {"from_date": "2000-01-01 12:00:00+00:00", "to_date": "2030-12-31 23:59:59+00:00"}
I believe marmot should never transform customer data, and this is likely an oversight/assumption.
I'm not a Go dev, but it looks like this might be happening here in the sqlite3 driver you're using: https://github.com/mattn/go-sqlite3/blob/master/sqlite3.go#L2231-L2258. I understand that if the driver behaves this way, its not immediately in your control. But I'm raising this issue to you first since you probably don't want to damage customer data anymore than I do. I would both consider alternative drivers and also add additional unit tests which test that data is never mutated.
https://github.com/superfly/litefs Does it.
I would have thought that it would be useful to solve the linearisation design issues so that we don’t end up with partially incomplete data being replicated due to a transaction rollback on the initiator db
Right now we store snapshots inside NATS object store. While this might satisfy use-cases for some of users, we should support all S3 compatible storage layers for doing a save/restore of snapshots. Primary testing targets:
It looks like a shared state machine is used for multiple Raft groups. This doesn't seem correct to me?
My understanding is that the Multi-Raft protocol does not guarantee commit order across different Raft groups. The user has to implement a custom distributed transaction layer (e.g. 2PC, or with some addition like Percolator or TrueTime) to establish partial ordering between transactions. TiKV uses Percolator, for example.
Trying to setup a example with a sufficiently complex DB: https://github.com/disarticulate/marmot/tree/master/example/single-leader to get a better idea of current NATS
configuration.
Ran into the topic error. Not sure what the error is, but looks like pk
refers to IsPrimaryKey bool
db:"pk"in the query
query := "SELECT name, type, notnull
, dflt_value, pk FROM pragma_table_info(?)"` from here:
https://github.com/maxpert/marmot/blob/master/db/sqlite.go
if !hasPrimaryKey {
tableInfo = append(tableInfo, &ColumnInfo{
Name: "rowid",
IsPrimaryKey: true,
Type: "INT",
NotNull: true,
DefaultValue: nil,
})
}
this stackoverflow: https://stackoverflow.com/questions/10472103/sqlite-query-to-find-primary-keys indicates the references pk
column is not a bool, but a count
of the number of keys in the primary key.
I assume the fix is to evaluate (unfortunately) IsPrimary as an array rather than bool.
Including one by google
Maybe it'd be a good idea to change the name
Cool project though
Use overmind to make it easy to run 3 instances to show it works and to bench it locally .
https://github.com/DarthSim/overmind
snapshot into NATS store so we don’t need S3 for demo. Less hassle and still HA. Add minio later if needed.
demo can use templ to make it easy to have a gui that updates reactively:
https://github.com/joerdav/go-htmx-examples
later this web gui approach can be used to provide a reactive web gui for managing marmot. As changes occur the web gui updates …
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.