Coder Social home page Coder Social logo

nanopack / yoke Goto Github PK

View Code? Open in Web Editor NEW
1.3K 77.0 65.0 13.06 MB

Postgres high-availability cluster with auto-failover and automated cluster recovery.

License: Other

Go 51.99% Shell 0.16% Ruby 5.52% HTML 22.65% JavaScript 1.87% CSS 17.81%
postgres auto-failover redundancy golang nanobox availability-cluster cluster nanopack

yoke's Introduction

yoke logo Build Status

Yoke is a Postgres redundancy/auto-failover solution that provides a high-availability PostgreSQL cluster that's simple to manage.

Requirements

Yoke has the following requirements/dependencies to run:

  • A 3-server cluster consisting of a 'primary', 'secondary', and 'monitor' node
  • 'primary' & 'secondary' nodes need ssh connections between each other (w/o passwords)
  • 'primary' & 'secondary' nodes need rsync (or some alternative sync_command) installed
  • 'primary' & 'secondary' nodes should have postgres installed under a postgres user, and in the path. Yoke tries calling 'postgres' and 'pg_ctl'
  • 'primary' & 'secondary' nodes run postgres as a child process so it should not be started independently

Each node in the cluster requires its own config.ini file with the following options (provided values are defaults):

[config]
# the IP which this node will broadcast to other nodes
advertise_ip=
# the port which this node will broadcast to other nodes
advertise_port=4400
# the directory where postgresql was installed
data_dir=/data
# delay before node decides what to do with postgresql instance
decision_timeout=30
# log verbosity (trace, debug, info, warn error, fatal)
log_level=warn
# REQUIRED - the IP:port combination of all nodes that are to be in the cluster (e.g. 'role=m.y.i.p:4400')
primary=
secondary=
monitor=
# SmartOS REQUIRED - either 'primary', 'secondary', or 'monitor' (the cluster needs exactly one of each)
role=
# the postgresql port
pg_port=5432
# the directory where node status information is stored
status_dir=./status
# the command you would like to use to sync the data from this node to the other when this node is master
sync_command=rsync -ae "ssh -o StrictHostKeyChecking=no" --delete {{local_dir}} {{slave_ip}}:{{slave_dir}}

[vip]
# Virtual Ip you would like to use
ip=
# Command to use when adding the vip. This will be called as {{add_command}} {{vip}}
add_command=
# Command to use when removing the vip. This will be called as {{remove_command}} {{vip}}
remove_command=

[role_change]
# When this nodes role changes we will call the command with the new role as its arguement '{{command}} {{(master|slave|single}))'
command=

Startup

Once all configurations are in place, start yoke by running:

./yoke ./primary.ini

Note: The ini file can be named anything and reside anywhere. All Yoke needs is the /path/to/config.ini on startup.

Yoke CLI - yokeadm

Yoke comes with its own CLI, yokeadm, that allows for limited introspection into the cluster.

Building the CLI:

cd ./yokeadm
go build
./yokeadm
Usage:
yokeadm (<COMMAND>:<ACTION> OR <ALIAS>) [GLOBAL FLAG] <POSITIONAL> [SUB FLAGS]
Available Commands:
  • list : Returns status information for all nodes in the cluster
  • demote : Advises a node to demote

Documentation

Complete documentation is available on godoc.

Licence

Mozilla Public License Version 2.0

open source

yoke's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

yoke's Issues

net.Dial always nil

I'm not quite sure what I am doing wrong. I've set up three nodes, one per server, and start up yoke on each. However, the result is always the same:

checking if postgres is up <nil> dial tcp 127.0.0.1:5432: getsockopt: connection refused 5432

As a result, each node is always has the message 'waiting for the cluster to be ready.'

Any idea what I am doing wrong?

Thanks!

archive_command is always 'exit 0' ?

There doesn't appear to be a way to set the archive_command. I need the master node to push WAL files to an AWS S3 bucket in order for streaming replication to work with my current (and preferred) setup. It seems like the default archive command for the postgresql.conf is injected as part of a big string in pgconfig.go. Am I missing something important here in the configuration? If not, is there a way to add the desired archive_command as part of the yoke.ini configuration?

Single binary design

What about combine functionality of yoke and yokeadm to a single binary how it made in projects like consul and docker?

For example:

yoke server -config=/path/to/config.ini - starts a yoke server
yoke cluster list - give me a cluster list

This modification should simplify deployment and monitoring.

log.Fatal instead of panics

There are so many panic messages when i try to play with yoke.

For example:

bash-4.2$ yoke /etc/yoke/yoke.ini
2015-11-18 16:32:13 INFO  Creating scribble database at 'status'...
2015-11-18 16:32:13 INFO  database has already been created... skipping.
2015-11-18 16:32:13 INFO  going to start
2015-11-18 16:32:13 INFO  starting
2015-11-18 16:32:13 INFO  [action] starting db
panic: exec: "postgres": executable file not found in $PATH

goroutine 1 [running]:
main.main()
    /root/yoke/main.go:92 +0xfef

goroutine 5 [syscall]:
os/signal.loop()
    /usr/lib/golang/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/lib/golang/src/os/signal/signal_unix.go:27 +0x35

goroutine 7 [runnable]:
net/rpc.(*Server).Accept(0xc2080d8c80, 0x7f1809d93628, 0xc2080380e0)
    /usr/lib/golang/src/net/rpc/server.go:615
created by github.com/nanopack/yoke/state.(*state).ExposeRPCEndpoint
    /root/go/src/github.com/nanopack/yoke/state/rpc.go:59 +0x242

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1

goroutine 8 [runnable]:
github.com/nanopack/yoke/monitor.func·001()
    /root/go/src/github.com/nanopack/yoke/monitor/action.go:58
created by github.com/nanopack/yoke/monitor.NewPrefix
    /root/go/src/github.com/nanopack/yoke/monitor/action.go:62 +0x27a

goroutine 9 [runnable]:
github.com/nanopack/yoke/monitor.func·001()
    /root/go/src/github.com/nanopack/yoke/monitor/action.go:58
created by github.com/nanopack/yoke/monitor.NewPrefix
    /root/go/src/github.com/nanopack/yoke/monitor/action.go:62 +0x27a

Maybe use a log.Fatal instead of panics to show only useful information? Obviously in this case program should only to show: exec: "postgres": executable file not found in $PATH without any stack traces and exited with error status code.

Specify pg_bin_path in config file

Instead of running PostgreSQL binaries (e.g. pg_ctl, postgres) from $PATH.

Also it will be good to verify PostgreSQL binaries in pg_bin_path when the program parses config file.

[ Explain] queries are load balanced?

I have gone through the docs. However, it is not clear whether read queries are load balanced on both master and slave. Can you please clarify? I am wondering whether I can use yoke on AWS with elb , instead of virtual ip?

Can not start yoke after shutdown

On the second and next runs yoke raises this error message:

bash-4.2$ yoke /etc/yoke/yoke.ini
Unable to create database, 'status' already exists. Please specify a different location.

That's because we're call scribble.New on every program run. And scribble.New can't load already existing database.

It looks like bug in https://github.com/nanobox-io/golang-scribble ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.