Internal RPC error with basic fdbcli connection about foundationdb

[dkoston]

@alexmiller-apple thanks for the reply. For those following the thread, I ended up doing the following:

• pass in the connection string as an environment variable to my app, i.e. fdb:fdb@docker_container_name:4500
• have the app do dns resolution to turn hostname into IP address
• have the app then overwrite the fdb.cluster file with the resolved IP address in the connection string
• then call fdb.MustOpenDefault() or whatever your bindings/client library calls for

This allows the app to re-resolve the hostname each time it restarts. If the connection drops, you can call the same function during reconnect to make sure the IP is up to date.

Having hostname resolution in fdbcli and the bindings themselves would be nice but it's not much code to write to work around the issue for now.

[dkoston]

@atombender I'm getting the same. Did you sort this out?

To add to this report, this error isn't specific to fdbcli, it also happens with the go client:

package main

import (
	"github.com/apple/foundationdb/bindings/go/src/fdb"
	"log"
	"fmt"
)

func main() {
	// Different API versions may expose different runtime behaviors.
	fdb.MustAPIVersion(510)

	// Open the default database from the system cluster
	db := fdb.MustOpenDefault()

	// Database reads and writes happen inside transactions
	ret, e := db.Transact(func(tr fdb.Transaction) (interface{}, error) {
	tr.Set(fdb.Key("hello"), []byte("world"))
	return tr.Get(fdb.Key("foo")).MustGet(), nil
	// db.Transact automatically commits (and if necessary,
	// retries) the transaction
	})
	if e != nil {
	log.Fatalf("Unable to perform FDB transaction (%v)", e)
	}

	fmt.Printf("hello is now world, foo was: %s\n", string(ret.([]byte)))
}

Connecting on the docker container to fdbcli works fine:

[~/testing/foundationdb-kubernetes](master [!])$ de docker_fdb_1
root@22f18ed2d173:/var/lib/foundationdb# fdbcli
Using cluster file `fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb> writemode on
fdb> set "foo" "bar"
Committed (1052744750)
fdb> get "foo"
`foo' is `bar'

[dkoston]

This may be based on port mapping in docker. I:

• rebuilt my cluster into a single node and mapped "4500:4500"
• updated fbc.cluster to point to fdb:[email protected]:4500 rather than a mapped port and was able to connect.

Perhaps this line is checking to see if the port numbers match and failing out when the mapped port on the host doesn't match the source port on the container:

https://github.com/apple/foundationdb/blob/master/fdbrpc/FlowTransport.actor.cpp#L655

There's also another issue which is that fdbcli seems to try and use the config from the foundationdb server to connect to the cluster controller:

$ fdbcli -C fdb.cluster
Using cluster file `fdb.cluster'.

The database is unavailable; type `status' for more information.

Welcome to the fdbcli. For help, type `help'.
fdb> status

Using cluster file `fdb.cluster'.

Unable to communicate with the cluster controller at 172.18.0.2:4500 to get
status.

To fix this issue, I tried binding to 127.0.0.1:4500 but that is still having issues:

Could not communicate with a quorum of coordination servers:
  127.0.0.1:4500  (unreachable)

[dkoston]

Ah, fixed it on a single node by binding to 0.0.0.0:4500 and public address 127.0.0.1:4500

Still trying to sort out multiple nodes as that setup doesn't work

[dkoston]

Here's a docker-compose setup that handles the heavy lifting of running a cluster for you:

https://github.com/dkoston/foundationdb-kubernetes/tree/master/docker

Not sure what to advertise on the public_address to get a cluster working and to be able to connect outside docker.

Changing run.sh to use --public_address auto:4500 allows the cluster to spin up successfully but the host machine cannot connect. Using --public_address 127.0.0.1:4500 stops the cluster from spinning up as the machines can't talk to each other over the docker network.

I was hoping to proxy_pass with nginx on port 4500 but that results in Unable to locate a cluster controller within 2 seconds. Check that there are server processes running.

[atombender]

Thanks, @dkoston! I was never able to work around the error I got, since I'm deploying this on Kubernetes. I will look at your stuff and see if I can use it.

[alexmiller-apple]

Sorry, to finally leave a comment here:

FDB is probably a bit overly aggressive in asserting things about its network configuration, in a way that's meant to be helpful, but particularly in a container/kubernetes world might not be helpful. In particular, the NAT-ing behavior that docker does when re-exposing containerized services to the host results in two very unexpected effects: what the process thinks is its IP is not reachable externally, and what the process thinks its port is doesn't match what a peer sees its port as.

The short easy fix here is to just run your client in the same docker-compose created network, because then all the NAT behavior disappears. The longer fix involves things like allowing hostnames to be used for inter-cluster process lookup, and figuring out what to do about the port mismatch issues.

[seancarroll]

little late to the party, @dkoston do you have a sample of what you ended up doing?

[dkoston]

@seancarroll here’s a gist: https://gist.github.com/dkoston/9b41cfe44c82a345d3a2a664ae5b41cc

The libraries are from production code but some caveats:

• threw together main.go on my phone so may have errors and it’s not tested
• the tests resolve one of my domains that has no guarantee of being around or not changing IPs so you may want to flip them to point at a domain you control
• copy/paste on phone messed up formatting. As always, gofmt should be used