apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads but also has excellent performance for write-intensive workloads. Users interact with the database using API language binding.

To learn more about FoundationDB, visit foundationdb.org

Documentation

Documentation can be found online at https://apple.github.io/foundationdb/. The documentation covers details of API usage, background information on design philosophy, and extensive usage examples. Docs are built from the source in this repo.

Forums

The FoundationDB Forums are the home for most of the discussion and communication about the FoundationDB project. We welcome your participation! We want FoundationDB to be a great project to be a part of and, as part of that, have established a Code of Conduct to establish what constitutes permissible modes of interaction.

Contributing

Contributing to FoundationDB can be in contributions to the code base, sharing your experience and insights in the community on the Forums, or contributing to projects that make use of FoundationDB. Please see the contributing guide for more specifics.

Getting Started

Binary downloads

Developers interested in using FoundationDB can get started by downloading and installing a binary package. Please see the downloads page for a list of available packages.

Compiling from source

Developers on an OS for which there is no binary package, or who would like to start hacking on the code, can get started by compiling from source.

The official docker image for building is foundationdb/build, which has all dependencies installed. The Docker image definitions used by FoundationDB team members can be found in the dedicated repository.

To build outside of the official docker image, you'll need at least these dependencies:

1. Install cmake Version 3.13 or higher CMake
2. Install Mono
3. Install Ninja (optional, but recommended)

If compiling for local development, please set -DUSE_WERROR=ON in cmake. Our CI compiles with -Werror on, so this way you'll find out about compiler warnings that break the build earlier.

Once you have your dependencies, you can run cmake and then build:

1. Check out this repository.
2. Create a build directory (you can have the build directory anywhere you like).
3. cd <PATH_TO_BUILD_DIRECTORY>
4. cmake -G Ninja <PATH_TO_FOUNDATIONDB_DIRECTORY>
5. ninja # If this crashes it probably ran out of memory. Try ninja -j1

Language Bindings

The language bindings that are supported by cmake will have a corresponding README.md file in the corresponding bindings/lang directory.

Generally, cmake will build all language bindings for which it can find all necessary dependencies. After each successful cmake run, cmake will tell you which language bindings it is going to build.

Generating compile_commands.json

CMake can build a compilation database for you. However, the default generated one is not too useful as it operates on the generated files. When running make, the build system will create another compile_commands.json file in the source directory. This can than be used for tools like CCLS, CQuery, etc. This way you can get code-completion and code navigation in flow. It is not yet perfect (it will show a few errors) but we are constantly working on improving the development experience.

CMake will not produce a compile_commands.json, you must pass -DCMAKE_EXPORT_COMPILE_COMMANDS=ON. This also enables the target processed_compile_commands, which rewrites compile_commands.json to describe the actor compiler source file, not the post-processed output files, and places the output file in the source directory. This file should then be picked up automatically by any tooling.

Note that if building inside of the foundationdb/build docker image, the resulting paths will still be incorrect and require manual fixing. One will wish to re-run cmake with -DCMAKE_EXPORT_COMPILE_COMMANDS=OFF to prevent it from reverting the manual changes.

Using IDEs

CMake has built in support for a number of popular IDEs. However, because flow files are precompiled with the actor compiler, an IDE will not be very useful as a user will only be presented with the generated code - which is not what she wants to edit and get IDE features for.

The good news is, that it is possible to generate project files for editing flow with a supported IDE. There is a CMake option called OPEN_FOR_IDE which will generate a project which can be opened in an IDE for editing. You won't be able to build this project, but you will be able to edit the files and get most edit and navigation features your IDE supports.

For example, if you want to use Xcode to make changes to FoundationDB you can create an Xcode project with the following command:

cmake -G Xcode -DOPEN_FOR_IDE=ON <FDB_SOURCE_DIRECTORY>

You should create a second build-directory which you will use for building and debugging.

FreeBSD

1. Check out this repo on your server.

2. Install compile-time dependencies from ports.

3. (Optional) Use tmpfs & ccache for significantly faster repeat builds

4. (Optional) Install a JDK for Java Bindings. FoundationDB currently builds with Java 8.

5. Navigate to the directory where you checked out the foundationdb repo.

6. Build from source.

   sudo pkg install -r FreeBSD \
       shells/bash devel/cmake devel/ninja devel/ccache  \
       lang/mono lang/python3 \
       devel/boost-libs devel/libeio \
       security/openssl
   mkdir .build && cd .build
   cmake -G Ninja \
       -DUSE_CCACHE=on \
       -DUSE_DTRACE=off \
       ..
   ninja -j 10
   # run fast tests
   ctest -L fast
   # run all tests
   ctest --output-on-failure -v

Linux

There are no special requirements for Linux. A docker image can be pulled from foundationdb/build that has all of FoundationDB's dependencies pre-installed, and is what the CI uses to build and test PRs.

cmake -G Ninja <FDB_SOURCE_DIR>
ninja
cpack -G DEB

For RPM simply replace DEB with RPM.

MacOS

The build under MacOS will work the same way as on Linux. To get boost and ninja you can use Homebrew.

cmake -G Ninja <PATH_TO_FOUNDATIONDB_SOURCE>

To generate a installable package,

ninja
$SRCDIR/packaging/osx/buildpkg.sh . $SRCDIR

Windows

Under Windows, only Visual Studio with ClangCl is supported

1. Install Visual Studio 2019 (IDE or Build Tools), and enable llvm support
2. Install CMake 3.15 or higher
3. Download Boost 1.77.0
4. Unpack boost to C:\boost, or use -DBOOST_ROOT=<PATH_TO_BOOST> with cmake if unpacked elsewhere
5. Install Python if is not already installed by Visual Studio
6. (Optional) Install OpenJDK 11 to build Java bindings
7. (Optional) Install OpenSSL 3.x to build with TLS support
8. (Optional) Install WIX Toolset to build Windows installer
9. mkdir build && cd build
10. cmake -G "Visual Studio 16 2019" -A x64 -T ClangCl <PATH_TO_FOUNDATIONDB_SOURCE>
11. msbuild /p:Configuration=Release foundationdb.sln
12. To increase build performance, use /p:UseMultiToolTask=true and /p:CL_MPCount=<NUMBER_OF_PARALLEL_JOBS>

foundationdb's Issues

Need fstream to compile on Mac

Perhaps you guys have a different setup, but I needed to make the following changes to compile on my Mac:

===================================================================
--- fdbserver/fdbserver.actor.cpp             (revision f069fdf042094deecfdc65a31c6f4e8b03f0a81b)
+++ fdbserver/fdbserver.actor.cpp          (revision bff15c68afe6a370436091e654e4bcd8a24728c0)
@@ -36,6 +36,7 @@
#include "DataDistribution.h"
#include "NetworkTest.h"
#include "IKeyValueStore.h"
+#include <fstream>
#include <stdarg.h>
#include <stdio.h>
#include "pubsub.h"

TLS and Administration pages are not in the site map, and hard to find

Going to the FoundationDB.org SiteMap, I'd expected there to be some mention of the word "SSL" or "TLS" or "Administration", but there seems to be none.

Inevitably, I found both of these using the search bar (initially I could only find TLS in the repository code here, but I found it eventually by searching "Transport Layer Security" very specifically using the Sphinx search functionality). But these should really be linked up front -- they're quite important for real deployments!

create systemd init script

a systemd init script would allow easy installation and management on all modern linux platforms

how to improve connections count from client to server in go sdk or in c sdk

I found that db will be created from a global map in go sdk
when use db in multi go routine，too little connections can't process too many Transactions
how to improve connections count?

Atomic_Add(key, -1) can “leak” keys with value 0 if you are not careful

As dissussed in the forum (https://forums.foundationdb.org/t/atomic-add-key-1-can-leak-keys-with-value-0-if-you-are-not-careful/148) I'm opening an issue here.

One of the issue we had in the past, was with atomic ADD leaking values decremented to 0, which was a common issue when updating static counters used in indexes.

Say we have an index that adds (..., 0, VALUE, ID) = '' for each indexed document, and also atomically increments a counter like (..., 1, VALUE) = (64 bits counter) when a document is indexed for VALUE, and decrements it when the document is later mutated to some other value.

The issue was that decrementing a value down to 0 would keep the key present in the database.

• For "natural" indexes, or ones with a small pool of values that have a high probability to be reused again, this may not be an issue.
• For indexes on "time based" or random-ish fields this is an issue: if the value is a timestamp, or some transient ID that will never be reused, we would leak a ton of (..., TIMESTAMP) = 00 00 00 00 00 00 00 00 in the dataset which could consume a lot of space in the long run (I had to scrub these multiple times, and they would amount to more than 3/4 of the total data size in one extreme case).

Has there been any thing done about this case? I remember asking if maybe decrementing the key down to 0 could maybe also remove it. Or maybe add having a COMPARE_AND_CLEAR atomic operator which would only delete a key if it its value was equal to the argument.

ie: in pseudo code:

SET((..., 0, VALUE, ID), '')
ATOMIC_ADD((..., 1, VALUE), <FF FF FF FF>)
ATOMIC_COMPARE_AND_CLEAR((..., 1, VALUE), <00 00 00 00>)

I had to do some custom logic which would automatically disable this feature when the attribute was of type DateTime or Guid, but this was a bit clunky.

Build errors with golang bindings

Hello,

Could you please provide some more detailed steps how to build golang bindings. From your docker vm for build I get following error, while C and Python bindings were compiled without errors:

make fdb_go
Checking       hgVersion.h
Compiling      fdb
# github.com/apple/foundationdb/bindings/go/src/fdb
/foundationdb/lib/libfdb_c.so: undefined reference to `std::istream::ignore(long)'
/foundationdb/lib/libfdb_c.so: undefined reference to `std::basic_istream<wchar_t, std::char_traits<wchar_t> >::ignore(long)'
collect2: error: ld returned 1 exit status
bindings/go/include.mk:98: recipe for target '/foundationdb/bindings/go/build/pkg/linux_amd64/github.com/apple/foundationdb/bindings/go/src/fdb.a' failed
make: *** [/foundationdb/bindings/go/build/pkg/linux_amd64/github.com/apple/foundationdb/bindings/go/src/fdb.a] Error 2

Adding an explicit read conflict range skips the keys in write cache

If the user adds an explicit read conflict range over one or more keys, then any keys in the write cache are skipped from the explicit conflict range insert. For example, suppose we have an increment counter method that looks like this:

private void incrementCounter(Transaction tr) {
    long counterValue = Tuple.fromBytes(tr.snapshot().get(COUNTER_KEY).join()).getLong(0);
    tr.set(COUNTER_KEY, Tuple.from(counterValue + 1).pack());
    tr.addReadConflictKey(COUNTER_KEY);
}

(Presumably, we are reading at snapshot and then writing and adding a conflict range because we only want to conditionally update the counter, but we always want to know the counter value. This is a little contrived, but there are less contrived examples out there as well.) Then suppose we start two transactions and commit them concurrently:

try (Transaction tr1 = db.createTransaction(); Transaction tr2 = db.createTransaction()) {
    CompletableFuture.allOf(tr1.getReadVersion(), tr2.getReadVersion()).join();
    incrementCounter(tr1);
    incrementCounter(tr2);
    tr1.commit().join();
    tr2.commit().join();
}

Then both commits succeed, but the counter still says "1" as its final value, despite two transactions concurrently incrementing it.

The user can fix this by correctly ordering their writes and read conflict range operations, but it probably isn't the expected semantics for such a call.

GDPR and other suggestions

Awesome, I am using CockroachDB for now because I need easy scalling.
It would be great to:

• Add support for GDPR
• NodeJS driver (I saw some talk about it here but I wanted to show my support)
• Benchmark on a smaller scale (3, 5, 7 nodes), because not everyone plans on using 400 cores out of the box

I also heard SQL performance suffer, it would be interesting to address those and benchmark it and compare to other NoSQL databases

I also heard Apple didn't actually used FoundationDB because of some limitations. Huh. :P

Have an amazing day!

Link to the architecture.pdf file is broken

In the file architecture.rst file (https://github.com/apple/foundationdb/blob/master/documentation/sphinx/source/architecture.rst), the link to the Architecture.pdf file is broken.

Link is https://github.com/apple/foundationdb/blob/master/images/Architecture.pdf
but should be https://github.com/apple/foundationdb/blob/master/documentation/sphinx/source/images/Architecture.pdf

Memory storage engine does not properly report available space

It does not limit based on available memory but rather just on disk space (default storage is 1G).

[Question] Feasibility of implementing a SQL Layer using SQLite's Virtual Table mechanism?

Hello,

I believe it's technically possible to redirect all SQLite's read and write operations to Foundationdb, using Sqlite's so-call Virtual Table mechanism (http://www.sqlite.org/vtab.html).

But the question is how feasible it is?

If someone implement such idea (Sql layer via Sqlite's virtual table mechanism), will the fact that SQLite supports only a single connection prevent such idea to become scale-able (in terms of foundationdb scalability)?

PS, the mORMot framework (for HTTP/REST, DB Access Engine and ORM) is already applying such idea for wrapping access to other database systems using the Sqlite's virtual table approach, and it's working like a charm.

I've been a desktop software developer for a long time so please forgive me if the questions might sound dump :)

Consider setting CPU affinity for fdbserver processes

From https://forums.foundationdb.org/t/setting-cpu-affinity-for-fdbservers/84, it was noted that we don't set CPU affinity for any of the processes we start, and there's likely advantages to doing so. It appears once upon a time, we had code in the resolver that would pin itself to a core, but it was commented out, and I agree doing it from fdbmonitor sounds like a better solution.

Implementing this doesn't sound like too much work, and a quick run through a performance test should tell us if it makes any noticeable impact.

Publish Java client to Maven Central

The Java client should be published to Maven Central so that it is easily accessible to users of Maven and Gradle.

bindings/go: Broken link in README.md

The link to FoundationDB in the README points to https://www.foundationdb.org/documentation/ (which returns a 404) and should probably be changed to https://apple.github.io/foundationdb/

(The Tutorial link is also broken)

Java examples use the wrong package

Was there a re-organization of the package structure in the Java API before launch? Seems like the docs/examples use the prefix "com.apple.foundationdb.*" but the packages actually provided do not include the "apple" segment.

Plans for SQL Layer?

What are the current plans for the SQL layer?

Build instructions on mac

Hi,

I followed the build instructions for building on a mac and installed the necessary dependencies. However, the make command does not seem to be doing much. The OS I have is macOS Sierra. Is there something that I could be doing wrong here?

$ make
flow/coveragetool/generated.mk:40: warning: overriding commands for target `GENTARGET'
flow/actorcompiler/generated.mk:40: warning: ignoring old commands for target `GENTARGET'
fdbclient/vexillographer/generated.mk:40: warning: overriding commands for target `GENTARGET'
flow/coveragetool/generated.mk:40: warning: ignoring old commands for target `GENTARGET'
flow/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbclient/vexillographer/generated.mk:40: warning: ignoring old commands for target `GENTARGET'
fdbrpc/generated.mk:113: warning: overriding commands for target `GENTARGET'
flow/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
fdbclient/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbrpc/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
fdbbackup/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbclient/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
fdbcli/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbbackup/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
bindings/c/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbcli/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
bindings/java/generated.mk:113: warning: overriding commands for target `GENTARGET'
bindings/c/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
fdbmonitor/generated.mk:113: warning: overriding commands for target `GENTARGET'
bindings/java/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
bindings/flow/tester/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbmonitor/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
bindings/flow/generated.mk:113: warning: overriding commands for target `GENTARGET'
bindings/flow/tester/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
FDBLibTLS/generated.mk:113: warning: overriding commands for target `GENTARGET'
bindings/flow/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
make: *** No rule to make target `.objs/fdbserver/ClusterController.actor.g.cpp.o', needed by `bin/fdbserver'.  Stop.

macos + go binding error

branch: master
error desc:

In file included from ../../apple/foundationdb/bindings/go/src/fdb/cluster.go:27:
/usr/local/include/foundationdb/fdb_c.h:35:2: error: Requested API version requires a newer version of this header
#error Requested API version requires a newer version of this header
 ^
1 error generated.

Bindings stack tester converted to Markdown

The specifications for the bindings tester are currently in "txt" formatted files. The files are found under bindings/bindingtester/spec. Ideally, it could be converted to Markdown for cleaner rendering.

Lua(JIT) bindings?

I would be super excited to see Lua(JIT) bindings.

Lua has coroutines, which appear to be a good match to FD's Futures.

Dockerfile Instructionss

Not sure if there's a place that is documenting the build steps but we have been using the Dockerfile under build and one critical step that's missing is the need to have

export CC=/opt/x-toolchain/bin/x86_64-nptl-linux-gnu-gcc
export CXX=/opt/x-toolchain/bin/x86_64-nptl-linux-gnu-g++

For it to work.

Node.js Bindings

We have historically maintained bindings in Node that wrapped our C bindings. However, it was initially developed for an older version of Node, and as time went on, it became incompatible with modern versions of the language. At this point, it is not particularly useful for Node users due to those incompatibilities. It also could probably benefit from some of the new Node features that have been developed in the intervening releases. As a result, it would seem like we should remove the Node bindings until such time as a suitable replacement is available. If there is demand for such a binding, the old Node bindings will still exist in history, so our previous work could plausibly be used as a jumping off point for someone who wanted to create their own.

Document MacOS and Windows TLS build instructions

Building the TLS Plugin requires LibreSSL, which is provided within the docker image for building in Linux, but would need to be installed by hand for MacOS and Windows currently. Doing so, or making it easier to do so, needs to be documented in our build instructions.

Broken image in the Architecture documentation

https://apple.github.io/foundationdb/architecture.html

It appears you're trying to use a PDF in an image tag, which won't work. I've attached a PNG you can use on the page, which you can link to the PDF

Update versionstamp mutation types to make them consistent

As the API stands now, the SET_VERSIONSTAMP_KEY mutation type allows users to choose an arbitrary location for their versionstamp by appending an offset to the end of their key. In contrast, the SET_VERSIONSTAMP_VALUE mutation type forces users to place their version at the beginning. The proposal of this issue is thus the following two things:

1. It should be possible to place a versionstamp at an arbitrary offset of a value.
2. The API for the two of them should be the same (i.e., append an offset with a fixed number of bytes to the end).

Because values can be longer than 65 kB, the way to do this that keeps them consistent is to have both mutations use a 4 byte offset instead of a 2 byte one. (In theory, 3 bytes could be used, but probably inventing 24-bit integers for this use case is worse than just sending an extra byte over the wire.) We can API version guard this so that only users who have requested API version 520 or newer will get the change, so old code will continue to work as desired.

Problems compiling on macOS 10.13.4: (some destructor) has a non-throwing exception specification but can still throw [-Werror,-Wexceptions]

Hi

It is most likely a user error.

$ c++ --version
Apple LLVM version 9.1.0 (clang-902.0.39.1)
Target: x86_64-apple-darwin17.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

fdbclient/MultiVersionAssignmentVars.h:135:2: note: destructor has a implicit non-throwing exception specification
~DLThreadSingleAssignmentVar() {
^
fdbclient/MultiVersionAssignmentVars.h:138:4: error: '~DLThreadSingleAssignmentVar' has a non-throwing exception specification but can still throw [-Werror,-Wexceptions]
ASSERT(futureRefCount == 1);
^
./flow/Error.h:84:81: note: expanded from macro 'ASSERT'
#define ASSERT( condition ) if (!((condition) || isAssertDisabled(LINE))) { throw internal_error(); }

I can work around it by adding noexcept(false) to the destructors, but is there a better way?

Docker container doesn't pin dependency versions

It looks like after this change the build container no longer pins the exact versions of every system package installed. It also appears to allow apt to pull in transitive dependencies, rather than specifying all packages explicitly.

This makes it a little bit harder to understand exactly what the build environment depends on, and also prevents our builds from being reproducible. In the worst case, you could imagine a situation where after all the testing has occurred during a release, Ubuntu upstream bumps a package version which causes a subtle change in behavior.

My guess is this was done because in practice, the team at Apple never rebuilds the container, but has a known good copy of it published somewhere? Is that right? Or was there some other rationale?

I think the two most straightforward solutions are either to pin the versions again, or to publish an "official" container, direct people to use it, and keep this file around mostly as documentation.

What do you think?

Evangelism

How is FoundationDB reaching out to get the internet community on board after open sourcing?

If not yet decided, maybe a github issue thread here is good enough for now.

Sites to get on boarded (found by googling FoundationDB vs ... )

• https://db-engines.com/en/system/FoundationDB%3BMongoDB - I can't see how to report the mistake on that webpage. Maybe someone here around can figure it out.

Swift Bindings

Are there any plans for official Swift bindings for FoundationDB?

Internal RPC error with basic fdbcli connection

$ /usr/local/bin/fdbcli -C ./client.cluster
Using cluster file `./client.cluster'.
Internal Error @ fdbrpc/FlowTransport.actor.cpp 657:
  atos -o fdbcli.debug -arch x86_64 -l 0x101fe7000 0x10233a5cd 0x1022a5e91 0x1022a514d 0x101feca78 0x102361916 0x102349631 0x1020bae5b 0x10200a6de 0x7fff59e48115
[lots more errors]

Server: Custom Docker container based on ubuntu:16.04 that has installed foundationdb-clients_5.1.5-1_amd64.deb and foundationdb-server_5.1.5-1_amd64.deb. Using default config.

Client: Installed with macOS installer, FoundationDB-5.1.5.pkg. macOS 10.13.3.

Full fdbcli output and logs from server here.

Missing SQL-Parser

Will be the SQL-Parser part of the Open Source version ?

Memory storage engine can selectively abort/skip recovery

Just a tracking issue since we (Wavefront) have something pretty close in the 3.x branch. The idea is to have memory storage nodes abort recovery if the cluster is healthy (and just join as an empty node). It would also abort recovery (if in-flight) should the cluster turn healthy. Most of the time, they spend N minutes (roughly one minute per G of storage_memory) reading from disk for a healthy cluster only to just discard their data.

Python: fdb network thread blocks main thread when gevent is monkey patched

Hiya,

We make use of gevent's monkey.patch_all functionality in order to make all of the pure-Python libraries play nicely with gevent. This is pretty successful, but it currently causes the fdb network thread to block the main application thread.

Demo: https://gist.github.com/zorkian/824e86d0d7f001965f62dc8ae7ad28db

This is tested in gevent 1.1.1 as well as the latest 1.3b1.

You can alleviate the blocking by using gevent.monkey.patch_all(thread=False), but this currently doesn't work in our environment due to certain modules we use that aren't implemented to use gevent and require threading.

I've been playing around with the fdb library to see if I can make it work, but haven't succeeded as yet. Figured I'd file the issue just to put it on the radar.

Flow directory tester snapshot reads are not RYW

Right now, the flow directory layer does not support snapshot reads. This in and of itself is a shortcoming of the flow bindings, but it is not so bad really. The flow binding tester attempts to get around this with the following bit of code:

if (isDirectory && isSnapshot) {
    Version readVersion = wait(instruction->tr->getReadVersion());
    instruction->tr = Reference<Transaction>(new Transaction(data->db));
    instruction->tr->setVersion(readVersion);
}

The problem is that though this ensures that the "snapshot" transaction has the same view of the database as the regular transaction, it is missing read-your-writes support. This means that the tests can fail if there is something along the lines of:

1. Transaction starts.
2. The directories are modified.
3. The directories are read at snapshot isolation level.

The flow tester will miss the modifications, so it will print out different results.

I ran into this error when running the tests locally. Here is the input that I can give to the bindingtester to make it reproduce:

python bindingtester.py --test-name directory --num-ops 261 flow --seed 3583312077 --compare

(The number of ops, 261, is the minimum needed to get it to reproduce with that seed.) I was running it against commit 5fdcd62 which, at the time, was the tip of the release-5.2 branch, to the extent that that is important for reproduction. Here is a print out of the relevant section of adding --print-test --all to the command:

  2175. 'RESET'
  2176. 'PUSH' 0
  2177. 'DIRECTORY_REMOVE_DATABASE'
  2178. 'COMMIT'
  2179. 'WAIT_FUTURE'
  2180. 'RESET'
  2181. 'PUSH' 3
  2182. 'DIRECTORY_CHANGE'
  2183. 'PUSH' 'default-6\x1e\xde-M\\R:\xea\x19\xdd\xe7\xa3\x96\xaa\x9d'
  2184. 'PUSH' ''
  2185. 'PUSH' u'default1'
  2186. 'PUSH' 1
  2187. 'DIRECTORY_CREATE_DATABASE'
  2188. 'PUSH' 4
  2189. 'DIRECTORY_CHANGE'
  2190. 'PUSH' u'1'
  2191. 'PUSH' 1
  2192. 'PUSH' 1
  2193. 'DIRECTORY_REMOVE'
  2194. 'PUSH' u'\u70c1\uc3f9\ufe98\u34e1\U0002a2b2\uffa3\u6283\u5f6a\u5a34\ufc6c\u7526\u3df9\u0c1e\u6b44\u532e\u7269\u2d1c\ub766\u583b\uacd9\u3b0d\uc34e\u4218\u21f4'
  2195. 'PUSH' ()
  2196. 'PUSH' 2
  2197. 'DIRECTORY_PACK_KEY'
  2198. 'DIRECTORY_UNPACK_KEY'
  2199. 'PUSH' 27
  2200. 'DIRECTORY_CHANGE'
  2201. 'PUSH' '\x13\x11\x05'
  2202. 'PUSH' 0
  2203. 'DIRECTORY_CREATE_SUBSPACE'
  2204. 'PUSH' 97
  2205. 'DIRECTORY_CHANGE'
  2206. 'PUSH' u'2'
  2207. 'PUSH' u'3'
  2208. 'PUSH' u'3'
  2209. 'PUSH' 3
  2210. 'PUSH' 0
  2211. 'DIRECTORY_MOVE'
  2212. 'PUSH' 0
  2213. 'PUSH' 1
  2214. 'DIRECTORY_LIST_SNAPSHOT'

This leads to the following discrepancy between flow and python:

Incorrect result: 
  flow   - ('tester_output', 'stack', 269, 2214) = '\x01\x021\x00\xff\x00'
  python - ('tester_output', 'stack', 269, 2214) = '\x01\x00'

I hacked up the binding tester to actual emit non-snapshot reads whenever it thinks it should actually emit a snapshot read, and then the test passed. So, I don't think it's actually a bug in the directory layer, just that the lack of RYW in snapshot tests.

This can cause spurious errors, so it would be nice to get it fixed. In some sense, the "correct" solution is to make the flow directory layer support snapshot reads. I'm not sure if there is an easier way. In theory, we could just give the snapshot tester a regular transaction. This can lead to a different kind of spurious error (additional not_committed errors), so that's not air-tight. In theory, we could also log all of the writes and replay them within the snapshot transaction or do something like keep two transactions around and use both of them for writes, but perform non-snapshot reads and commits on one and "snapshot" reads on the other. But this might be harder than just supporting snapshot reads.

There should be a `make install` target

Currently, FoundationDB does not have a make install target for Linux based machines -- you're expected to use the dpkg or rpm which have the right layout (but I have no idea how this is generated). But most distros will be expecting this, and it would be good form and a lot easier to write installers, e.g. like the NixOS installer. It would also be good for macOS. Some of the file layout requirements surprised me (initially); copying from my initial post on the FoundationDB forums:

• Please add a make install target. This should not rely on any particular packaging file, it should just run install on the files in the right places. Unless I completely missed it (I don't think I did?), this is an oversight that makes 3rd party packaging much more annoying. In one particular annoyance, fdbrestore, fdbdr, dr_agent etc are all actual symlinks to one binary (backup_agent), and determine their operation based on the exe name. I didn't realize this until I had dug in and read the source code! So instead I had to 'extract' the fdb .dpkg and look at the filesystem hierarchy. Then I mostly replicated that. But make install would have done it for me...

  Please note that it is essential that a command like make install respect the traditional PREFIX= and DESTDIR= environment variables, so that maintainers can install them into arbitrary places, which is vital. These environment variables do not need to have any special meaning to the FoundationDB source code, they merely need to exist, and be respected, when finally copying files around during make install. (FDB, as far as I can see, doesn't rely so much on fixed hardcoded paths outside of /etc/foundationdb, and even that is fixable in the fdbmonitor file.) I'm sure you're aware of it, but I only mention it because adding make install but not adding that will just have distro maintainers right back here. :)

building on linux

Is it possible to build this on Linux or other unixes?

Performant way to extract locality information and shard boundaries

Currently the only way to get the full picture of where shards are located is to get the shards (that's relatively fast, a single iteration through a system key space) and then ask for the storage servers for each of those shard one-by-one. It doesn't help that asking for IP addresses must be throttled so that you don't cause the responsible stateless processes to become pegged. There should be an easier way to accomplish this.

We need this API mainly to build coprocessors that live on the same machine as the DB and needing to figure out what data is local.

Rust bindings?

I'm interested in providing Rust bindings for FoundationDB. Would there be interest in including these within the FoundationDB repository along side the other languages?

Improvements to Java Tuples API

We at @wavefrontHQ, while we use Tuples extensively, we have developed tools to both pack and extract elements efficiently (saving memory allocations and improving performance). We'd like to donate those pieces (since it needs to be maintained with the mainline Java Tuple class) at some point.

Tracking it in this issue in case people are interested.

For instance, things like:

• public static byte[] pack(Object... list) to pack a bunch of elements together without creating an intermediate Tuple object
• Various get(byte[] input, int skip) calls with the ability to "output" the offset within tuple after extracting the element so that, again, one does not have to parse the whole tuple to get an element (often times, you know a fixed offset and would want to just extract, say, a string at offset 8 for every key you see)
• The class uses thread locals heavily as well to cache intermediate objects that are used for decoding so that it doesn't thrash the heap

Flow bindings: Directory layer should support snapshot reads

This would allow us to inspect directories without adding read conflict ranges, which might be something someone would want to do.

In some sense, this is the root cause of #157. However, that issue points to a bug in the tester that is causing spurious errors. Fixing this issue would give us #157 basically for free, but this somewhat of a separate issue.

Python docs code is invalid on Python 3

In Python 3 bytes and string are different. Hence bytes need to be prefixed explicitly with 'b' in Python 3. This is not necessary in Python 2. The docs don't have the prefix causing error in Python 3.

➜  foundationdb-5.1.5 python
Python 2.7.14 (default, Mar 12 2018, 13:54:56)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import fdb
>>> fdb.api_version(510)
>>> db = fdb.open()
>>> db['hello']

Python 3

➜  foundationdb-5.1.5 python3
Python 3.6.4 (default, Mar 12 2018, 13:42:53)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import fdb
>>> fdb.api_version(510)
>>> db = fdb.open()
>>> db[b'hello']
>>> db['hello'] # Breaks
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 893, in __getitem__
    return Database.__database_getitem(self, key)
  File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 265, in wrapper
    ret = func(*largs, **kwargs)
  File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 946, in __database_getitem
    return tr[key].value
  File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 450, in __getitem__
    return self.get(key)
  File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 410, in get
    key = keyToBytes(key)
  File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 1271, in keyToBytes
    raise TypeError('Key must be of type ' + bytes.__name__)
TypeError: Key must be of type bytes

mac 5.1.5.pkg install, ERROR: Database is locked (1038)

installed with FoundationDB-5.1.5.pkg, then start fdbcli, enter set a b , a error comed out : ERROR: Database is locked (1038)

Support for 3.x clusters to live migrate

I remember @ajbeamon mentioning that there's something that needs to be changed in the 3.x code (IIRC?) that will allow us to ship a foundationdb-client package that can talk to a both a 3.x cluster and a 5.x cluster. I can't seem to find that in the code, can someone point us in the right direction?

Publish Python bindings to PyPI

It will be good if we can publish Python bindings to PyPI so that we can do pip install foundationdb . Currently bindings are present as a tar.gz file and can be installed as below :

pip install https://www.foundationdb.org/downloads/5.1.5/bindings/python/foundationdb-5.1.5.tar.gz

Similar issue with respect to Java and Maven : #219

Documentation: embedded pdf is not visible in Chrome

Link: https://apple.github.io/foundationdb/architecture.html

Architecture.pdf is embedded well in other browsers, but not in Chrome. Possible fix can be to convert it to PNG format. Will be more consistent with other images in documentation (they all are in PNG format).

Need more examples on go api

need some go examples

see the code comments below

package main

import (
	"fmt"
	"log"

	"github.com/apple/foundationdb/bindings/go/src/fdb"
	"github.com/apple/foundationdb/bindings/go/src/fdb/directory"
	"github.com/apple/foundationdb/bindings/go/src/fdb/tuple"
)

func main() {
	fdb.MustAPIVersion(510)
	db := fdb.MustOpenDefault()

	nmqDir, err := directory.CreateOrOpen(db, []string{"nmqtest"}, nil)
	if err != nil {
		log.Fatal(err)
	}

	testApp := nmqDir.Sub("testapp")
	testTopic := "testtopic"

	// store messages
	_, err = db.Transact(func(tr fdb.Transaction) (ret interface{}, err error) {
		for i := 1; i < 70; i++ {
			SCKey := testApp.Pack(tuple.Tuple{testTopic, fmt.Sprintf("%02d", i), fmt.Sprintf("%02d", i-1)})
			tr.Set(SCKey, []byte("11111"))
		}
		return
	})

	// query messages
	var keys []string
	_, err = db.Transact(func(tr fdb.Transaction) (ret interface{}, err error) {
                // I want to query 5 messages from the testTopic from a specify position
                // so, is there any solutions for this, may like the python code:
                // tr.get_range(begin, fdb.KeySelector.first_greater_than(begin) + 5)
		pr, _ := fdb.PrefixRange([]byte("testapp"))
		pr.Begin = testApp.Pack(tuple.Tuple{testTopic, fmt.Sprintf("%02d", 4)})
		pr.End = testApp.Pack(tuple.Tuple{testTopic, fmt.Sprintf("%02d", 69)})
		ir := tr.GetRange(pr, fdb.RangeOptions{Limit: 5}).Iterator()
		for ir.Advance() {
			k := ir.MustGet().Key
			keys = append(keys, string(k))
		}
		return
	})

	fmt.Println(keys)

       // clear all messages
	_, err = db.Transact(func(tr fdb.Transaction) (ret interface{}, err error) {
		tr.ClearRange(testApp.Sub(testTopic))
		return
	})
}

so, please add more examples of tuples range and offset ,thans very much : )

Large cluster memory limits (>30TB)

Perhaps this has already been addressed but we have always observed that 3.x clusters would need to have their process memory limit in /etc/foundationdb/foundationdb.conf increased beyond 8G (we have 10G for a cluster that has 30+TB of KVs (100+TB on disk). Perhaps at that size the recommendation would be to always have separate transaction processes.

Backup to "blobstore"

There is some cryptic information about the blobstore backup option and originally we figure you can just do something like: fdbbackup start --blob_credentials blob-creds -d blobstore://[email protected]/<my_s3_bucket>/<subfolder>

The first thing is that it's trying to connect in http (and not https) so we relaxed the S3 bucket so that it'd allow non-secure connections. Then, it seems like it's not really using whatever that's after the host:port and url params (the <my_s3_bucket>/<subfolder> part from the example above) but instead tries to do a PUT against /FDB_BACKUP_V2 (that's a constant in the code) which of course fails. At this point, we started to wonder is this really S3 (but the code does expect the endpoint to respond with an S3-like API).

Bottom-line, how does one specify an S3 bucket as the backup destination?

Linux build from source with docker: fdbmonitor missing

I dont know why it is missing, according to the Makefile it should be in bin/ just like the other binaries.
Is this a critical process that I need to operate foundationdb? Because I dont see why this simple 1 process "cluster" keeps failing in various ways:

And if I try to add another process to the cluster, even the first fails to connect to the cluster (itself):

Log: http://pastebin.centos.org/691466/

Python asyncio bindings

Would you like add asyncio python 3+ bindings?