cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.

---

CockroachDB is a cloud-native distributed SQL database designed to build, scale, and manage modern, data-intensive applications.

• What is CockroachDB?
• Docs
• Starting with Cockroach Cloud
• Starting with CockroachDB
• Client Drivers
• Deployment
• Need Help?
• Contributing
• Design
• Comparison with Other Databases
• See Also

What is CockroachDB?

CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. It scales horizontally; survives disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports strongly-consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.

For more details, see our FAQ or architecture document.

Docs

For guidance on installation, development, deployment, and administration, see our User Documentation.

Starting with CockroachCloud

We can run CockroachDB for you, so you don't have to run your own cluster.

See our online documentation: Quickstart with CockroachCloud

Starting with CockroachDB

1. Install CockroachDB: using a pre-built executable or build it from source.
2. Start a local cluster and connect to it via the built-in SQL client.
3. Learn more about CockroachDB SQL.
4. Use a PostgreSQL-compatible driver or ORM to build an app with CockroachDB.
5. Explore core features, such as data replication, automatic rebalancing, and fault tolerance and recovery.

Client Drivers

CockroachDB supports the PostgreSQL wire protocol, so you can use any available PostgreSQL client drivers to connect from various languages.

• For recommended drivers that we've tested, see Install Client Drivers.
• For tutorials using these drivers, as well as supported ORMs, see Example Apps.

Deployment

• CockroachCloud - Steps to create a free CockroachCloud cluster on your preferred Cloud platform.
• Manual - Steps to deploy a CockroachDB cluster manually on multiple machines.
• Cloud - Guides for deploying CockroachDB on various cloud platforms.
• Orchestration - Guides for running CockroachDB with popular open-source orchestration systems.

Need Help?

• CockroachDB Community Slack - Join our slack to connect with our engineers and other users running CockroachDB.
• CockroachDB Forum and Stack Overflow - Ask questions, find answers, and help other users.
• Troubleshooting documentation - Learn how to troubleshoot common errors, cluster setup, and SQL query behavior.
• For filing bugs, suggesting improvements, or requesting new features, help us out by opening an issue.

Building from source

See our wiki for more details.

Contributing

We welcome your contributions! If you're looking for issues to work on, try looking at the good first issue list. We do our best to tag issues suitable for new external contributors with that label, so it's a great way to find something you can help with!

See our wiki for more details.

Engineering discussions take place on our public mailing list, [email protected]. Also please join our Community Slack (there's a dedicated #contributors channel!) to ask questions, discuss your ideas, and connect with other contributors.

Design

For an in-depth discussion of the CockroachDB architecture, see our Architecture Guide. For the original design motivation, see our design doc.

Licensing

Current CockroachDB code is released under a combination of two licenses, the Business Source License (BSL) and the Cockroach Community License (CCL).

When contributing to a CockroachDB feature, you can find the relevant license in the comments at the top of each file.

For more information, see the Licensing FAQs.

Comparison with Other Databases

To see how key features of CockroachDB stack up against other databases, check out CockroachDB in Comparison.

See Also

• Tech Talks (by CockroachDB founders, engineers, and customers!)
• CockroachDB User Documentation
• The CockroachDB Blog
• Key design documents
  • Serializable, Lockless, Distributed: Isolation in CockroachDB
  • Consensus, Made Thrive
  • Trust, But Verify: How CockroachDB Checks Replication
  • Living Without Atomic Clocks
  • The CockroachDB Architecture Document

cockroach's Issues

make testrace PKG=./util/metrics

race testing may occasionally fail, probably due to channels filling up.

E0929 15:39:02.424096 08458 metrics.go:566] a raw subscriber has allowed their channel to fill up. dropping their metrics on the floor rather than blocking.

E0929 15:39:02.424554 08458 metrics.go:566] a raw subscriber has allowed their channel to fill up. dropping their metrics on the floor rather than blocking.

E0929 15:39:02.424627 08458 metrics.go:569] this raw subscriber has caused dropped metrics at least 3 times in a row. closing the channel.

--- FAIL: TestUpdateSubscribers (0.00 seconds)

metrics_test.go:256: received no raw metrics from the MetricSystem after 2 milliseconds.

metrics_test.go:264: received no processed metrics from the MetricSystem after 2 milliseconds.

FAIL

FAIL github.com/cockroachdb/cockroach/util/metrics 0.045s

make: *** [testrace] Error 1

The command "make testrace" exited with 2.

Done. Your build exited with 1.

Distributed transactions

This is an umbrella issue for now for accounting purposes. Smaller issues should be created for the various paths a transaction can take.

add TLS to HTTP server

entry point is server/server.go.

Since we want https to be the standard (or even enforced) option to talk to the server in the client, the new Transport.DialTLS option might have some benefits worth looking at (but it looks like we don't need it, at least not now).
https://golang.org/pkg/net/http/#Transport.DialTLS

build-docker-dev failing at gflags step due to invalid SSL cert

+ cd /go/src/github.com/cockroachdb/cockroach/_vendor/usr/tmp
+ curl -L https://gflags.googlecode.com/files/gflags-2.0-no-svn-files.tar.gz
+ tar --no-same-owner -xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (60) SSL certificate problem: certificate is not yet valid
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
2014/11/22 14:04:25 The command [./build/devbase/vendor.sh] returned a non-zero code: 2

Integrate with Swagger

Per discussion on #224. http://swagger.io/

Disallow the insertion of empty keys

From @Tobstar87:

I'm wondering if we allow empty keys to be inserted. The rockdb handler doesn't handle it very well, it will crash whenever encountering Key("") for anything that's not an insert as it seeks to &start[0] (similarly to my recent bug fix in rocksdb.scan()).
Either we'll have to disallow that key or treat it differently.

Add counter support to REST Explorer

Refs: #68363 (ConcurrentTransactionTest#test_transaction_isolation__read_committed, first hit Aug 12)

Setup a table:

CREATE TABLE "developers" ("id" bigserial primary key, "name" character varying, "first_name" character varying, "salary" int DEFAULT '70000', "firm_id" bigint, "mentor_id" int, "legacy_created_at" timestamp, "legacy_updated_at" timestamp, "legacy_created_on" timestamp, "legacy_updated_on" timestamp);
 insert into developers (id) values (1);

In shell 1, type:

BEGIN;
SELECT "developers"."id", "developers"."name", "developers"."salary", "developers"."firm_id", "developers"."mentor_id", "developers"."legacy_created_at", "developers"."legacy_updated_at", "developers"."legacy_created_on", "developers"."legacy_updated_on" FROM "developers" WHERE "developers"."id" = 1 LIMIT 1;
UPDATE "developers" SET "salary" = 200000 WHERE "developers"."id" = 1;

In shell 2, run:

BEGIN;

SELECT "developers"."id", "developers"."name", "developers"."salary", "developers"."firm_id", "developers"."mentor_id", "developers"."legacy_created_at", "developers"."legacy_updated_at", "developers"."legacy_created_on", "developers"."legacy_updated_on" FROM "developers" WHERE "developers"."id" = 1 LIMIT 1;

Back to shell 1:

SELECT "developers"."id", "developers"."name", "developers"."salary", "developers"."firm_id", "developers"."mentor_id", "developers"."legacy_created_at", "developers"."legacy_updated_at", "developers"."legacy_created_on", "developers"."legacy_updated_on" FROM "developers" WHERE "developers"."id" = 1 LIMIT 1;

UPDATE "developers" SET "salary" = 80000 WHERE "developers"."id" = 1;

SELECT "developers"."id", "developers"."name", "developers"."salary", "developers"."firm_id", "developers"."mentor_id", "developers"."legacy_created_at", "developers"."legacy_updated_at", "developers"."legacy_created_on", "developers"."legacy_updated_on" FROM "developers" WHERE "developers"."id" = 1 LIMIT 1;

COMMIT;

Now in shell 2, we see a salary 200000, which is a dirty read (we never committed a salary of 200000):

SELECT "developers"."id", "developers"."name", "developers"."salary", "developers"."firm_id", "developers"."mentor_id", "developers"."legacy_created_at", "developers"."legacy_updated_at", "developers"."legacy_created_on", "developers"."legacy_updated_on" FROM "developers" WHERE "developers"."id" = 1 LIMIT 1;
;
  id | name | salary | firm_id | mentor_id | legacy_created_at | legacy_updated_at | legacy_created_on | legacy_updated_on
-----+------+--------+---------+-----------+-------------------+-------------------+-------------------+--------------------
   1 | NULL | 200000 |    NULL |      NULL | NULL              | NULL              | NULL              | NULL
(1 row)

This can take a bit of time to reproduce (i've had to do this a few times each reproduction), but an external ORM, the contributor to the ORM and I have hit it multiple times, so it is not a fluke.

Jira issue: CRDB-10701

Failed to bootstrap cluster

When I run at first time cockroach, it works:

./cockroach init -rpc="localhost:0" -certs="resources/test_certs" -stores="ssd=/tmp/db"

But if I stop and then start it again by same command with same params, then it raises the follow error:

E1209 14:09:11.300891   21009 server.go:160] Failed to bootstrap cluster: store.go:393: bootstrap failed; non-empty map with first key "%\x80\x80\x80\x86ˑ\xca\xee\x00"

I must delete /tmp/db If I want that it works again.

Ordered Key Encoding for float64 keys

Based on http://sqlite.org/src4/doc/trunk/www/key_encoding.wiki

tests: design for a network simulator debugging tool

Megastore has a pseudo-random test framework capable of exploring the space of all possible orderings and delays of communications between simulated nodes or threads, and deterministically reproduces the same behavior given the same seed.

This would be unbelievably useful.

/cc @toddlipcon since he’s had experience designing something like this for HDFS HA.

Multiraft: real Storage implementation

We need to implement the multiraft.Storage interface backed by the rocksdb store.

ensure we're linking jemalloc for rocksdb

Multiraft: retry on election

Whenever there is an election, we need to re-propose all commands in the multiraft.group.pending map, since these may have been lost. (they may not have been, but all commands should be idempotent and so it's simpler to just re-propose everything than to try to be more precise about this)

tests: acceptance test for starting a cluster

在**大陆关注 cockroach-db的同学们，咱们来一次meetup怎么样？

大家好，我来自华为开源软件能力中心的程广卫，cockroach-db是我关注的开源项目，想和大家交流 cockroach-db的未来及参与贡献开发等，有兴趣的可以加我微信 18500988099，或者电话我

meetup的地点暂定北京，时间为11月22日或者23日
本次meetup由华为赞助场地等

linux: link error with -lroachproto

cd /home/vagrant/gopath/src/github.com/cockroachdb/cockroach/proto; make static_lib
make[1]: Entering directory `/home/vagrant/gopath/src/github.com/cockroachdb/cockroach/proto'
protoc --gogo_out=. --cpp_out=lib --proto_path=.:../../../../:../../../../code.google.com/p/gogoprotobuf/protobuf:../../../../code.google.com/p/gogoprotobuf/gogoproto api.proto config.proto data.proto errors.proto
/home/vagrant/gopath/bin/protoc-gen-gogo: 1: /home/vagrant/gopath/bin/protoc-gen-gogo: Syntax error: "(" unexpected
--gogo_out: protoc-gen-gogo: Plugin failed with status code 2.
make[1]: *** [api.pb.go] Error 1
make[1]: Leaving directory `/home/vagrant/gopath/src/github.com/cockroachdb/cockroach/proto'
make: *** [roachproto] Error 2

http: graceful shutdown of server

From: https://code.google.com/p/go/issues/detail?id=4674:

Status update: Go 1.3 will have all the necessary net/http changes (Server.ConnState and
Server.SetKeepAlivesEnabled) for other people to implement graceful shutdown properly,
outside the net/http package.

Go 1.3 is out and we are compiling against it. We should shut down the servers gracefully.

Take a look at func (s *server) stop() in server/server.go

[question] What is the source of the transaction mechanism?

After reading your design paper I started looking around, and got the feeling that your transactions are based on http://cs-www.cs.yale.edu/homes/dna/papers/calvin-sigmod12.pdf

Is this correct? :)

If so, have you seen http://www.vldb.org/pvldb/vol7/p329-mahmoud.pdf ?

It seems to be based on similar semantics, but is possibly simpler and more efficient..

Data race in kv/... tests

==================
WARNING: DATA RACE
Read by goroutine 10:
  github.com/cockroachdb/cockroach/util/hlc.*ManualClock.UnixNano·fm()
      /go/src/github.com/cockroachdb/cockroach/kv/local_sender_test.go:122 +0x39
  github.com/cockroachdb/cockroach/util/hlc.(*Clock).PhysicalNow()
      /go/src/github.com/cockroachdb/cockroach/util/hlc/hlc.go:160 +0x92
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /go/src/github.com/cockroachdb/cockroach/storage/range.go:436 +0xfde
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /go/src/github.com/cockroachdb/cockroach/storage/range.go:280 +0x33c
  github.com/cockroachdb/cockroach/storage.func·015()
      /go/src/github.com/cockroachdb/cockroach/storage/store.go:671 +0x150
  github.com/cockroachdb/cockroach/util.RetryWithBackoff()
      /go/src/github.com/cockroachdb/cockroach/util/retry.go:80 +0x71
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /go/src/github.com/cockroachdb/cockroach/storage/store.go:702 +0x468
  github.com/cockroachdb/cockroach/kv.func·005()
      /go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:129 +0x43d
  github.com/cockroachdb/cockroach/util.RetryWithBackoff()
      /go/src/github.com/cockroachdb/cockroach/util/retry.go:80 +0x71
  github.com/cockroachdb/cockroach/kv.(*LocalSender).Send()
      /go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:144 +0x196
  github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).heartbeat()
      /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:479 +0x597

Previous write by goroutine 127:
  github.com/cockroachdb/cockroach/kv.func·015()
      /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender_test.go:305 +0x148
  github.com/cockroachdb/cockroach/util.IsTrueWithin()
      /go/src/github.com/cockroachdb/cockroach/util/testing.go:83 +0x74
  github.com/cockroachdb/cockroach/kv.TestTxnCoordSenderHeartbeat()
      /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender_test.go:312 +0x62c
  testing.tRunner()
      /usr/src/go/src/pkg/testing/testing.go:422 +0x10f

Goroutine 10 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).sendOne()
      /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:287 +0xa96
  github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send()
      /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x1af
  github.com/cockroachdb/cockroach/client.(*KV).Call()
      /go/src/github.com/cockroachdb/cockroach/client/kv.go:123 +0x40d
  github.com/cockroachdb/cockroach/kv.TestTxnCoordSenderHeartbeat()
      /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender_test.go:290 +0x452
  testing.tRunner()
      /usr/src/go/src/pkg/testing/testing.go:422 +0x10f

Goroutine 127 (finished) created at:
  testing.RunTests()
      /usr/src/go/src/pkg/testing/testing.go:504 +0xb46
  testing.Main()
      /usr/src/go/src/pkg/testing/testing.go:435 +0xa2
  main.main()
      github.com/cockroachdb/cockroach/kv/_test/_testmain.go:125 +0xdc
==================

Full log: http://cl.ly/text/072t0G0Z1A2r

Multiraft: real Transport implementation

We need to replace multiraft.LocalRPCTransport (outside of tests) with our real multiraft.Transport implementation, including mapping from node IDs to real addresses.

Orchestrate CockroachDB with a Juju Charm

What is Juju

Juju is the Service Orchestration Solution developed by Canonical (the company behind Ubuntu).

It focused on creating relations between services instead of the single deployment of an application. Charms are the language of Juju. A Charm is a collection of scripts that details how to deploy, manage, scale up and down, and hook a piece of software to its environment.
Several charms can then be attached together in a Bundle that represents a prefered/recommended way of deploying and scaling a complete infrastructure.
Charms are open source, and distributed through the Charm Store

Benefits

For CockroachDB, adding a Charm as a deployment method would have several key benefits.

• Testing is much easier by providing autoscaling and one-click / one-liner deployments to any cloud, and bare metal deployment
• Integration made easy with hooks to several pieces of software (chaos monkeys, but also data sources and data consumers)
• When CockroachDB moves to beta and to production, easy hooks to monitoring, backup (...) solutions already in the charmstore. It will make it easy for anyone to deploy and maintain cockroachDB up & running.

Requirements & Effort

Writing a Charm can be done in any language, including adapting Puppet, Chef or Ansible. If you know how to deploy and manage an application, getting a first version can take from less than a day to three days.

If you want to dig into more details, Juju is Open Source and written in Go. You can read/fork the code of the various pieces here : https://github.com/juju

Where to start?

The best way is to start with the online documentation

Canonical also provides Charm Schools, which are 2hrs webinar training sessions to get you up to speed.

kv rest api: counter should be indicated within a range response

currently, the ScanResponse is as follows:

type ScanResponse struct {
    ResponseHeader `protobuf:"bytes,1,opt,name=header,embedded=header" json:"header"`
    // Empty if no rows were scanned.
    Rows             []KeyValue `protobuf:"bytes,2,rep,name=rows" json:"rows"`
    XXX_unrecognized []byte     `json:"-"`
}

If we query for a range with a counter, there is no indication that the counter row is an actual counter other than the fact that the row value is empty.

If we query the row directly using the counter endpoint, it returns the raw counter value, not a json-serialized proto response.

Specify default zone configurations

E0525 17:43:15.273643 05697 range.go:161] failed init accounting configuration map: prefix.go:111: no default prefix specified
E0525 17:43:15.281753 05697 range.go:161] failed init permissions configuration map: prefix.go:111: no default prefix specified
E0525 17:43:15.281928 05697 range.go:161] failed init zones configuration map: prefix.go:111: no default prefix specifiedadmin REST request failed: zone_cli.go:49: 500 Internal Server Error: zone.go:127: the default zone configuration cannot be deleted

Multiraft: rebalancing/membership changes

Introduce interfaces for the rebalancing process to add and remove members from a raft group.

Multiraft: leader leases

We need to introduce a concept of leader leases so a leader can take read-only actions without writing them to the raft log. This means

• Ensuring that no node calls an election without first waiting for at least the leader lease timeout (including just after starting up)
• Keeping track of heartbeat responses and reverting to follower status if a timeout elapses without a quorum of heartbeat responses.

encoding: binary encoding not preserving lexicographic ordering

From Spencer: Discovered a problem with binary encoding. In removing the terminator
byte suffix, the ordering was ruined. For example:

key1
key1\x00
key2

would actually sort as:

key1
key2
key1\x00

This is because key1\x00 would have a continuation high order bit set.

Database RocksDB

Instead of RocksDB, you could use a database built in Go (to a better deploy):

• Bolt is a pure Go key/value store inspired by the LMDB project. Bolt currently works on Mac OS and Linux. Windows support is coming soon.

https://github.com/boltdb/bolt
https://godoc.org/github.com/boltdb/bolt

• Package kv implements a simple and easy to use persistent key/value (KV) store:

https://github.com/cznic/kv
http://godoc.org/github.com/cznic/kv

A first cut at accounting

Check if phabricator plays nicely with github issues

'make testrace' fails in storage & storage/engine

Time to fix this, #54 and get on with it.

cd /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb; make static_lib
make[1]: Entering directory '/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb'
g++ -g -Wall -Werror -Wsign-compare -I. -I./include -std=c++11  -DROCKSDB_PLATFORM_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_ATOMIC_PRESENT -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=google -DZLIB -DBZIP2   -DHAVE_JEMALLOC -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -c util/build_version.cc -o util/build_version.o 
rm -f librocksdb.a
ar -rs librocksdb.a db/builder.o db/c.o db/column_family.o db/compaction.o db/compaction_picker.o db/db_filesnapshot.o db/dbformat.o db/db_impl.o db/db_impl_debug.o db/db_impl_readonly.o db/db_iter.o db/file_indexer.o db/filename.o db/forward_iterator.o db/internal_stats.o db/log_reader.o db/log_writer.o db/memtable.o db/memtable_list.o db/merge_helper.o db/merge_operator.o db/repair.o db/table_cache.o db/table_properties_collector.o db/tailing_iter.o db/transaction_log_impl.o db/version_edit.o db/version_set.o db/write_batch.o table/adaptive_table_factory.o table/block_based_table_builder.o table/block_based_table_factory.o table/block_based_table_reader.o table/block_builder.o table/block.o table/block_hash_index.o table/block_prefix_index.o table/bloom_block.o table/cuckoo_table_builder.o table/cuckoo_table_reader.o table/filter_block.o table/flush_block_policy.o table/format.o table/iterator.o table/merger.o table/meta_blocks.o table/plain_table_builder.o table/plain_table_factory.o table/plain_table_index.o table/plain_table_key_coding.o table/plain_table_reader.o table/table_properties.o table/two_level_iterator.o util/arena.o util/auto_roll_logger.o util/blob_store.o util/bloom.o util/build_version.o util/cache.o util/coding.o util/comparator.o util/crc32c.o util/dynamic_bloom.o util/env.o util/env_hdfs.o util/env_posix.o util/filter_policy.o util/hash.o util/hash_cuckoo_rep.o util/hash_linklist_rep.o util/hash_skiplist_rep.o util/histogram.o util/iostats_context.o utilities/backupable/backupable_db.o utilities/document/document_db.o utilities/document/json_document.o utilities/geodb/geodb_impl.o utilities/merge_operators/put.o utilities/merge_operators/string_append/stringappend2.o utilities/merge_operators/string_append/stringappend.o utilities/merge_operators/uint64add.o utilities/redis/redis_lists.o utilities/spatialdb/spatial_db.o utilities/ttl/db_ttl_impl.o util/ldb_cmd.o util/ldb_tool.o util/log_buffer.o util/logging.o util/murmurhash.o util/options_builder.o util/options.o util/perf_context.o util/rate_limiter.o util/skiplistrep.o util/slice.o util/statistics.o util/status.o util/string_util.o util/sync_point.o util/thread_local.o util/vectorrep.o util/xxhash.o port/port_posix.o port/stack_trace.o 
make[1]: Leaving directory '/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb'
cd /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto; LDFLAGS="-L/usr/local/lib -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " CFLAGS="-I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb/include -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " CXXFLAGS="-I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb/include -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " make static_lib
make[1]: Entering directory '/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto'
make[1]: Nothing to be done for 'static_lib'.
make[1]: Leaving directory '/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto'
cd /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib; LDFLAGS="-L/usr/local/lib -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " CFLAGS="-I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb/include -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " CXXFLAGS="-I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb/include -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " make static_lib
make[1]: Entering directory '/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib'
make[1]: Nothing to be done for 'static_lib'.
make[1]: Leaving directory '/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib'
cd /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/sql/parser; LDFLAGS="-L/usr/local/lib -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " CFLAGS="-I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb/include -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " CXXFLAGS="-I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb/include -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " make
CGO_LDFLAGS="-L/usr/local/lib -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -L/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " CGO_CFLAGS="-I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb/include -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " CGO_CXXFLAGS="-I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/_vendor/rocksdb/include -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/proto/lib -I/home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/roachlib " go test -race -run ".*" "./..." -logtostderr -timeout 10s
?       github.com/cockroachdb/cockroach    [no test files]
ok      github.com/cockroachdb/cockroach/gossip 1.409s
==================
WARNING: DATA RACE
Read by goroutine 16:
  code.google.com/p/biogo.store/interval.(*Tree).Get()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:221 +0x6b
  github.com/cockroachdb/cockroach/util.(*IntervalCache).get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:371 +0xcf
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:158 +0x74
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Previous write by goroutine 14:
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:391 +0x27a
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Goroutine 16 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).InternalHeartbeatTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:222 +0xca
  github.com/cockroachdb/cockroach/kv.(*coordinator).heartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator.go:238 +0x18d

Goroutine 14 (finished) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).Put()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:95 +0x123
  github.com/cockroachdb/cockroach/kv.TestCoordinatorAddRequest()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:78 +0x34f
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f
==================
==================
WARNING: DATA RACE
Read by goroutine 16:
  github.com/cockroachdb/cockroach/util.(*intervalKey).Start()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:340 +0x4c
  github.com/cockroachdb/cockroach/util.(*intervalKey).Overlap()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:345 +0x3a
  code.google.com/p/biogo.store/interval.(*Tree).Get()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:221 +0xf1
  github.com/cockroachdb/cockroach/util.(*IntervalCache).get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:371 +0xcf
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:158 +0x74
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Previous write by goroutine 14:
  github.com/cockroachdb/cockroach/util.(*intervalKey).SetStart()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:342 +0x3a
  code.google.com/p/biogo.store/interval.(*Node).delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:407 +0x368
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:390 +0x256
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Goroutine 16 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).InternalHeartbeatTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:222 +0xca
  github.com/cockroachdb/cockroach/kv.(*coordinator).heartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator.go:238 +0x18d

Goroutine 14 (finished) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).Put()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:95 +0x123
  github.com/cockroachdb/cockroach/kv.TestCoordinatorAddRequest()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:78 +0x34f
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f
==================
==================
WARNING: DATA RACE
Read by goroutine 16:
  github.com/cockroachdb/cockroach/util.(*intervalKey).End()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:341 +0x4c
  github.com/cockroachdb/cockroach/util.(*intervalKey).Overlap()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:345 +0xc3
  code.google.com/p/biogo.store/interval.(*Tree).Get()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:221 +0xf1
  github.com/cockroachdb/cockroach/util.(*IntervalCache).get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:371 +0xcf
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:158 +0x74
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Previous write by goroutine 14:
  github.com/cockroachdb/cockroach/util.(*intervalKey).SetEnd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:343 +0x3a
  code.google.com/p/biogo.store/interval.(*Node).adjustRange()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:170 +0x610
  code.google.com/p/biogo.store/interval.(*Node).fixUp()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:144 +0x3b
  code.google.com/p/biogo.store/interval.(*Node).delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:433 +0x386
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:390 +0x256
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Goroutine 16 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).InternalHeartbeatTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:222 +0xca
  github.com/cockroachdb/cockroach/kv.(*coordinator).heartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator.go:238 +0x18d

Goroutine 14 (finished) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).Put()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:95 +0x123
  github.com/cockroachdb/cockroach/kv.TestCoordinatorAddRequest()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:78 +0x34f
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f
==================
==================
WARNING: DATA RACE
Read by goroutine 16:
  code.google.com/p/biogo.store/interval.(*Node).doMatch()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:617 +0x3f
  code.google.com/p/biogo.store/interval.(*Tree).Get()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:222 +0x175
  github.com/cockroachdb/cockroach/util.(*IntervalCache).get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:371 +0xcf
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:158 +0x74
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Previous write by goroutine 14:
  code.google.com/p/biogo.store/interval.(*Node).delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:406 +0x28b
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:390 +0x256
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Goroutine 16 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).InternalHeartbeatTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:222 +0xca
  github.com/cockroachdb/cockroach/kv.(*coordinator).heartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator.go:238 +0x18d

Goroutine 14 (finished) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).Put()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:95 +0x123
  github.com/cockroachdb/cockroach/kv.TestCoordinatorAddRequest()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:78 +0x34f
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f
==================
==================
WARNING: DATA RACE
Read by goroutine 16:
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:391 +0x290
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Previous write by goroutine 14:
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:391 +0x2af
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Goroutine 16 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).InternalHeartbeatTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:222 +0xca
  github.com/cockroachdb/cockroach/kv.(*coordinator).heartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator.go:238 +0x18d

Goroutine 14 (finished) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).Put()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:95 +0x123
  github.com/cockroachdb/cockroach/kv.TestCoordinatorAddRequest()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:78 +0x34f
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 27:
  github.com/cockroachdb/cockroach/kv.func·006()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:142 +0x23c
  github.com/cockroachdb/cockroach/util.IsTrueWithin()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/testing.go:82 +0x64
  github.com/cockroachdb/cockroach/kv.TestCoordinatorHeartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:149 +0x457
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f

Previous read by goroutine 36:
  github.com/cockroachdb/cockroach/util/hlc.*ManualClock.UnixNano·fm()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:36 +0x39
  github.com/cockroachdb/cockroach/util/hlc.(*Clock).Update()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/hlc/hlc.go:169 +0x193
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:418 +0x38b
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Goroutine 27 (running) created at:
  testing.RunTests()
      /usr/local/go/src/pkg/testing/testing.go:504 +0xb46
  testing.Main()
      /usr/local/go/src/pkg/testing/testing.go:435 +0xa2
  main.main()
      github.com/cockroachdb/cockroach/kv/_test/_testmain.go:81 +0xdc

Goroutine 36 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).InternalHeartbeatTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:222 +0xca
  github.com/cockroachdb/cockroach/kv.(*coordinator).heartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator.go:251 +0x40f
==================
==================
WARNING: DATA RACE
Write by goroutine 27:
  github.com/cockroachdb/cockroach/storage.(*Store).Close()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:146 +0x195
  github.com/cockroachdb/cockroach/kv.(*LocalKV).Close()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:134 +0xfe
  github.com/cockroachdb/cockroach/kv.(*DB).Close()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:63 +0x7f
  github.com/cockroachdb/cockroach/kv.TestCoordinatorHeartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:153 +0x554
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f

Previous read by goroutine 36:
  github.com/cockroachdb/cockroach/storage.(*Store).GetRange()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:242 +0xb1
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:425 +0x1a2
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:121 +0x4b7
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Goroutine 27 (running) created at:
  testing.RunTests()
      /usr/local/go/src/pkg/testing/testing.go:504 +0xb46
  testing.Main()
      /usr/local/go/src/pkg/testing/testing.go:435 +0xa2
  main.main()
      github.com/cockroachdb/cockroach/kv/_test/_testmain.go:81 +0xdc

Goroutine 36 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).InternalHeartbeatTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:222 +0xca
  github.com/cockroachdb/cockroach/kv.(*coordinator).heartbeat()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator.go:251 +0x40f
==================
==================
WARNING: DATA RACE
Read by goroutine 43:
  github.com/cockroachdb/cockroach/storage.(*Store).LookupRange()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:256 +0x10f
  github.com/cockroachdb/cockroach/kv.(*LocalKV).lookupReplica()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:144 +0x1ac
  github.com/cockroachdb/cockroach/kv.(*LocalKV).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:109 +0x10d
  github.com/cockroachdb/cockroach/kv.(*DB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:70 +0x102

Previous write by goroutine 37:
  github.com/cockroachdb/cockroach/storage.(*Store).Close()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:147 +0x1ba
  github.com/cockroachdb/cockroach/kv.(*LocalKV).Close()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/local_kv.go:134 +0xfe
  github.com/cockroachdb/cockroach/kv.(*DB).Close()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:63 +0x7f
  github.com/cockroachdb/cockroach/kv.TestCoordinatorEndTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:187 +0x56c
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f

Goroutine 43 (running) created at:
  github.com/cockroachdb/cockroach/kv.(*DB).InternalResolveIntent()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/db.go:245 +0xca
  github.com/cockroachdb/cockroach/kv.(*coordinator).EndTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator.go:198 +0x359
  github.com/cockroachdb/cockroach/kv.TestCoordinatorEndTxn()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/kv/coordinator_test.go:181 +0x404
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f

Goroutine 37 (finished) created at:
  testing.RunTests()
      /usr/local/go/src/pkg/testing/testing.go:504 +0xb46
  testing.Main()
      /usr/local/go/src/pkg/testing/testing.go:435 +0xa2
  main.main()
      github.com/cockroachdb/cockroach/kv/_test/_testmain.go:81 +0xdc
==================
I0927 14:56:07.440885 31689 simulation.go:62] simulating gossip network with 3 nodes
I0927 14:56:07.443450 31689 server.go:88] serving on /tmp/unix-socket039167907...
I0927 14:56:07.443531 31689 server.go:88] serving on /tmp/unix-socket413785254...
I0927 14:56:07.443577 31689 server.go:88] serving on /tmp/unix-socket447364045...
I0927 14:56:07.443656 31689 gossip.go:347] bootstrapping gossip protocol using host /tmp/unix-socket413785254
I0927 14:56:07.443904 31689 gossip.go:347] bootstrapping gossip protocol using host /tmp/unix-socket039167907
I0927 14:56:07.444122 31689 gossip.go:347] bootstrapping gossip protocol using host /tmp/unix-socket413785254
I0927 14:56:07.448062 31689 client.go:124] client /tmp/unix-socket039167907 connected
I0927 14:56:07.449830 31689 client.go:187] client /tmp/unix-socket039167907 starting heartbeat
I0927 14:56:07.450346 31689 client.go:124] client /tmp/unix-socket413785254 connected
I0927 14:56:07.451210 31689 client.go:187] client /tmp/unix-socket413785254 starting heartbeat
I0927 14:56:07.452469 31689 server.go:103] gossip: client /tmp/unix-socket039167907 sent 1 info(s)
I0927 14:56:07.453322 31689 server.go:103] gossip: client /tmp/unix-socket447364045 sent 2 info(s)
I0927 14:56:07.455057 31689 server.go:103] gossip: client /tmp/unix-socket413785254 sent 1 info(s)
W0927 14:56:07.455684 31689 gossip.go:420] missing sentinel gossip cluster-id; assuming partition and reconnecting
I0927 14:56:07.455958 31689 server.go:102] done serving on /tmp/unix-socket039167907
I0927 14:56:07.456055 31689 server.go:102] done serving on /tmp/unix-socket413785254
I0927 14:56:07.456128 31689 server.go:102] done serving on /tmp/unix-socket447364045
I0927 14:56:07.456350 31689 simulation.go:62] simulating gossip network with 1 nodes
I0927 14:56:07.467819 31689 server.go:88] serving on /tmp/unix-socket480063176...
I0927 14:56:07.467883 31689 server.go:102] done serving on /tmp/unix-socket480063176
PASS
Found 8 data race(s)
FAIL    github.com/cockroachdb/cockroach/kv 1.111s
ok      github.com/cockroachdb/cockroach/multiraft  1.100s
ok      github.com/cockroachdb/cockroach/proto  1.026s
ok      github.com/cockroachdb/cockroach/rpc    1.879s
ok      github.com/cockroachdb/cockroach/server 2.091s
?       github.com/cockroachdb/cockroach/server/status  [no test files]
?       github.com/cockroachdb/cockroach/simulation [no test files]
ok      github.com/cockroachdb/cockroach/sql/parser 1.134s
==================
WARNING: DATA RACE
Read by goroutine 78:
  github.com/cockroachdb/cockroach/storage.(*blockingEngine).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:244 +0x7e
  github.com/cockroachdb/cockroach/storage/engine.GetProto()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/engine/engine.go:162 +0x8a
  github.com/cockroachdb/cockroach/storage/engine.(*MVCC).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:101 +0x1ab
  github.com/cockroachdb/cockroach/storage.(*Range).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:674 +0xe0
  github.com/cockroachdb/cockroach/storage.(*Range).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:601 +0xf31
  github.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:367 +0x21b
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:307 +0x455
  github.com/cockroachdb/cockroach/storage.func·016()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:500 +0x171

Previous write by goroutine 77:
  github.com/cockroachdb/cockroach/storage.(*blockingEngine).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:245 +0xee
  github.com/cockroachdb/cockroach/storage/engine.GetProto()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/engine/engine.go:162 +0x8a
  github.com/cockroachdb/cockroach/storage/engine.(*MVCC).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:101 +0x1ab
  github.com/cockroachdb/cockroach/storage.(*Range).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:674 +0xe0
  github.com/cockroachdb/cockroach/storage.(*Range).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:601 +0xf31
  github.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:367 +0x21b
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:307 +0x455
  github.com/cockroachdb/cockroach/storage.func·015()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:489 +0x16c

Goroutine 78 (running) created at:
  github.com/cockroachdb/cockroach/storage.TestRangeCommandQueue()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:505 +0x8fd
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f

Goroutine 77 (running) created at:
  github.com/cockroachdb/cockroach/storage.TestRangeCommandQueue()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:494 +0x7aa
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f
==================
==================
WARNING: DATA RACE
Read by goroutine 22:
  sync.raceRead()
      /usr/local/go/src/pkg/sync/race.go:37 +0x35
  sync.(*WaitGroup).Add()
      /usr/local/go/src/pkg/sync/waitgroup.go:60 +0xbe
  github.com/cockroachdb/cockroach/storage.(*blockingEngine).block()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:236 +0xc7
  github.com/cockroachdb/cockroach/storage.TestRangeCommandQueue()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:485 +0x658
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f

Previous write by goroutine 77:
  sync.raceWrite()
      /usr/local/go/src/pkg/sync/race.go:41 +0x35
  sync.(*WaitGroup).Wait()
      /usr/local/go/src/pkg/sync/waitgroup.go:122 +0x176
  github.com/cockroachdb/cockroach/storage.(*blockingEngine).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:246 +0x127
  github.com/cockroachdb/cockroach/storage/engine.GetProto()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/engine/engine.go:162 +0x8a
  github.com/cockroachdb/cockroach/storage/engine.(*MVCC).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:101 +0x1ab
  github.com/cockroachdb/cockroach/storage.(*Range).Get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:674 +0xe0
  github.com/cockroachdb/cockroach/storage.(*Range).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:601 +0xf31
  github.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:367 +0x21b
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:307 +0x455
  github.com/cockroachdb/cockroach/storage.func·015()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:489 +0x16c

Goroutine 22 (running) created at:
  testing.RunTests()
      /usr/local/go/src/pkg/testing/testing.go:504 +0xb46
  testing.Main()
      /usr/local/go/src/pkg/testing/testing.go:435 +0xa2
  main.main()
      github.com/cockroachdb/cockroach/storage/_test/_testmain.go:189 +0xdc

Goroutine 77 (running) created at:
  github.com/cockroachdb/cockroach/storage.TestRangeCommandQueue()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range_test.go:494 +0x7aa
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:422 +0x10f
==================
==================
WARNING: DATA RACE
Read by goroutine 98:
  code.google.com/p/biogo.store/interval.(*Tree).Get()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:221 +0x6b
  github.com/cockroachdb/cockroach/util.(*IntervalCache).get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:371 +0xcf
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:158 +0x74
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Previous write by goroutine 96:
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:391 +0x27a
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Goroutine 98 (running) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194

Goroutine 96 (finished) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194
==================
==================
WARNING: DATA RACE
Read by goroutine 98:
  github.com/cockroachdb/cockroach/util.(*intervalKey).Start()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:340 +0x4c
  github.com/cockroachdb/cockroach/util.(*intervalKey).Overlap()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:345 +0x3a
  code.google.com/p/biogo.store/interval.(*Tree).Get()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:221 +0xf1
  github.com/cockroachdb/cockroach/util.(*IntervalCache).get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:371 +0xcf
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:158 +0x74
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Previous write by goroutine 96:
  github.com/cockroachdb/cockroach/util.(*intervalKey).SetStart()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:342 +0x3a
  code.google.com/p/biogo.store/interval.(*Node).delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:407 +0x368
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:390 +0x256
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Goroutine 98 (running) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194

Goroutine 96 (finished) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194
==================
==================
WARNING: DATA RACE
Read by goroutine 98:
  github.com/cockroachdb/cockroach/util.(*intervalKey).End()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:341 +0x4c
  github.com/cockroachdb/cockroach/util.(*intervalKey).Overlap()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:345 +0xc3
  code.google.com/p/biogo.store/interval.(*Tree).Get()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:221 +0xf1
  github.com/cockroachdb/cockroach/util.(*IntervalCache).get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:371 +0xcf
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:158 +0x74
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Previous write by goroutine 96:
  github.com/cockroachdb/cockroach/util.(*intervalKey).SetEnd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:343 +0x3a
  code.google.com/p/biogo.store/interval.(*Node).adjustRange()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:170 +0x610
  code.google.com/p/biogo.store/interval.(*Node).fixUp()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:144 +0x3b
  code.google.com/p/biogo.store/interval.(*Node).delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:433 +0x386
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:390 +0x256
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Goroutine 98 (running) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194

Goroutine 96 (finished) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194
==================
==================
WARNING: DATA RACE
Read by goroutine 98:
  code.google.com/p/biogo.store/interval.(*Node).doMatch()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:617 +0x3f
  code.google.com/p/biogo.store/interval.(*Tree).Get()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:222 +0x175
  github.com/cockroachdb/cockroach/util.(*IntervalCache).get()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:371 +0xcf
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:158 +0x74
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Previous write by goroutine 96:
  code.google.com/p/biogo.store/interval.(*Node).delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:406 +0x28b
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:390 +0x256
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Goroutine 98 (running) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194

Goroutine 96 (finished) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194
==================
==================
WARNING: DATA RACE
Read by goroutine 98:
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:391 +0x290
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Previous write by goroutine 96:
  code.google.com/p/biogo.store/interval.(*Tree).Delete()
      /home/tobias/Code/Go/src/code.google.com/p/biogo.store/interval/interval.go:391 +0x2af
  github.com/cockroachdb/cockroach/util.(*IntervalCache).del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:386 +0xe9
  github.com/cockroachdb/cockroach/util.(*baseCache).removeElement()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:183 +0xed
  github.com/cockroachdb/cockroach/util.(*baseCache).Del()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/util/cache.go:159 +0x94
  github.com/cockroachdb/cockroach/storage.(*CommandQueue).Remove()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/command_queue.go:125 +0x6d
  github.com/cockroachdb/cockroach/storage.func·010()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:453 +0x343
  github.com/cockroachdb/cockroach/storage.(*Range).addReadWriteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:464 +0xada
  github.com/cockroachdb/cockroach/storage.(*Range).AddCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/range.go:309 +0x4b7
  github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/store.go:429 +0x245
  github.com/cockroachdb/cockroach/storage.(*testDB).executeCmd()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:45 +0x369

Goroutine 98 (running) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194

Goroutine 96 (finished) created at:
  github.com/cockroachdb/cockroach/storage.(*testDB).Increment()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/db_test.go:78 +0xca
  github.com/cockroachdb/cockroach/storage.(*IDAllocator).allocateBlock()
      /home/tobias/Code/Go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:85 +0x194
==================
PASS
Found 7 data race(s)
FAIL    github.com/cockroachdb/cockroach/storage    1.193s
ok      github.com/cockroachdb/cockroach/storage/engine 1.400s
ok      github.com/cockroachdb/cockroach/structured 1.061s
ok      github.com/cockroachdb/cockroach/util   3.650s
ok      github.com/cockroachdb/cockroach/util/encoding  1.010s
ok      github.com/cockroachdb/cockroach/util/hlc   1.010s
?       github.com/cockroachdb/cockroach/util/log   [no test files]
ok      github.com/cockroachdb/cockroach/util/metrics   1.012s

problem with steps listed in CONTRIBUTING.md

The information given in CONTRIBUTING.md regarding getting and building doesn't seem to be working. A git clone of cockroach, then a git clone of rocksdb into _vendor and then running make seems to work. (after installing rocksdb dependencies ofcourse).
Does the file need an update?

Multiraft: snapshot/compaction integration

We need to be able to produce and apply snapshots so we can garbage collect the raft logs. This will work somewhat differently from MemoryStorage: We can synthesize snapshots on-demand in the Storage.Snapshot method instead of tying snapshot creation to an explicit Compact call. We can then advance Storage.FirstIndex whenever we discard old log entries.

REST Kinks

since the REST API will be prominent in our upcoming beta, let's collect some potentially confusing points

• values are base64-encoded by the JSON marshaler
• keys returned by range scans are also base64-encoded
• byte-wise ordering means that "A" < "a" (well, that's just how it is)

Since it doesn't look like we can change anything about the above without a considerable amount of hacks, let's at least document this properly in examples. Speaking of document, I wasn't able to find a synopsis of the REST API. Did I miss it @andybons?

Multiraft: coalesced heartbeats

To reduce the overhead of heartbeats when there are a large number of raft groups, we want to ignore outgoing MsgHeartbeats sent by raft. Instead, we want to send a single heartbeat message for each pair of servers, and then expand those on the receiving side into a heartbeat for each matching group.

API

What's the API for writing a driver for Cockroach? It will be great to write the driver for https://github.com/rvagg/node-levelup.

Multiraft Implementation

One question about maxOffset setting

Hi, in RemoteClockMonitor.MonitorRemoteOffsets() implementation, if node's clock offset goes beyond the maxOffset parameter, this node will Fatalf and suicide.

Parameter maxOffset is set to 250ms by default, any reason for choosing this number?
and as the design goal, Cockroachdb should be geo-distributed, then how to set this number for cross-datacenter deployment if without some infrastructure like Atom/GPS synchronized clock using in Spanner. Does normal NTP work?

Waiting for your clarification..:-)

kv rest api: counter operations shouldn't work on Entry values and vice versa

A guide to setup a singe-node cockroach

After I compiled the cockroach, It made me confused to setup a single-node cockroach.
./cockroach init hdd=/data /data/zone.cfg
What does the default-zone-config look like?
./cockroach start -data _dirs=data -gossip_bootstrap=127.0.0.1:9001
What's the gossip_bootstrap?

I think we need a guide to setup a single-node cockroach for the new guys.

Range splitting

A first cut at management/statusz UI

ProposeRaftCommand panic in testrace (s.raft == nil)

looks like s.raft == nil in this run: https://circleci.com/gh/tschottdorf/cockroach/230.
It's on a dev branch, but since the error is sporadic and races are mentioned in the affected method, I assume this affects master as well.

// ProposeRaftCommand submits a command to raft.
func (s *Store) ProposeRaftCommand(idKey cmdIDKey, cmd proto.InternalRaftCommand) {
    // s.raft should be constant throughout the life of the store, but
    // the race detector reports a race between this method and s.Stop.
    s.mu.RLock()
    defer s.mu.RUnlock()

    s.raft.propose(idKey, cmd) // s.raft == nil here, panic ensues.
}

"ld: symbol(s) not found for architecture x86_64" when I execute make

I follow step by step the CONTRIBUTING.md. All ok but the last step make raises this error:

EDIT: I use mac 10.10.1 yosemite

go build  -i -o cockroach
# github.com/cockroachdb/cockroach/storage/engine
Undefined symbols for architecture x86_64:
  "_LZ4_compressBound", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in librocksdb.a(block_based_table_builder.o)
  "_LZ4_compressHC2_limitedOutput", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in librocksdb.a(block_based_table_builder.o)
  "_LZ4_compress_limitedOutput", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in librocksdb.a(block_based_table_builder.o)
  "_LZ4_decompress_safe_partial", referenced from:
      rocksdb::UncompressBlockContents(char const*, unsigned long, rocksdb::BlockContents*) in librocksdb.a(format.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [build] Error 2

Listen on random ports for tests instead of :8080

Currently the servers that are used when testing the healthz endpoints are using the default port from the flag (8080). These should be randomized via port :0

Relevant error log from failing test:

E0525 17:43:15.273643 05697 range.go:161] failed init accounting configuration map: prefix.go:111: no default prefix specified
--- FAIL: TestHealthz (0.01 seconds)    server_test.go:70: error requesting healthz at http://localhost:8080/_admin/healthz: Get http://localhost:8080/_admin/healthz: dial tcp 127.0.0.1:8080: connection refused
--- FAIL: TestGzip (0.00 seconds)   server_test.go:101: could not make request to http://localhost:8080/_admin/healthz: Get http://localhost:8080/_admin/healthz: dial tcp 127.0.0.1:8080: connection refused
E0525 17:43:15.281753 05697 range.go:161] failed init permissions configuration map: prefix.go:111: no default prefix specified
E0525 17:43:15.281928 05697 range.go:161] failed init zones configuration map: prefix.go:111: no default prefix specifiedadmin REST request failed: zone_cli.go:49: 500 Internal Server Error: zone.go:127: the default zone configuration cannot be deleted

Remove boot2docker hacks from `make acceptance`

http://blog.docker.com/2014/10/docker-1-3-signed-images-process-injection-security-options-mac-shared-directories/

Probably should be able to remove some mac-specific things in there now.

Super low prio, obviously.

make testrace within the Travis build

There are still intermittent race detector panics due to timeout issues during the tests. These should be fixed so that make testrace can be run as part of the Travis build.

build: clean up makefiles

From #35: "But perhaps we should be vendoring protobuf (and snappy). God, Makefiles suck, why is
librocksdb.a being relinked every time I run make?" – @petermattis

get mailing list notifications sorted out

@cockroachdb/developers

SSL for all inter-node comms

from irc:
the cluster should be able to generate SSL certs (invoke openssl I guess) for internode comms as well as act as a certificate authority to generate certs for client access.
the end goal would be to allow people to set up a cockroach cluster without having to understand any of that stuff, but for larger orgs, they can specify configuration more particularly

The rebalance operation