Coder Social home page Coder Social logo

paypal / junodb Goto Github PK

View Code? Open in Web Editor NEW
2.6K 29.0 163.0 19.23 MB

JunoDB is PayPal's home-grown secure, consistent and highly available key-value store providing low, single digit millisecond, latency at any scale.

License: Apache License 2.0

Shell 2.83% Go 38.44% Python 0.63% Makefile 0.07% Dockerfile 0.10% Java 34.44% HTML 20.38% CSS 0.52% JavaScript 0.30% Ruby 2.30%

junodb's Introduction

JunoDB - A secure, consistent and highly available key-value store

License Build Docker

What is JunoDB

JunoDB is PayPal's home-grown secure, consistent and highly available key-value store providing low, single digit millisecond, latency at any scale.

Getting Started with the JunoDB Server

Clone the repository from github

git clone https://github.com/paypal/junodb.git

Set BUILDTOP variable

export BUILDTOP=<path_to_junodb_folder>/junodb
cd $BUILDTOP

Continue building JunoDB server with

  1. Docker build or
  2. Manual build

Docker Build

Note : Docker build supported platforms

  • Linux (Ubuntu)
  • OS X (macOS)

Install Dependencies

Install Docker Engine version 20.10.0+ Check for existing docker version

docker version

Install Docker if not installed or version is older than 20.10.0

docker/setup.sh
#If you are not added to the docker group, you will have to logout and login in the machine after running docker/setup.sh

If the user is not added to docker group, you may add manually. Logout and re-login after this step.

sudo usermod -a -G docker $USER

Verify that your username is added to docker group

groups

#or
cat /etc/group | grep docker

Build JunoDB

#Login to docker hub account 
docker login

# Build junodb docker images
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient


docker/build.sh 

Run JunoDB

# Setup junodb network and start junodb services
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient


# JunoDB proxy service listens on port 
# :5080 TLS and :8080 TCP
docker/start.sh 

Shutdown JunoDB services

# This will shutdown junodb services
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient

docker/shutdown.sh 

Manually Run JunoDB services

#This can be done instead of ./start.sh to start up the docker services

cd $BUILDTOP/docker/manifest

# To run junodb services in --detach mode (recommended)
docker compose up -d

# Juno proxy service listens on port 
# :5080 TLS and :8080 TCP

#To view the running containers 
docker ps

# To stop junodb services
docker compose down

Generate Secrets for Dev


NOTE: secrets for TLS and Encryption can be generated for dev/testing.

sh $BUILDTOP/docker/manifest/config/secrets/gensecrets.sh

## generated secrets
# server.crt/server.pem - certificate/key for junodb proxy for TLS 
# ca.crt - CA cert
# keystore.toml - sample keystore file

Validate JunoDB

Login to docker client and check connection with proxy

docker exec -it junoclient bash -c 'nc -vz proxy 5080'

You can also test the junodb server by running junocli and junoload


JunoCLI

The following commands log in to the docker client and run the ./junocli command directly. The proxy ip is aliased as "proxy"

  1. CREATE
docker exec -it junoclient bash -c '/opt/juno/junocli create -s proxy:8080 -c config.toml -ns test_ns test_key test_value'
  1. GET
docker exec -it junoclient bash -c '/opt/juno/junocli get -s proxy:8080 -c config.toml -ns test_ns test_key'
  1. UPDATE
docker exec -it junoclient bash -c '/opt/juno/junocli update -s proxy:8080 -c config.toml -ns test_ns test_key test_value_updated'
  1. DESTROY
docker exec -it junoclient bash -c '/opt/juno/junocli destroy -s proxy:8080 -c config.toml -ns test_ns test_key'

More about junocli here


Junoload

The following command logs in to the docker client and runs the ./junoload command directly. Junoload can be used for benchmarking juno server. The proxy ip is aliased as "proxy"

docker exec -it junoclient bash -c '/opt/juno/junoload -s proxy:5080 -ssl -c config.toml -o 1'

More about junoload here



Manual Build

Note : Manual build supported platforms

  • Linux (Ubuntu 20.04)

The following sections explain the process for manually building the JunoDB server without Docker. These instructions are based on an Ubuntu 20.04.5 system

Install Dependencies

Install OpenSSL 1.0.2g+

sudo apt install openssl

Install multilog

sudo apt install daemontools

Install dependencies for rocksdb

sudo apt-get install build-essential libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev -y

Install Python
#install python
sudo apt-get install python3.8
#set soft link
cd /usr/bin
sudo ln -s python3.8 python

Build JunoDB

binary_build/build.sh

Run JunoDB

export JUNO_BUILD_DIR=$BUILDTOP/release-binary/code-build
script/deploy.sh

Validate JunoDB

#Validate if deploy was successful by checking if the proxy (junoserv), storage (junostorageserv), and etcd (junoclusterserv) processes are running
ps -eaf | grep juno
#There should be 41 processes running
#5 for junoclusterserv (3 logs, 1 etcdsvr.py, 1 etcdsvr_exe)
#20 for junostorageserv (6 logs, 1 manager, 12 workers, 1 monitor)
#16 for junoserv (6 logs, 1 manager, 8 workers, 1 monitor)

Test out the server using junocli and junoload command

See instructions for junocli here
See instructions for junoload here

Run functional tests

#Assuming user is in $BUILDTOP folder
test/functest/configsetup.sh
cd test/functest
$BUILDTOP/release-binary/tool/go/bin/go test -v -config=config.toml

Run unit tests

#Assuming user is in $BUILDTOP folder
cd test/unittest
$BUILDTOP/release-binary/tool/go/bin/go test -v

Shutdown JunoDB Services

#Assuming user is in $BUILDTOP folder
script/deploy.sh stop

junodb's People

Contributors

ahmadkashif avatar art2ip avatar dependabot[bot] avatar jostanislas avatar juno-yuanyu avatar lalithanatraj99 avatar neetishpathak avatar nit-tripathi avatar varuntechie avatar venkatsridhar95 avatar verapaypal avatar yapingshi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

junodb's Issues

Implement Python Client for JunoDB

I think JunoDB can really use a Python client.
Something simple to start with that implements most of the basic APIs and then extends with more advanced use cases.

The first implementation could be synchronous and later async methods or mode could also be added.

Why is keyrange deletion operation present twice?

In pkg/etcd/etcdwriter.go I can see that we are putting a deletion operation twice with the same key range, is this an intended behaviour or missed refactor?

for zoneid := 0; zoneid < int(c.NumZones); zoneid++ {
if !c.IsRedistZone(zoneid) {
continue
}
beginKey := KeyNodeIpport(zoneid, int(c.Zones[zoneid].NumNodes))
endKey := KeyNodeIpport(zoneid+1, 0)
op.AddDeleteWithRange(beginKey, endKey)
}
for zoneid := 0; zoneid < int(c.NumZones); zoneid++ {
if !c.IsRedistZone(zoneid) {
continue
}
beginKey := KeyNodeShards(zoneid, int(c.Zones[zoneid].NumNodes))
endKey := KeyNodeShards(zoneid+1, 0)
op.AddDeleteWithRange(beginKey, endKey)
}

PS : Apologies If this does not follow standard issue raising templates, couldn't find one in developer guidelines.

Validate the Docker image in Docker Latest version 23.0.6

What I Did
I found Docker latest version was release 23.0.6

What I Expected
To be able to seamlessly run the JunoDB in the latest Docker

What I Saw Instead
Once JunoDB is tested against latest Docker version (23.0.6), update the ReadMe file accordingly

Refactors `EtcdReader.readNodesShards` method for More Efficient Zone Initialization and Node Shards Assignment

Issue Overview:

In the etcd package and file etcdreader.go, the readNodesShards method is responsible for reading Nodes Shards assignment through the etcd reader. During this process, it calls cluster.NewZoneFromConfig to initialize zones if c.Zones[zoneid] == nil. However, there is an opportunity to improve efficiency and avoid redundancy in zone and its nodes initialization. It's important to note that cluster.NewZoneFromConfig not only initializes the zone but also executes zone.initShardsAsssignment(numZones, numShards) to populate nodes. This behavior conflicts with the logic in readNodesShards, which manually overrides node data using c.Zones[zoneid].Nodes[nodeid].StringToNode.

Proposed Solution:

To enhance code efficiency and eliminate redundancy, we propose refactoring the readNodesShards function. Specifically, we suggest replacing the usage of cluster.NewZoneFromConfig with the new cluster.NewZone function. This new function will initialize zones without populating the Nodes field, leaving that task to be performed later in the code when it is needed. By making this change, we can avoid overwriting node data during zone initialization and improve overall efficiency.

Impact:

This change will optimize the readNodesShards function and reduce redundant population of the Nodes field during zone initialization. It is expected to have a positive impact on performance. This optimization can lead to more efficient resource utilization and better overall system performance.

Related Code:

Here's the proposed change to the readNodesShards function:

// Replace this line
c.Zones[zoneid] = cluster.NewZoneFromConfig(uint32(zoneid), uint32(nodeid+1), c.NumZones, c.NumShards)

// With this line
c.Zones[zoneid] = cluster.NewZone(uint32(zoneid), uint32(nodeid+1))

And also in cluster package

// NewZone creates a new zone with the specified attributes.
func NewZone(zoneid uint32, numNodes uint32) *Zone {
	zone := newZone(zoneid, numNodes)
	return &zone
}

// newZone initializes and returns a new zone with the specified attributes.
func newZone(zoneid uint32, numNodes uint32) Zone {
	return Zone{
		Zoneid:   zoneid,
		NumNodes: numNodes,
		Nodes:    make([]Node, 1, numNodes),
	}
}

Additional Context:

This change is suggested as part of ongoing efforts to optimize and improve the codebase of the Juno project. It aims to make zone initialization and Nodes Shards assignment more efficient while maintaining code correctness.

LUA Client for junodb

Giving LUA client for junodb will help in proxy level code to store data and retrieve when needed

Persistency

Hey, just wondering how Juno is persistent on disk? Thanks

Client libraries?

I can see in the documentation that junicli can be used to query against the db, but I don't see any sdks or packages for any language that can integrated with applications.
Is this not available now?
I'm specifically looking for a c# client library.

Improve unit tests

I see there is test/unittest directory, but seems that all tests there are related to SS. I also take a look pkg and cmd directories, seems most packages in those directories have no test or very little tests? As a database I think there should be more unit tests.

Consistency level

What consistency level do you implement? For linearizable read and writes based on a quorum logic, reads must do write repair and writes must do pre-read. Your architecture docs are a bit sparse on this topic. Also, for distributed locking you need atomic-compare-and-swap which can't be implemented by a quorum based architecture as this requires consensus. As the article mentions Junodb is used for distributed locking, I wonder if the architecture description of Juno is missing important pieces.

Issue with Docker Build on MACOS

Docker build is erroring out with the following error:

" **_****> [builder 3/3] RUN ./build.sh:
#0 0.099 /juno
#0 0.102 qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

Dockerfile:35

33 | COPY /build.sh /build.sh
34 |
35 | >>> RUN ./build.sh
36 |
37 |

ERROR: failed to solve: process "/bin/sh -c ./build.sh" did not complete successfully: exit code: 255"
make: *** [build] Error 1****_**

Please suggest any resolution. Docker version is 24.0.2.

Thanks
Abhishek Mittal

Detailed log below

cd /Users/I589015/go-dev-juno/junodb/docker/build/src/juno; echo git rev-parse --short=8 HEAD 2> /dev/null >> /Users/I589015/go-dev-juno/junodb/docker/build/src/juno/git_revision.txt ;
if [ -f /Users/I589015/go-dev-juno/junodb/docker/build/src/juno/third_party/apply_patch.sh ]; then /Users/I589015/go-dev-juno/junodb/docker/build/src/juno/third_party/apply_patch.sh ; fi ;
Cloning into 'rocksdb'...
remote: Enumerating objects: 122556, done.
remote: Counting objects: 100% (607/607), done.
remote: Compressing objects: 100% (400/400), done.
remote: Total 122556 (delta 292), reused 376 (delta 172), pack-reused 121949
Receiving objects: 100% (122556/122556), 201.69 MiB | 300.00 KiB/s, done.
Resolving deltas: 100% (93758/93758), done.
Note: switching to '72cf57d4b40e5c809839afc1c62dc13e533b8ec6'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Cloning into 'forked/tecbot/gorocksdb'...
remote: Enumerating objects: 928, done.
remote: Total 928 (delta 0), reused 0 (delta 0), pack-reused 928
Receiving objects: 100% (928/928), 379.25 KiB | 1016.00 KiB/s, done.
Resolving deltas: 100% (573/573), done.
Note: switching to '57a309fefefb9c03d6dcc11a0e5705fc4711b46d'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 57a309f Merge pull request #106 from JelteF/patch-1
wget --no-check-certificate https://dl.google.com/go/go1.18.2.linux-amd64.tar.gz -O /Users/I589015/go-dev-juno/junodb/docker/build/go1.18.2.linux-amd64.tar.gz ;
--2023-07-09 19:50:22-- https://dl.google.com/go/go1.18.2.linux-amd64.tar.gz
Resolving dl.google.com (dl.google.com)... 142.250.71.14
Connecting to dl.google.com (dl.google.com)|142.250.71.14|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 141732542 (135M) [application/x-gzip]
Saving to: ‘/Users/I589015/go-dev-juno/junodb/docker/build/go1.18.2.linux-amd64.tar.gz’

/Users/I589015/go-dev-juno/junodb/docke 100%[===============================================================================>] 135.17M 11.0MB/s in 13s

2023-07-09 19:50:35 (10.8 MB/s) - ‘/Users/I589015/go-dev-juno/junodb/docker/build/go1.18.2.linux-amd64.tar.gz’ saved [141732542/141732542]

TODO --

cd build && docker build --build-arg GOLANG_VERSION=1.18.2 --tag
ghcr.io/paypal/junodb/juno-dev:latest --target juno-dev .
[+] Building 4.3s (15/15) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 858B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:20.04 3.4s
=> [auth] library/ubuntu:pull token for registry-1.docker.io 0.0s
=> [juno-base 1/4] FROM docker.io/library/ubuntu:20.04@sha256:c9820a44b950956a790c354700c1166a7ec648bc0d215fa438d3a339812f1d01 0.0s
=> [internal] load build context 0.8s
=> => transferring context: 13.63MB 0.8s
=> CACHED [juno-base 2/4] RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y cmake g++ make wget 0.0s
=> CACHED [juno-base 3/4] COPY /go1.18.2.linux-amd64.tar.gz / 0.0s
=> CACHED [juno-base 4/4] RUN tar xzvf go1.18.2.linux-amd64.tar.gz -C /usr/local/ 0.0s
=> CACHED [juno-dev-base 1/4] COPY /src/juno/third_party/rocksdb /vendor/src/rocksdb 0.0s
=> CACHED [juno-dev-base 2/4] WORKDIR /vendor/src/rocksdb 0.0s
=> CACHED [juno-dev-base 3/4] RUN make static_lib; make INSTALL_PATH=/vendor/rocksdb install-static 0.0s
=> CACHED [juno-dev 1/2] COPY --from=juno-dev-base /vendor/rocksdb/include /usr/local/include 0.0s
=> CACHED [juno-dev 2/2] COPY --from=juno-dev-base /vendor/rocksdb/lib /usr/local/lib 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:3c85febcc62fd66fd243fbb12a22a1123ae9ddd88a7cb5546f68da9452891a83 0.0s
=> => naming to ghcr.io/paypal/junodb/juno-dev:latest 0.0s
cd build && docker build --build-arg GOLANG_VERSION=1.18.2 --tag
ghcr.io/paypal/junodb/juno-build:latest --target builder .
[+] Building 1.9s (16/16) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 858B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:20.04 0.5s
=> [juno-base 1/4] FROM docker.io/library/ubuntu:20.04@sha256:c9820a44b950956a790c354700c1166a7ec648bc0d215fa438d3a339812f1d01 0.0s
=> [internal] load build context 0.9s
=> => transferring context: 63.02MB 0.9s
=> CACHED [juno-base 2/4] RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y cmake g++ make wget 0.0s
=> CACHED [juno-base 3/4] COPY /go1.18.2.linux-amd64.tar.gz / 0.0s
=> CACHED [juno-base 4/4] RUN tar xzvf go1.18.2.linux-amd64.tar.gz -C /usr/local/ 0.0s
=> CACHED [juno-dev-base 1/4] COPY /src/juno/third_party/rocksdb /vendor/src/rocksdb 0.0s
=> CACHED [juno-dev-base 2/4] WORKDIR /vendor/src/rocksdb 0.0s
=> CACHED [juno-dev-base 3/4] RUN make static_lib; make INSTALL_PATH=/vendor/rocksdb install-static 0.0s
=> CACHED [juno-dev 1/2] COPY --from=juno-dev-base /vendor/rocksdb/include /usr/local/include 0.0s
=> CACHED [juno-dev 2/2] COPY --from=juno-dev-base /vendor/rocksdb/lib /usr/local/lib 0.0s
=> [builder 1/3] COPY /src/juno /juno 0.3s
=> [builder 2/3] COPY /build.sh /build.sh 0.0s
=> ERROR [builder 3/3] RUN ./build.sh 0.1s

[builder 3/3] RUN ./build.sh:
#0 0.099 /juno
#0 0.102 qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory


Dockerfile:35

33 | COPY /build.sh /build.sh
34 |
35 | >>> RUN ./build.sh
36 |
37 |

ERROR: failed to solve: process "/bin/sh -c ./build.sh" did not complete successfully: exit code: 255
make: *** [build] Error 1

Feature Request: TimeSeries Database & Feature modularization with libraries

Hi,

Two request

  1. Like redis database, if we can provide capability to store time series values and options to retrieve as time grouped , that will be awesome
  2. Also looking at redis features are build separately from main and can be added during run time like

redis-server --loadmodule /redis/modules/redistimeseries.so

Giving such option gives lot of scopes to add new features easily

Thanks
SUR5AN

Facing Issue while building from docker

` => CANCELED [internal] load build context 0.0s
=> => transferring context: 0.0s
=> CACHED [juno-build 1/1] FROM ghcr.io/paypal/junodb/juno-build:latest 0.0s
=> CACHED [juno-runtime-base 2/3] RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -yqq daemontools curl dnsutils ed less netcat net-tools sysstat iputils-ping tzdata && apt-get auto 0.0s
=> CACHED [juno-runtime-base 3/3] RUN groupadd -r juno --gid=1000 && useradd -r -g juno --uid=1000 juno 0.0s
=> ERROR [junoclustercfg 1/5] COPY --from=juno-build /juno/bin/clusterctl /juno/bin/junocfg /juno/bin/junoctl /juno/cmd/etcdsvr/etcdctl /opt/juno/ 0.0s

[junoclustercfg 1/5] COPY --from=juno-build /juno/bin/clusterctl /juno/bin/junocfg /juno/bin/junoctl /juno/cmd/etcdsvr/etcdctl /opt/juno/:


Dockerfile:53

52 |
53 | >>> COPY --from=juno-build
54 | >>> /juno/bin/clusterctl
55 | >>> /juno/bin/junocfg
56 | >>> /juno/bin/junoctl
57 | >>> /juno/cmd/etcdsvr/etcdctl
58 | >>> /opt/juno/
59 |

ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref moby::128g9rtlb7qrgrase1wz73do8: "/juno/bin/clusterctl": not found
make: *** [build_junoclustercfg] Error 1`

anyone faced similar issue ?

Unable to build junoDB using docker

I was following the steps mentioned in the README.md file to build junoDB using docker. But the build.sh threw an error:

Screenshot 2023-06-05 at 5 52 06 PM

This error originates from Makefile located at path ~/docker/Makefile at line number 58 while creating a build for junoclustercfg.

Upon further investigation, it looks like some dependency folder is missing.

  1. Build command uses dockerfile located at ~/docker/cluster-service-build/Dockerfile.
image
  1. For building a docker image for junoclustercfg it copies files from /juno/bin folder from juno-build. As shown below
image

But when juno-build is investigated it turns out there is no folder bin located at /juno

  1. This can be traced back to Dockerfile (located at ~/docker/build/Docker)used to build juno-build . It copies /src/juno to /juno folder. So /src/juno must contains bin folder. But it isn't there.
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.