Coder Social home page Coder Social logo

kubernetes-sigs / apiserver-builder-alpha Goto Github PK

View Code? Open in Web Editor NEW
766.0 30.0 234.0 141.24 MB

apiserver-builder-alpha implements libraries and tools to quickly and easily build Kubernetes apiservers/controllers to support custom resource types based on APIServer Aggregation

License: Apache License 2.0

Go 95.93% Makefile 3.44% Shell 0.14% Dockerfile 0.24% Smarty 0.24%
apiserver-aggregation kubernetes server-sdk k8s-sig-api-machinery

apiserver-builder-alpha's Introduction

apiserver-builder-alpha

build Go Report Card

Unless you absolutely need apiserver-aggregation, you are recommended to use Kubebuilder instead of apiserver-builder for building Kubernetes APIs. Kubebuilder builds APIs using CRDs and addresses limitations and feedback from apiserver-builder.

Apiserver Builder is a collection of libraries and tools to build native Kubernetes extensions using Kubernetes apiserver aggregation. Aggregated apiserver empowers you to customize your apiserver to do following things cannot achieved by CR[D]:

  • Makes your apiserver adopt different storage APIs rather than ETCDv3
  • Extends long-running subresources/endpoints like websocket for your own resources
  • Integrates your apiserver with whatever other external systems

DEMO

Installation

go install sigs.k8s.io/apiserver-builder-alpha/cmd/[email protected]

or manual download from release pages.

Motivation

Addon apiservers are a Kubernetes extension point allowing fully featured Kubernetes APIs to be developed on the same api-machinery used to build the core Kubernetes APIS, but with the flexibility of being distributed and installed separately from the Kubernetes project. This allows APIs to be developed outside of the Kubernetes repo and installed separately as a package.

Building addon apiservers directly on the raw api-machinery libraries requires non-trivial code that must be maintained and rebased as the raw libraries change. The goal of this project is to make building apiservers in Go simple and accessible to everyone in the Kubernetes community.

apiserver-builder provides libraries, code generators, and tooling to make it possible to build and run a basic apiserver in an afternoon, while providing all of the hooks to offer the same capabilities when building from scratch.

Highlights

  • Tools to bootstrap type definitions, controllers (powered by controller-runtime framework), tests and documentation for new resources
  • Tools to build and run the extension control plane standalone and in minikube and remote clusters.
  • Easily watch and update Kubernetes API types from your controller
  • Easily add new resources and subresources
  • Provides sane defaults for most values, but can be overridden

Examples

  • BasicExample: Various simple resource examples.
  • KineExample: Plumbs aggregated apiserver over SQL-storages including sqlite, mysql, etc..
  • PodLogsExample: Serves pod/logs in aggregation layer to offload kube-apiserver connections.
  • PodExecExample: Serves pod/exec in aggregation layer to offload kube-apiserver connections.

Guides

Note: The guides are presented roughly in the order of recommended progression.

Building APIs concept guide

Conceptual information on how APIs and the Kubernetes control plane is structure and how to build new API extensions using apiserver-builder.

If you want to get straight to building something without knowing all the details of what is going on, skip ahead to the tools guide and come back to this later.

api building concept guide

Tools user guide

Instructions on how to use the tools packaged with apiserver-builder to build and run a new apiserver.

tools guide

Step by step example

List of commits showing apiserver-boot commands run and the corresponding changes:

https://github.com/kubernetes-sigs/apiserver-builder-alpha/commits/example-simple

Coding and libraries user guide

Instructions for how to implement custom APIs on top of the apiserver-builder libraries.

libraries guide

Concept guides

Conceptual information on addon apiservers, such as how auth works and how they interact with the main Kubernetes API server and API aggregator.

Concepts

Additional material

Using delegated auth with minikube

Instructions on how to run an apiserver using delegated auth with a minikube cluster

Details here

Community, discussion, contribution, and support

Learn how to engage with the Kubernetes community on the community page.

You can reach the maintainers of this project at:

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

apiserver-builder-alpha's People

Contributors

alexandercampbell avatar avagin avatar aylei avatar calebamiles avatar carlory avatar clarkmcc avatar cloudzp avatar cofyc avatar criahu avatar directxman12 avatar drewwells avatar ibazhitov avatar interma avatar jimmidyson avatar k8s-ci-robot avatar louyihua avatar lovebaby979 avatar metmajer avatar mooncak avatar pjarominsap avatar pwittrock avatar rivencxl avatar tamalsaha avatar tanin47 avatar tossmilestone avatar vitaliyshevchenko avatar wanlinghao avatar weixiao-huang avatar xiang90 avatar yue9944882 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

apiserver-builder-alpha's Issues

support custom registry

It seems that there is no way to provide custom registry. We current use generic registry by default.

can it be customized? probably we should provide a doc to show how to do it.

Provide simple way to get all Kubernetes libs vendored

User story: As a Kubernetes contributor, I want to be able to create a new project with the exact set of vendored libraries used by kubernetes/kubernetes at a given release and have those vendored libraries be managed by another tool.

Background: Go vendoring is hard. Most vendoring tools automatically try to pick the set of transitive dependencies from the direct dependencies, but they often have unpredictable results. Upgrading the transitive set of dependencies is challenging, and it is not at all obvious to the user if the correct things are happening.

We should come up with an official solution for what is provided by apiserver-boot init vendor

Some things to consider:

  • Do we want to prepackage the deps or use a tool to fetch them?
  • Do we want to use Bazel to bootstrap the vendor much like can be done for gazelle (I think this could have a lot of potential)?
    • Provide bazel rule which defines a Kubernetes release version and then set the vendored code to that version
  • How can we make it play nice with glide and dep without adding complexity?

Generated docs have different node_modules

I mentioned this in kops, but it's probably better to track it here.

After generating the API docs, there are some changes in the node_modules.

        modified:   docs/apireference/build/node_modules/ejs/Jakefile
        modified:   docs/apireference/build/node_modules/ejs/README.md
        modified:   docs/apireference/build/node_modules/ejs/ejs.js
        modified:   docs/apireference/build/node_modules/ejs/ejs.min.js
        modified:   docs/apireference/build/node_modules/ejs/lib/ejs.js
        modified:   docs/apireference/build/node_modules/ejs/lib/utils.js
        modified:   docs/apireference/build/node_modules/ejs/package.json
        modified:   docs/apireference/build/node_modules/ejs/test/ejs.js
        modified:   docs/apireference/build/node_modules/highlight.js/docs/css-classes-reference.rst
        modified:   docs/apireference/build/node_modules/highlight.js/lib/highlight.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/index.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/cpp.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/gauss.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/kotlin.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/lua.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/python.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/rust.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/scheme.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/typescript.js
        modified:   docs/apireference/build/node_modules/highlight.js/lib/languages/yaml.js
        modified:   docs/apireference/build/node_modules/highlight.js/package.json
        modified:   docs/apireference/build/node_modules/jquery/AUTHORS.txt
        modified:   docs/apireference/build/node_modules/jquery/LICENSE.txt
        modified:   docs/apireference/build/node_modules/jquery/README.md
        modified:   docs/apireference/build/node_modules/jquery/dist/core.js
        modified:   docs/apireference/build/node_modules/jquery/dist/jquery.js
        modified:   docs/apireference/build/node_modules/jquery/dist/jquery.min.js
        modified:   docs/apireference/build/node_modules/jquery/dist/jquery.min.map
        modified:   docs/apireference/build/node_modules/jquery/dist/jquery.slim.js
        modified:   docs/apireference/build/node_modules/jquery/dist/jquery.slim.min.js
        modified:   docs/apireference/build/node_modules/jquery/dist/jquery.slim.min.map
        modified:   docs/apireference/build/node_modules/jquery/package.json
        modified:   docs/apireference/build/node_modules/jquery/src/.eslintrc.json
        modified:   docs/apireference/build/node_modules/jquery/src/attributes/attr.js
        modified:   docs/apireference/build/node_modules/jquery/src/attributes/val.js
        modified:   docs/apireference/build/node_modules/jquery/src/callbacks.js
        modified:   docs/apireference/build/node_modules/jquery/src/core.js
        modified:   docs/apireference/build/node_modules/jquery/src/core/init.js
        modified:   docs/apireference/build/node_modules/jquery/src/core/ready-no-deferred.js
        modified:   docs/apireference/build/node_modules/jquery/src/core/ready.js
        modified:   docs/apireference/build/node_modules/jquery/src/css.js
        modified:   docs/apireference/build/node_modules/jquery/src/css/curCSS.js
        modified:   docs/apireference/build/node_modules/jquery/src/data/Data.js
        modified:   docs/apireference/build/node_modules/jquery/src/deferred.js
        modified:   docs/apireference/build/node_modules/jquery/src/deprecated.js
        modified:   docs/apireference/build/node_modules/jquery/src/effects.js
        modified:   docs/apireference/build/node_modules/jquery/src/event.js
        modified:   docs/apireference/build/node_modules/jquery/src/manipulation.js
        modified:   docs/apireference/build/node_modules/jquery/src/manipulation/getAll.js
        modified:   docs/apireference/build/node_modules/jquery/src/offset.js
        modified:   docs/apireference/build/node_modules/jquery/src/queue.js
        modified:   docs/apireference/build/node_modules/jquery/src/serialize.js
        modified:   docs/apireference/build/node_modules/jquery/src/traversing.js

It just looks like different versions - I have older versions than the docs you generated:

ejs:

...
-  "version": "2.5.6"
+  "version": "2.5.5"

highlightjs:

...
-  "version": "9.10.0"
+  "version": "9.9.0"

jquery:

...
-  "version": "3.2.1"
+  "version": "3.1.1"

Support for creating subresources

We should support apiserver-boot create subresource <name> --group --version --kind to create the bootstrap code for a subresource.

.tar.gz archives created by apiserver-builder-release have wrong architecture

I am working on the creation of .deb and .rpm packages for the apiserver-builder project in #150. To have the project built via make build, I am using the following approach in the Makefile:

.PHONY: build
build: clean ## Create release artefacts for darwin:amd64, linux:amd64 and windows:amd64. Requires etcd, glide, hg.
	go run ./cmd/apiserver-builder-release/main.go vendor --version $(VERSION)
	go run ./cmd/apiserver-builder-release/main.go build --version $(VERSION)

apiserver-builder-release uses linux:amd64, darwin:amd64 and windows:amd64 as default targets in main.go, therefore the use of --targets is omitted here. Now, running make build creates the following build artefacts in .:

-rw-r--r--   1 metmajer  staff  42244411 Sep 22 22:04 apiserver-builder-0.1-alpha.14-darwin-amd64.tar.gz
-rw-r--r--   1 metmajer  staff  42244411 Sep 22 22:03 apiserver-builder-0.1-alpha.14-linux-amd64.tar.gz
-rw-r--r--   1 metmajer  staff  42244411 Sep 22 22:04 apiserver-builder-0.1-alpha.14-windows-amd64.tar.gz

As it turns out, all archives have the same size and are built for Darwin:

metmajer$ tar -xzf apiserver-builder-0.1-alpha.14-linux-amd64.tar.gz -C /tmp
metmajer$ file /tmp/bin/apiregister-gen
apiregister-gen: Mach-O 64-bit executable x86_64

Anything I'm doing wrong here?

Unexpected invocation of NewXxxKindREST() in controller during package v1.init()

I've spotted that when defining a custom REST storage, its constructor is being called both from apiserver and controller-manager binaries (panic() is inserted manually in the code):

$ bin/controller-manager --kubeconfig=kubeconfig
panic: panic

goroutine 1 [running]:
test/pkg/handler.NewHandler(0xc42078a380, 0xc42068f6b0, 0xc42078a380, 0xc42068f6b0, 0x0)
        /go/src/test/pkg/handler/handler.go:56 +0x99
test/pkg/apis/apps/v1.NewApplicationREST(0x3c95540, 0xc4207fe640)
        /go/src/test/pkg/apis/apps/v1/application_rest.go:203 +0xc0
test/pkg/apis/apps/v1.init()
        /go/src/test/pkg/apis/apps/v1/zz_generated.api.register.go:49 +0x1eb
test/pkg/controller/application.init()
        /go/src/test/pkg/controller/application/zz_generated.api.register.go:99 +0x5d
test/pkg/controller.init()
        /go/src/test/pkg/controller/zz_generated.api.register.go:41 +0x44
main.init()
        /go/src/test/cmd/controller/main.go:31 +0x5a```

This happens because NewXxxRest() is called during package-level variable initialization in v1/zz_generated.api.register.go:

        appsApplicationStorage = builders.NewApiResourceWithStorage( // Resource status endpoint
                apps.InternalApplication,
                ApplicationSchemeFns{},
                func() runtime.Object { return &Application{} },     // Register versioned resource
                func() runtime.Object { return &ApplicationList{} }, // Register versioned resource list
                NewApplicationREST(),
        )

NewXxxREST() constructor may contain some other logic besides simply initializing the custom REST registry type. For example it may establish a connection to some service. And we'll end up with an extra unused connection to the external service, established from the controller manager.

I think we should either fix this or explicitly state (in the documentation and/or in the generated code) that NewXxxREST() should only construct new empty instance of XxxREST without performing any heavy initialization.

Imported package "pkg/apis/apps/install" is not found

The generated code references an inexisting package "pkg/apis/apps/install".
Reproduced with v0.1-beta.11, but the same holds for master.

$ apiserver-boot init --domain example.com
$ apiserver-boot create-resource --domain example.com --group apps --version v1 --kind Application
$ apiserver-boot generate
$ apiserver-boot build
$ go test ./pkg/...
pkg/client/clientset_generated/internalclientset/scheme/register.go:5:2: cannot find package "github.com/ibazhitov/xxx/pkg/apis/apps/install" in any of:
/go/src/github.com/ibazhitov/xxx/vendor/github.com/ibazhitov/xxx/pkg/apis/apps/install (vendor tree)
/usr/lib/go-1.8/src/github.com/ibazhitov/xxx/pkg/apis/apps/install (from $GOROOT)
/home/clm/wrk/go/src/github.com/ibazhitov/xxx/pkg/apis/apps/install (from $GOPATH)

Const in Types not deep copied

Tested with:

apiserver-boot init repo --domain mycorp.io
apiserver-boot create group version resource --group testing --version v1alpha1 --kind MyTestKind

Then edited mytestkind_types.go

type CustomType string

const (
	CUSTOM1 CustomType = "test1"
	CUSTOM2 CustomType = "test2"
)

// MyTestKindSpec defines the desired state of MyTestKind
type MyTestKindSpec struct {
	Custom CustomType
}
apiserver-boot build executables
go build -o bin/apiserver cmd/apiserver/main.go
pkg/apis/testing/zz_generated.api.register.go:79:9: undefined: CustomType

more user friendly tag format

From @pwittrock

I am thinking we will want to change the format of the tag to be more user friendly.
e.g.

// +resource=name:foo,rest:FooRESTImpl
And the same for subsourcee

// +subresource=name:scale,request:Scale,rest:ScaleUniversityREST

Cannot implements rest.GetterWithOptions in custom REST storage

Actually, starting the apiserver with a REST resource implementing GetterWithOptions with NewGetOptions returning a type registered in the scheme of the REST resource model will fail with the following error :
error in registering resource: {resourcename}, no kind {GetOptionKind} is registered for version "v1"

This is caused by the fact that when constructing the APIGroupInfo ( in pkg/builders/api_group_builder#Build), the field OptionsExternalVersion is not populated, and therefore set to the default that's defined in k8s.io/apiserver/pkg/server/genericapiserver#NewDefaultAPIGroupInfo : &schema.GroupVersion{Version: "v1"}, which does not contains our {GetOptionKind}

the endpoint installer is then using this default groupversion when trying to resolve the type of get custom get options, resulting in given error.

This is quickly fixed by adding a single line :

i.OptionsExternalVersion = &i.GroupMeta.GroupVersion

after the line

i.GroupMeta.GroupVersion = g.Versions[0].GroupVersion

in pkg/builders/api_group_builder#Build

Generate and register new storage backend

This is a feature request.

Right now, StartApiServer flow is highly opinionated about a few things like the storage backend to be used (enforces etcd) or the options passed onto the apiserver. While this allows the apiserver to work out-of-the box rather easily, it limits more advanced scenarios, like implementing your own storage layer and easily point to it.

I suggest implementing a generator for a new storage layer, i.e. apiserver-boot create storage --name=xyz that generates stub code for xyz in pkg/storage/xyz and some code in cmd/apiserver that points out how it can be configured and enabled.

Add support for arrays of types.

If in your resource definition you define a type:

type Foo struct {
val1 string "json... val2 string "json...
}
and then use it say from the Spec

type SomeSpec struct {
Foos []Foo
}

correct code on the unversioned type does not get generated. Work around is to use only basic array types and wrap the array of Foos as a struct and then use only basic arrays there.

Additional fields for apiserver.yam (imagePullSecrets and serviceAccount)

Hi, I have created pull request, below is description why I would like to have those changes:

Currently I'm unable to generate and deploy api server with just one command, because I'm using secured private docker registry and k8s cluster with RBAC enabled.

What solves problem for me is editing apiserver.yaml and adding ImagePullSecrets and ServiceAccount to apiserver deployment. But doing this manually is cumbersome.

I would like to run it on one command like this (and this pull request enables it):

apiserver-boot run in-cluster --name group_name --namespace default --image private_repo/IMAGE --image-pull-secrets private_repo_secret --service-account service_account_with_rights

Currently I need to run:

    apiserver-boot build container --image $IMAGE_NAME
    docker push $IMAGE_NAME
    rm config/apiserver.yaml
    apiserver-boot build config --name remoteenvsgroup --namespace default --image $IMAGE_NAME

and then edit apiserver.yaml and run kubectl apply.

Also I have seen controller-secret flag but it is not used and I'm not sure if it place holder for image-pull-secrets functionality.

Support validation and defaulting impl living in a separate package.

We should support validation and defaulting living in a separate package from the resource type definition using a comment-directive e.g.

// +resource:validationpackage:

As part of this, some of the registration code probably needs to be moved around. The end goal is to minimize the number of transitive dependencies when depending on the _types.go package.

blank fields in "openssl req" subj argument will fail the command for older openssl versions.

It looks order openssl versions will throw errors if some fields in the subj arguments are blank.

https://github.com/kubernetes-incubator/apiserver-builder/blob/master/cmd/apiserver-boot/boot/build/build_resource_config.go#L182
https://github.com/kubernetes-incubator/apiserver-builder/blob/master/cmd/apiserver-boot/boot/build/build_resource_config.go#L196

The issue, minio/minio#3903, in another project suggests a solution.
I have verified -subj /C=un/ST=st/L=l/O=o/OU=ou/CN=%s-certificate-authority works.

Support generating protobuf files

We have a few cases where people wanted to create CRDs from Java code. I suppose having a proto file will make it possible to generate Java clients. Can this be supported?

Support for creating Initializers

We should have a apiserver-boot create initializer <name> --group <group> --version <version> --kind <kind> command to bootstrap an initializer for a given resource.

"No client certificate CA names sent"

Loving this project btw!

One problem I'm having is reusing the client certificate that minikube uses ootb. I don't think the API server is properly requesting client CA names for optional client certificates, even if client-ca-file flag is used.

Here's the command I'm using to start the API server (which works when using the bearer token provided btw):

$ ./apiserver --authentication-kubeconfig ~/.kube/auth_config \
  --authorization-kubeconfig ~/.kube/auth_config \
  --client-ca-file /home/jdyson/.minikube/ca.crt \
  --requestheader-client-ca-file /home/jdyson/.minikube/ca.crt \
  --requestheader-username-headers=X-Remote-User \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --etcd-servers=http://localhost:2379 \
  --secure-port=9443 \
  --tls-ca-file ./apiserver.local.config/certificates/apiserver.crt

and then running:

$ kubectl get integrations

Error from server (Forbidden): User "system:anonymous" cannot list <kind>.<apigroup> in the namespace "default". (get <kind>.<apigroup>)

Using openssl to see if the client cert CA name was sent to the client shows it wasn't:

$ openssl s_client -state -connect localhost:9443 -CAfile -CAfile ./apiserver.local.config/certificates/apiserver.crt

CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=0 CN = localhost@1495047152
verify return:1
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read server session ticket A
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/CN=localhost@1495047152
   i:/CN=localhost@1495047152
---
Server certificate
<SNIP>
subject=/CN=localhost@1495047152
issuer=/CN=localhost@1495047152
---
No client certificate CA names sent
Peer signing digest: SHA384
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1396 bytes and written 434 bytes
---
<SNIP>

Notce the No client certificate CA names sent line - assuming that is the problem?

Better error messaging for apiserver-boot

  • Check for tools before running commands - etcd, openssl, tar, cp
  • Check that required commands are run before later commands - init before create-resource, build before run

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.