Coder Social home page Coder Social logo

dockyard's Introduction

Dockyard - Container And Artifact Repository

Dockyard

What is Dockyard ?

Dockyard is a container and artifact repository storing and distributing container image, software artifact and virtual images of KVM or XEN. It's key features and goals include:

  • Multi supported distribute protocols include Docker Registry V1 & V2 and App Container Image Discovery.
  • Multi supported software artifact format include jar, war, tar and so on.
  • Multi supported virtual images of KVM, XEN, VirtualBox and so on.
  • Container image, software artifact and virtual images encryption, verification and vulnerability analysis.
  • Custome distribute protocol by framework base HTTPS and peer to peer.
  • Authentication in distributing process and authorization for public and private repository.
  • Supporting mainstream object storage service like Amazon S3, Google Cloud Storage.
  • Built-in object storage service for deployment convenience.
  • Web UI portal for all functions above.

Why it matters ?

The Dockyard's Story :)

Runtime configuration

runmode = "dev"

listenmode = "https"
httpscertfile = "cert/containerops/containerops.crt"
httpskeyfile = "cert/containerops/containerops.key"

[site]
domain = "containerops.me"

[log]
filepath = "log/backend.log"
level = "info"

[database]
driver = "mysql"
uri = "containerops:containerops@/containerops?charset=utf8&parseTime=True&loc=Asia%2FShanghai"

[deployment]
domains = "containerops.me"

[dockerv1]
standalone = "true"
version = "0.9"
storage = "/tmp/data/dockerv1"

[dockerv2]
distribution = "registry/2.0"
storage = "/tmp/data/dockerv2"

[appc]
storage = "/tmp/data/appc"

[updateserver]
keymanager = "/tmp/containerops_keymanager_cache"
storage = "/tmp/containerops_storage_cache"

Nginx configuration

It's a Nginx config example. You can change client_max_body_size what limited upload file size. You should copy containerops.me keys from cert/containerops.me to /etc/nginx, then run Dockyard with http mode and listen on 127.0.0.1:9911.

upstream dockyard_upstream {
  server 127.0.0.1:9911;
}

server {
  listen 80;
  server_name containerops.me;
  rewrite  ^/(.*)$  https://containerops.me/$1  permanent;
}

server {
  listen 443;

  server_name containerops.me;

  access_log /var/log/nginx/containerops-me.log;
  error_log /var/log/nginx/containerops-me-errror.log;

  ssl on;
  ssl_certificate /etc/nginx/containerops.me.crt;
  ssl_certificate_key /etc/nginx/containerops.me.key;

  client_max_body_size 1024m;
  chunked_transfer_encoding on;

  proxy_redirect     off;
  proxy_set_header   X-Real-IP $remote_addr;
  proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header   X-Forwarded-Proto $scheme;
  proxy_set_header   Host $http_host;
  proxy_set_header   X-NginX-Proxy true;
  proxy_set_header   Connection "";
  proxy_http_version 1.1;

  location / {
    proxy_pass         http://dockyard_upstream;
  }
}

Database Configuration

Database SQL

INSERT INTO mysql.user(Host,User,Password) VALUES ('localhost', 'containerops', password('containerops'));
CREATE DATABASE `containerops` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON containerops.* TO containerops@localhost IDENTIFIED BY 'containerops';
FLUSH PRIVILEGES;

Initlization Tables

./dockyard database migrate

Start dockyard service

  • Run directly:
./dockyard daemon run --address 0.0.0.0 --port 443
  • Run with Nginx:
./dockyard daemon run --address 127.0.0.1 --port 9911 &

How to build

We are using glide as package manager/

  • retrieve dependencies
glide install
  • build (with go 1.6+)
go build

How to involve

If any issues are encountered while using the dockyard project, several avenues are available for support:

Issue Tracker https://github.com/Huawei/dockyard/issues
Google Groups https://groups.google.com/forum/#!forum/dockyard-dev

Pull Requests

If you want to contribute to the template, you can create pull requests. All pull requests must be done to the develop branch. We are working on build an automated tests with ourself means containerops, now we just add Travis CI instead.

Who should join

  • Ones who want to choose a container image hub instead of docker hub.
  • Ones who want to ease the burden of container image management.

Certificate of Origin

By contributing to this project you agree to the Developer Certificate of Origin (DCO). This document was created by the Linux Kernel community and is a simple statement that you, as a contributor, have the legal right to make the contribution.

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
660 York Street, Suite 102,
San Francisco, CA 94110 USA

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Format of the Commit Message

You just add a line to every git commit message, like this:

Signed-off-by: Meaglith Ma <[email protected]>

Use your real name (sorry, no pseudonyms or anonymous contributions.)

If you set your user.name and user.email git configs, you can sign your commit automatically with git commit -s.

dockyard's People

Contributors

chengtiesheng avatar fivestarsky avatar gosharplite avatar guilhem avatar initlove avatar liangchenye avatar mabingo avatar osguydch avatar vijaysamanuriwm avatar wangkirin avatar zenlint avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dockyard's Issues

content scan feature

Snapshot : scan backend

updateservice/snapshot is an interface, developers could add different plugins to do their scan work.
Now I just adding a simple 'appv1' plugin which get the md5 of an appv1 package.

why call it 'Snapshot'

In the traditional repositories provider, like SUSE/Redhat distribution host servers, their are not only provide simple metadata, but also provide detail information. It helps user to 'search' required packages and download them.

why under updateservice

In TUF framework, there is a similar snapshot concept. If a package admin want to provide a secure snapshot service to his user, he/she can signature/encrypt snapshot.

How does it work -- Plugin mode

Just like other services under updateservice, we can develop/use Snapshot by a single interface. The interface will choose the suitable scan plugin.

Plugin Developer

There are two key functions:

  • Supported(proto string) bool

    This is to tell which protocol this plugin supports. For example, if a plugin could scan 'appv1',
    Supported("appv1") should be 'true'
    We also have Description function which is easier for user to understand what this plugin do.

  • Process (url, id, callback)

    • URL

      URL is a local url of an appv1 package or a local dir of a docker image. A plugin developer could read its content and do what ever he want.

    • ID

      ID is the callbackID (scan taskID), but was encrypted in order not to expose the inner database details. Plugin developer should use this to tell 'Plugin user' the scan result.

    • callback

    If the Plugin User provides this 'callback' function to plugin developer, the developer MUST call it in his/her plugin.
    Or the developer should post the scan result to Dockyard by RestAPI which will mentioned later.

Plugin User (or we can call them dockyard handler developer)

Plugin User provides should choose the right plugin and calling it. He/she can query the scan status by callbackID and collected the scan result by this ID too.

I added four APIs:

  • RegistScanHook (POST)

    Dockyard will have lots plugins. A dockyard user could choose one (or several in the future) suitable plugin for his/her repository. I call this action 'regist'.

  • ActiveScanHook (POST)
    A dockyard user could scan a package. Dockyard will find the registered scan plugin and use that to scan the package. I call this action 'active'

  • CallbackScanHook (POST)
    This is used for a plugin developer, he/she can post his/her scan result to dockyard by this ID.

  • QueryResultScanHook (GET)
    This is not done. It is also for dockyard user, he/she can query the scan result by calling this API.

Can't build master

I've got an error building master using Go 1.6:

$ git clone https://github.com/containerops/dockyard                                                                                                                                             
Cloning into 'dockyard'...
remote: Counting objects: 3763, done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 3763 (delta 1), reused 0 (delta 0), pack-reused 3710
Receiving objects: 100% (3763/3763), 3.29 MiB | 1.36 MiB/s, done.
Resolving deltas: 100% (1342/1342), done.
Checking connectivity... done.
$ cd dockyard     
$ make
go build -tags "" -o dockyard .
# github.com/containerops/dockyard/handler
../golang/src/github.com/containerops/dockyard/handler/blob.go:52: too many arguments in call to utils.MD5
Makefile:5: recipe for target 'all' failed
make: *** [all] Error 2

DB variable uninitialized

Hello,

The web server is panicing on every request because of the DB variable being nil.

I've added a call to models.OpenDatabase() in SetDockyardMacaron function to prevent that as a quick fix. Not sure where you would wish to put that. There, or in a init function.

Regards.

If listenmode is set to http in runtime.conf,docker push failed

Step 1: runtime.conf setting as below,
runmode = dev

listenmode = http
httpscertfile = cert/containerops/containerops.crt
httpskeyfile = cert/containerops/containerops.key

[log]
filepath = log/containerops-log

[db]
uri = localhost:6379
passwd = containerops
db = 8

[dockyard]
driver = amazons3cloud
path = data
domains = containerops.me
registry = 0.9
distribution = registry/2.0
standalone = true

Step 2: Execute the command as below,
docker push containerops.me/mabin/busybox:latest

Result:
root@ConDev:/home/workspace/docker-bin# docker push containerops.me/mabin/busybox:latest
The push refers to a repository [containerops.me/mabin/busybox](len: 1)
8c2e06607696: Image push failed
Error pushing to registry: Put https://containerops.me/v2/mabin/busybox/blobs/uploads/0f069c86957d9de1f83ad2f7997f3f43?_state=7508334567da5a77b11a87a6bd8d1ace&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4:

Docker daemon version,
root@ConDev:/home/workspace/docker-bin# docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64

New feature: volume manager

Volume is a resource to a runtime, just like image. In Docker, there is a 'plugin' designation of volume management, but it combines the runtime and volume together. I think we can add a 'volume manager' feature in Dockyard to decouple them.

If we look at the reference protocols, AppC is a good example. It provides an image discovery mechanism to decouple image and runtime. Volume manager could share the similar designation, the way of using volume in containers in the future could be like this:

  1. discover the volume resources and return the volume list
  2. apply a required volume resource and return a local mount point
  3. use the mount point directly by a runtime, either in the command line or the configuration file
  4. free the used volume when the runtime exist or die.

I develop it in my branch: https://github.com/liangchenye/dockyard/tree/volume.
There are two parts:

  1. volume manger server
    It is provides the uniform interface for user to retrieve the volume resource.
    https://github.com/liangchenye/dockyard/blob/volume/volume/README.md
  2. agent
    It tells the volume manger server the volume resource on a single node and
    do the real mount/umount work.
    https://github.com/liangchenye/dockyard/blob/volume/agent/README.md

Any suggestion is welcome!

How do I create the database?

I'm interested in using Dockyard, but I'm having difficulty creating the database. How do I create the migrations?

My Runtinme configuration about database is:

[database]
driver = mysql
uri = myuser:mypass@/mydockyarkdb

The README says "Initlization Tables":

dockyard db --action sync

However:

Error: unknown command "db" for "dockyard"
Run 'dockyard --help' for usage.
unknown command "db" for "dockyard"

I tried to run:

dockyard database migrate

But I get the following error:

`` is not officially supported, running under compatibility mode.
FATA[0000] Initlization database connection error.

Initial setup documentation

First, thanks for the work.

I've installed dockyard with go get, then launched it from a random directory since the dockyard command was in my $PATH.

I had trouble then figuring out that the binary was not finding the configuration file (the error message was not obvious). The README seems to imply that the project should be fetch with git clone, be build with go build and run in place ./dockyard.

Maybe, detailing a bit the installation and documenting the possible configuration file path and format would be useful (eg: toml in either /etc/containerops, $HOME/.containerops, ./conf).

Small thing, the README is also mentioning a daemon run subcommand.

librados.h file not found

No luck with build. Getting rados error:

make
go build -tags "" -o dockyard .

github.com/ceph/go-ceph/rados

Godeps/_workspace/src/github.com/ceph/go-ceph/rados/conn.go:5:11: fatal error: 'rados/librados.h' file not found
#include <rados/librados.h>
^
1 error generated.
make: *** [all] Error 2

dockyard expects mysql to have a database dockyard already created

I'm trying to get dockyard up and running. I started up a mysql instance. However, dockyard logs that it was not able to find a dockyard database in mysql. This error keeps coming up until I create the database itself. I think this is something that dockyard should do. Create the database if not present. If its present then just connect to it.

How do I get started?

I'm super interested in using Dockyard, as I'm interested in something that makes Clair easy to use.

However, I'm having difficulty understanding how I get the app running. How do I get started?

I see reference to a makefile in some issues, but it was removed in 0f4050e

  • How do I get started running the app?
  • Is it possible to create a docker-compose setup for running dockyard?

'undefined: config.ConfigContainer' when building Dockyard

Hi,

I'm trying to build Dockyard so I can test if it will work in my environment, but I keep getting the following error: "src/github.com/containerops/wrench/setting/setting.go:16: undefined: config.ConfigContainer".

The process I'm following is:

  • Fetch a golang environment using Docker: docker pull golang:1.6
  • Run a container to build: docker run -ti golang:1.6 /bin/bash
  • Inside the container, follow the instructions from the README:
go get github.com/containerops/dockyard
cd $GOPATH/src/github.com/containerops/dockyard
go build

After the 'go get' command, I get the error as above. I have tried with both the golang:1.6 and golang:1.5 images, but with the same result. Any suggestions?

Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.