Coder Social home page Coder Social logo

meli's Introduction

meli

ci codecov GoDoc Go Report Card

Meli is supposed to be a faster, and drop in, alternative to docker-compose. Faster in the sense that, Meli will try to pull as many services(docker containers) as it can in parallel.

Meli is a Swahili word meaning ship; so think of Meli as a ship carrying your docker containers safely across the treacherous container seas.

It's currently work in progress, API will remain unstable for sometime.

I only intend to support docker-compose version 3+

Meli is NOT intended to replicate every feature of docker-compose, it is primarily intended to enable you to pull, build and run the services in your docker-compose file as fast as possible.
If you want to exec in to a running container, use docker; if you want to run an adhoc command in a container, use docker; if you want..... you get the drift.

Installing/Upgrading

Download a binary release for your particular OS from the releases page
We have binaries for:

  • linux(64bit)
  • windows(64bit)
  • macOS(64bit)

Usage

meli --help

Usage of meli:
  -up
    	Builds, re/creates, starts, and attaches to containers for a service.
        -d option runs containers in the background
  -f string
    	path to docker-compose.yml file. (default "docker-compose.yml")
  -build
    	Rebuild services
  -v	Show version information.
  -version
    	Show version information.

cat docker-compose.yml

version: '3'
services:
  redis:
    image: 'redis:3.0-alpine'
    environment:
      - RACK_ENV=development
      - type=database
    ports:
      - "6300:6379"
      - "6400:22"

meli -up

redis :: Pulling from library/redis 
redis :: Pulling fs layer 
redis :: Pulling fs layer 
redis :: Downloading [======================>        ]  3.595kB/8.164kB
redis :: Downloading [==============================>]  8.164kB/8.164kB
redis :: Download complete [========================>]  8.164kB/8.164kB
redis :: The server is now ready to accept connections on port 6379

Usage as a library

You really should be using the official docker Go sdk
However, if you feel inclined to use meli;

package main

import (
	"context"
	"log"
	"os"

	"github.com/docker/docker/client"
	"github.com/komuw/meli"
)

func main() {
	dc := &meli.DockerContainer{
		ComposeService: meli.ComposeService{Image: "busybox"},
		LogMedium:      os.Stdout,
		FollowLogs:     true}

	ctx := context.Background()
	cli, err := client.NewEnvClient()
	if err != nil {
		log.Fatal(err, " :unable to intialize docker client")
	}
	defer cli.Close()

	meli.LoadAuth() // load dockerhub info
	err = meli.PullDockerImage(ctx, cli, dc)
	log.Println(err)

}

Benchmarks

Aaah, everyones' favorite vanity metric yet no one seems to know how to conduct one in a consistent and scientific manner.
Take any results you see here with a large spoon of salt; They are unscientific and reek of all that is wrong with most developer benchmarks.

Having made that disclaimer,

Benchmark test:
this docker-compose file

Benchmark script:
for docker-compose:
docker ps -aq | xargs docker rm -f; docker system prune -af; /usr/bin/time -apv docker-compose up -d
for meli:
docker ps -aq | xargs docker rm -f; docker system prune -af; /usr/bin/time -apv meli -up -d

Benchmark results(average):

tool Elapsed wall clock time(seconds)
docker-compose 10.411 seconds
meli 3.945 seconds

Thus, meli appears to be 2.6 times faster than docker-compose(by wall clock time).
You can checkout the current benchmark results from circleCI
However, I'm not making a tool to take docker-compose to the races.

Development

Build

git clone [email protected]:komuw/meli.git
go build -trimpath -o meli cli/cli.go
./meli -up -f /path/to/docker-compose-file.yml

TODO

  • add better documentation(godoc)
  • stabilise API(maybe)

meli's People

Contributors

dependabot[bot] avatar glensc avatar komuw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

meli's Issues

add a cli

  • add a simple cli

  • should only support:

  1. meli up should run containers
  2. meli should show all the options/menu /docs of the cli
  3. meli up -d should run containers in the background without following logs
  4. meli version should show meli version

expand env vars in volumes

The docker-compose file

version: '3'
services:
  onbuildcopy:
    build:
      context: .
      dockerfile: OnbuildCopyDockerfile
    command: tail -f /usr/src/app/OnbuildCopyDockerfile
    volumes:
      - $HOME/.aws:/root/.aws

fails with;
Error response from daemon: create $HOME/.aws: "$HOME/.aws" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

OoBuild COPY fails

this dockerfile:

FROM komuw/busyboxonbuild
RUN cat LICENSE
CMD cat README.md

and docker-compose.yml file:

version: '3'
services:
 oiko:
   build:
     context: .
   volumes:
     - ./:/usr/src/app

where komuw/busyboxonbuild is:

FROM busybox
ONBUILD COPY . /usr/src/app
WORKDIR /usr/src/app

fails with :

{"stream":"Step 1/3 : FROM komuw/busyboxonbuild\n"}
{"stream":"\u001b[91m# Executing 1 build trigger...\n\u001b[0m"}
{"stream":"Step 1/1 : COPY . /usr/src/app\n"}
{"stream":" ---\u003e 5a05ef3ce627\n"}
{"stream":"Step 2/3 : RUN cat LICENSE\n"}
{"stream":" ---\u003e Running in 9b7ef8f2aff2\n"}
{"stream":"\u001b[91mcat: can't open 'LICENSE': No such file or directory\n\u001b[0m"}
{"errorDetail":{"code":1,"message":"The command '/bin/sh -c cat LICENSE' returned a non-zero code: 1"},"error":"The command '/bin/sh -c cat LICENSE' returned a non-zero code: 1"}
2017/10/18 15:43:20
         service=oiko error=originalErr:: Error: No such image: meli_dockerfile
ThisErr::  :unable to create container

links are linked wrongly

for compose fiile:

  rabbitmq:
    image: rabbitmq:3.5.7-management

  app:
    links:
      - db

inside app container, ping db doesn't work. but ping meli_db.home.user.mydir.my-project works

meli up fails for osx auth

with an auth file like:
cat ~/.docker/config.json

{
	"auths": {
		"https://index.docker.io/v1/": {}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/17.09.0-ce (darwin)"
	},
	"credsStore": "osxkeychain"
}

we get failure;
meli -up

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/komuw/meli/api.GetAuth()
	/home/komu/go/src/github.com/komuw/meli/api/auth.go:66 +0x1c8b
main.main()
	/home/komu/go/src/github.com/komuw/meli/main.go:58 +0x531

Feature: add brew

I just want to report that it would be awesome if meli could be installed with brew

➔ brew search meli
==> Searching local taps...
timelimit
==> Searching taps on GitHub...
==> Searching blacklisted, migrated and deleted formulae...
➔

panic: runtime error: index out of range

Thank you for using meli and taking your time to report an issue to us.

What operating system and version of operating system are you using?

osx

What version of meli are you using? run meli meli --version

latest

What did you do? (be as detailed as you can)

meli -up

What did you expect to see/happen/not happen?

  • running services

What happened instead?

  • panic

Paste the log output generated by meli -up. Please remember to remove any sensitive items from the log before copy-pasting.

panic: runtime error: index out of range

goroutine 14 [running]:
github.com/komuw/meli/api.CreateContainer(0x1375a20, 0xc4200180d0, 0x13776c0, 0xc42011bd80, 0xc4200cf760, 0x0, 0x0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/komuw/meli/api/container.go:57 +0x15b7
main.startComposeServices(0x1375a20, 0xc4200180d0, 0xc42011bd80, 0xc42025f530, 0xc4200cf760)
	/root/go/src/github.com/komuw/meli/main.go:118 +0xb0
created by main.main
	/root/go/src/github.com/komuw/meli/main.go:93 +0x586

If you don't mind, copy-paste the docker-compose.yml file that you were using(or an equivalent compose file that reproduces the same issue)?

version: '3'
services:
  redis:
    image: 'redis:3.0-alpine'
    environment:
      - RACK_ENV=development
      - type=database
    ports:
      - 6379

container networking fails

When inside a container wget google.com fails with wget: bad address 'google.com' but works for docker-compose

for docker-compose:

nslookup google.com
Server: 127.0.0.11
Address 1: 127.0.0.11

Name: google.com
Address 1: 2a00:1450:4002:80a::200e mil04s28-in-x0e.1e100.net
Address 2: 216.58.205.174 mil04s28-in-f174.1e100.net

it returns both IPV6 and IPV4 addresses.

for meli

nslookup google.com
Server: 127.0.0.11
Address 1: 127.0.0.11

Name: google.com
Address 1: 2a00:1450:4002:807::200e mil04s25-in-x0e.1e100.net
Address 2: 216.58.205.78 mil04s25-in-f14.1e100.net

it returns both IPV6 and IPV4 addresses. but still wget fails

improve speed

  • currently we are at 2X faster than docker-compose.
  • we should improve that
  • use all the tools that golang provides: tests, benchmark, pprof, go tool trace, flamegraphs etc

depends on; #27

docker fails

docker run -it komuw/meli -up -f testdata/docker-compose.yml

2017/11/21 16:41:22 open testdata/docker-compose.yml: no such file or directory :unable to read docker-compose file

Other commands however work;;
docker run -it komuw/meli -v
Meli version: 0.1.7.2
docker run -it komuw/meli --help

Usage of /meli:
  -d	Run containers in the background
  -f string
    	path to docker-compose.yml file. (default "docker-compose.yml")
  -up
    	Builds, re/creates, starts, and attaches to containers for a service.
  -v	Show version information.
  -version
    	Show version information.

read docker-compose service context

In meli, when building image; we assume[1] that the context is the directory in which the dockerfile is in.

This is not always true; consider a docker-compose.yml file such as;

version: '3'
services:
  nestedservice:
    build:
      context: .
      dockerfile: nestedDir/level1/level2/nestedDockerfile
    labels:
      - "type=db"

where nestedDir/level1/level2/nestedDockerfile is;

FROM busybox
# refer to a file that is in the same dir as docker-compose file(ie in the . context)
ADD OnbuildCopyDockerfile /usr/share/OnbuildCopyDockerfile

CMD tail -fn3 .dockerenv

when using meli -up you will get an error such as

Status: Downloaded newer image for busybox:latest [==================================================>]  715.3kB/715.3kB
nestedservice :: Status: Downloaded newer image for busybox:latest 
                      
service=nestedservice error=originalErr:: Error: No such image: meli_nestedservice

ref:

  1. https://github.com/komuW/meli/blob/c61ffd08b50a3af1352ac2cc1d20e4399710e60f/api/image.go#L151

meli fails in directory with docker-compose file

How to reproduce:

  1. cd testdata/
  2. meli -up -d
2017/10/10 19:58:01 
	 service=buildservice error=originalErr:: originalErr:: open docker-compose.yml/buildserviceDockerfile: not a directory 
ThisErr::  :unable to open Dockerfile docker-compose.yml/buildserviceDockerfile 
ThisErr:: %!s(<nil>)
2017/10/10 19:58:01 

add .env support

Looks like meli doesn't support .env file.

➔ cat .env
APP_VERSION=1

➔ cat docker-compose.yml
version: "3"

services:
  app:
    image: nginx:${APP_VERSION}

➔ meli -up

	 service=app error=originalErr:: invalid reference format
ThisErr::  :unable to pull image nginx:${APP_VERSION}➔

ps: there's \n missing from error, my bash prompt does not start on new line due that

volumes starting with .(dot) fail.

    volumes:
      - .ala:/etc/ala
      - .cool:/etc/cool

results in error:

Error response from daemon: create .ala: ".ala" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

increase docker max-concurrent-downloads value

We should increase the max-concurrent-download value that is passed to docker daemon.

On linux, we can do that by adding this file[1] to /etc/docker/daemon.json.
However it is not possible to do so on windows.

It is possible to create that file like so;
echo "{\"max-concurrent-downloads\": 55}" > /etc/docker/daemon.json
see [3]

  1. https://docs.docker.com/engine/reference/commandline/dockerd/#linux-configuration-file
  2. https://docs.docker.com/engine/reference/commandline/dockerd/#windows-configuration-file
  3. moby/moby#22445 (comment)

host path issue

this compose file

  db:
    build:
      context: .
      dockerfile: volumeDockerfile
    volumes:
      - data-volume:/home
      - /var/run/docker.sock:/var/run/docker.sock

will fail with ;
{"status":"Status: Image is up to date for traefik:latest"}
2017/10/17 16:39:04
service=lb error=originalErr:: Error response from daemon: create meli_/var/run/docker.sock: "meli_/var/run/docker.sock" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

archive/tar: write too long

Thank you for using meli and taking your time to report an issue to us.

What operating system and version of operating system are you using?

ubuntu 16

What version of meli are you using? run meli meli --version

0.1.1

What did you do? (be as detailed as you can)

ran meli -up

What did you expect to see/happen/not happen?

service start

What happened instead?

service=yo error=originalErr:: originalErr:: archive/tar: write too long 
ThisErr::  :unable to walk dockefile context path Dockerfile 
ThisErr:: %!s(<nil>)

Paste the log output generated by meli -up. Please remember to remove any sensitive items from the log before copy-pasting.

meli -up

	 service=app error=originalErr:: originalErr:: archive/tar: write too long 
ThisErr::  :unable to walk dockefile context path Dockerfile 
ThisErr:: %!s(<nil>)

If you don't mind, copy-paste the docker-compose.yml file that you were using(or an equivalent compose file that reproduces the same issue)?

COPY-PASTE docker-compose.yml file HERE

meli up fails on a new machine

meli -up
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/komuw/meli/api.GetAuth()
/home/komu/go/src/github.com/komuw/meli/api/auth.go:66 +0x1c8b
main.main()
/home/komu/go/src/github.com/komuw/meli/main.go:58 +0x531

fix error handling

  • do not do log.Fatal
  • if a func has an error in it, maybe it ought to return that error and it is upto to the caller to decide wether to exit or just log it.

add ci

  • remember to use the race detector, go vet etc in there.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.