dataman-cloud / swan Goto Github PK

View Code? Open in Web Editor NEW

409.0 44.0 95.0 9.13 MB

A Distributed, Highly Available Mesos Scheduler, Inspired by the design of Google Borg

Home Page: https://www.shurenyun.com/product-swan.html

License: Apache License 2.0

Go 99.06% Makefile 0.68% Shell 0.26%

mesos mesos-scheduler swan marathon

swan's People

Contributors

Stargazers

Watchers

Forkers

cmingxu upccup weiwei04 f0x11 gitter-badger vitan fanzhang15 nisang akinswin lsword neven7 weiguang1017 foogao cloudfee cool-lab zhaixuepan punalpatel wwjiang007 xychu jiangzhe2014 merryfox douzl aliveon joerg84 qijianbo010 awesome-docker awesome-golang is00hcw fnet123 huncent realforce1024 xiaods xiaosl lfsadamn risongao wheatmr bbklab billyciscooracle caplost xixiang230 yianjiajia huahuiyang lorgine-li gorilla001 857851000 timerope muercy lengyun123456 biancl kaychang forthe2008 flytinge angeljack chenlong-dataman dockernuts dxc520 zhengkai6 lavender2020 raymondlwb triompha devopsmi slpcat koulq jaedongrelax kwff satchel9 romatotti kangzhenkang yangchangshun yangshuailing swinsey lukehuang forging2012 ruhip reinwind lujungit hombre9 fjfd tyokyo songyandong arunkumarramanan liushengsoftman wesleyhuang2014 awesomegolang skymysky kyl2016 ayuan20022003 chenqing24 galaxyyan corersky moayyaed cjjxfli abdqum iq-scm

swan's Issues

use macvlan on docker-machine/virtual box

创建 vm 时，需要加入以下参数：
--virtualbox-hostonly-nicpromisc allow-all
该参数取值默认是 deny，会导致 vm 网卡无法开启混淆模式。
https://docs.docker.com/machine/drivers/virtualbox/

replaceable variables for CMD type heath check

for example

health check type: CMD
health check value: ping { .ip } OR nc { .ip } { .web }

Atomic operation.

Now in Router -> BuildApplication, try FetchApplication before do real thing.
But it's not atomic.

Support user specified `SubDomain` in Swan

Currently swan name Apps as <appName>.<runAs>.<clusterName>,
is it possible to support user specified SubDomain as
<appName>.<runAs>.<subDomain or clusterName if no subDomain given> ?

We'd like to use this field to subset user Apps.

I want to get slotID when get task info. Currently, the returned value is like this:
"tasks": [
{
"agentHostname": "192.168.1.162",
"agentID": "7eb8072d-25f4-463b-a743-84767f5fd512-S15",
"appId": "nginx-yaoyun-datamanmesos",
"cpu": 0.01,
"created": "2017-03-13T07:55:21.055349122Z",
"healthy": true,
"id": "0-nginx-yaoyun-datamanmesos-edd6bede0fb8428ab05efd89f5c6d6b9",
"image": "nginx:1.10",
"mem": 16,
"offerID": "7eb8072d-25f4-463b-a743-84767f5fd512-O2286442",
"status": "slot_task_running",
"versionId": "1489391720"
}

reflected apis:
http:swan_url:port/v_beta/apps/xxxx
http:swan_url:port/v_beta/apps/xxxx/tasks/x

Does swan support passing registry auth info?

We need to support third-party registry, any way to pass the auth info using swan?

integration test required

保证代码质量稳定
保证重构不影响API对外表现

agent 模式增加 dns 端口指定

因为某些时候用户会有自己的dns服务，比如consul-agent，所以两者不能冲突。

bug found related to port reservation

DEBU[2017-03-21 10:40:11] listener &{janitor map[877cd748-dec8-451a-b525-102f375552aa:{877cd748-dec8-451a-b525-102f375552aa 192.168.1.161:9998/v_beta/agent/janitor/event }] {{%!s(int32=0) %!s(uint32=0)} %!s(uint32=0) %!s(uint32=0) %!s(int32=0) %!s(int32=0)}} have no interest in &{ task_state_failed appname3-member-datamanmesos fixed %!s(types.TaskInfoEvent=&{10.10.0.98 0-appname3-member-datamanmesos appname3-member-datamanmesos 0 slot_task_failed false datamanmesos member fixed})}
DEBU[2017-03-21 10:40:11] &{HEARTBEAT type:HEARTBEAT } event=mesos
DEBU[2017-03-21 10:40:11] logger handler report got event type: mesos_heartbeat handler=logger
DEBU[2017-03-21 10:40:11] update slot 0-appname3-member-datamanmesos
DEBU[2017-03-21 10:40:11] framework_id:<value:"7eb8072d-25f4-463b-a743-84767f5fd512-0030" > type:ACKNOWLEDGE acknowledge:<agent_id:<value:"7eb8072d-25f4-463b-a743-84767f5fd512-S13" > task_id:<value:"0-appname3-member-datamanmesos-b48ec89ad953476eab8527dcf5cd2c19" > uuid:"\244\252\272\336\326vA\227\250!\225'&.\\010" > sending-call=ACKNOWLEDGE
DEBU[2017-03-21 10:40:12] &{OFFERS type:OFFERS offers:<offers:<id:<value:"d7ef5d2b-f924-42d9-a274-c020afba6bce-O5003" > framework_id:<value:"7eb8072d-25f4-463b-a743-84767f5fd512-0030" > agent_id:<value:"7eb8072d-25f4-463b-a743-84767f5fd512-S13" > hostname:"192.168.1.160" url:<scheme:"http" address:<hostname:"192.168.1.160" ip:"192.168.1.160" port:5051 > path:"/slave(1)" > resources:<name:"cpus" type:SCALAR scalar:<value:0.02 > role:"" > resources:<name:"mem" type:SCALAR scalar:<value:32 > role:"*" > attributes:<name:"vcluster" type:TEXT text:<value:"clusterautotest" > > > > } event=mesos
DEBU[2017-03-21 10:40:12] logger handler report got event type: mesos_offers handler=logger
DEBU[2017-03-21 10:40:12] OfferHandler got offerId: d7ef5d2b-f924-42d9-a274-c020afba6bce-O5003 handler=offer
INFO[2017-03-21 10:40:12] Prepared task 0-appname-member-datamanmesos for launch with offer d7ef5d2b-f924-42d9-a274-c020afba6bce-O5003
panic: runtime error: slice bounds out of range

goroutine 26489 [running]:
panic(0xcbb380, 0xc820012040)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/Dataman-Cloud/swan/src/manager/framework/state.(*OfferWrapper).PortsRemain(0xc820e007e0, 0x0, 0x0, 0x0)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/state/offer_wrapper.go:39 +0x358
github.com/Dataman-Cloud/swan/src/manager/framework/state.(*Task).PrepareTaskInfo(0xc8207fc120, 0xc820e007e0, 0xc8203fada8)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/state/task.go:101 +0x1482
github.com/Dataman-Cloud/swan/src/manager/framework/state.(*Slot).ReserveOfferAndPrepareTaskInfo(0xc8203fad10, 0xc820e007e0, 0x0, 0x0)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/state/slot.go:194 +0x118
github.com/Dataman-Cloud/swan/src/manager/framework/scheduler.OfferHandler(0xc820e006c0, 0xc820e006c0, 0x0, 0x0)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/scheduler/hdl_offer.go:36 +0xb8f
github.com/Dataman-Cloud/swan/src/manager/framework/scheduler.(*Handler).Process(0xc820e006c0, 0x7ff50b24e7c8, 0xc820e006f0)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/scheduler/handler.go:43 +0x301
created by github.com/Dataman-Cloud/swan/src/manager/framework/scheduler.(*HandlerManager).Handle
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/scheduler/handler_manager.go:58 +0x36e

[Call for Discussion] Do not pull all struct definition within single `types` folder

It doesn't make too much sense to put all definitions within single one folder, type definition should put into where are they used mostly.

Any suggestion?

let swan directly connect mesos master url

currently only use zk-path to connect mesos master, in development env, we like to use swan and mesos-local to setup a testing environment. so directly connect master url is straightforward way.

Please support it, help save coder's lifetime.

snap, swan.db,wal have different access permission, should be consistent

swan can not work after raft error raises.

the raft error heaps quickly after it occurs and swan can not work at all.

error msg is like below:
2017-03-21 05:26:54.492339 E | rafthttp: failed to find member 3dc4b40af08a33b8 in cluster 1000
2017-03-21 05:26:54.492461 E | rafthttp: failed to find member 3dc4b40af08a33b8 in cluster 1000
2017-03-21 05:26:54.596364 E | rafthttp: failed to find member 3dc4b40af08a33b8 in cluster 1000
2017-03-21 05:26:54.596482 E | rafthttp: failed to find member 3dc4b40af08a33b8 in cluster 1000

expected:

not raise such error when restart swan or during swan is running. unless there is schema change.
when such error occurs, try not pop up all the time. It may result in the full disk.

swan proxy 的健康检查 api 异常

swan proxy 的健康检查 api 返回错误页面

ERROR

22:03:45 swan | 2017-02-15 22:03:45.229343 C | rafthttp: newURLs [0.0.0.0:2111] should never fail: URL scheme must be http, https, unix, or unixs: 0.0.0.0:2111
22:03:50 swan | time="2017-02-15 22:03:50" level=error msg="Add node failed, Error: raft: failed to process the request: node losts leader status"
22:03:50 swan | time="2017-02-15 22:03:50" level=info msg="192.168.212.3 - - [15/Feb/2017:22:03:50 +0800] "POST /v_beta/nodes HTTP/1.1" 500 61"
22:03:50 swan | time="2017-02-15 22:03:50" level=error msg="register to http://192.168.212.3:9999/v_beta/nodes got error: raft: failed to process the request: node losts leader status"
22:03:55 swan | time="2017-02-15 22:03:55" level=error msg="Add node failed, Error: raft: failed to process the request: node losts leader status"
22:03:55 swan | time="2017-02-15 22:03:55" level=info msg="192.168.212.3 - - [15/Feb/2017:22:03:55 +0800] "POST /v_beta/nodes HTTP/1.1" 500 61"
22:03:55 swan | time="2017-02-15 22:03:55" level=error msg="register to http://192.168.212.3:9999/v_beta/nodes got error: raft: failed to process the request: node losts leader status"

i found our app json is not elegant dsl design. please reference some good example to improve it.

https://sloppy.io/knowledge-base/sloppy-json-reference/

sloppy is powered by mesos. so the app json is good refernece to swan. i see the json is good design. please read it.

design discuss

swan 在功能上能不能拆成独立的二进制包呢，swan-master swan-agent swan-proxy swan-dns。二进制独立，源码放到一个repo里。

error msg is not correct if healthCheck field contains nothing when deploy app

run command:
http post 172.28.128.3:9999/v_beta/apps @example/template-replicates-nginx.json

in template-replicates-nginx.json:
healthCheck is set like below:
"healthCheck": {}

error msg is like: "no port name found in docker's PortMappings"
actually port name has been set in PortMappings.

manager start hung up

when started with listen address already used, manager hangs up and no error reported.

当应用状态为creating时删除该应用失败。

当应用状态为creating, instance状态为slot_task_pending_offer时，删除该应用，应用状态将变为并保持deleting状态。

No healthCheck field when get app info

add Cron job to Swan scope

sometime integration a crone job is amazon works. Kala is a simplistic, modern, and performant job scheduler written in Go. It lives in a single binary and does not have any dependencies.

Kala was inspired by the desire for a simpler Chronos (developed by Airbnb). Kala is Chronos for the rest of us.

https://github.com/ajvb/kala

swan 可以向容器内部内置需要的环境变量

比如网络方面注入索引端口对应的端口port，包括host模式

For standard tech review, we need input a workflow to review the feature or process.

Through Python or rust, i found the RFC is good template to provide overall control for feature implement.

Template:
https://github.com/rust-lang/rfcs/blob/master/0000-template.md

the Process:
https://github.com/rust-lang/rfcs#what-the-process-is

`runningInstances` falls down to zero after one or two days of running

This happened twice on my devel env.
Sample json:

    "cpus": 0.01,
    "mem": 16,
    "mode": "replicates",
    "runAs": "xychu",
    "constraints": [],
    "container": {
        "docker": {
            "parameters": [],
            "network": "BRIDGE",
            "portMappings": [],
            "image": "nginx"
        }
    },
    "appID": "app",
    "instances": 1,
    "labels": {},
    "env": {},
    "cmd": null
}```

增加应用发布过程中的报错信息

1、发布应用时资源不足，直接在实例状态上提示用户资源不足；
2、镜像下载失败，在实例状态上显示实例无法启动，在该实例的日志里展示下载镜像失败，这应该是executor产生的log；
3、镜像文件损坏，在实例状态上显示实例无法启动，在该实例的日志里展示镜像文件损坏，这应该是executor产生的log；
4、容器启动失败，在实例状态上显示实例无法启动，在该实例的日志里展示容器启动失败的具体原因，这应该是STDOUT或STDERR的内容。

其他应用运行时由于各种故障（外部硬件故障或软件自身故障）被重启迁移的信息也应该在实例详情里展示。

Subscribe mesos-master failed

The subscription to mesos-master has a certain probability of failure.

Log

INFO[2017-02-09 15:54:15] Find mesos masters: [192.168.59.104:5050]
2017-02-09 15:54:15.782796 I | Recv loop terminated: err=EOF
2017-02-09 15:54:15.782809 I | Send loop terminated: err=<nil>
INFO[2017-02-09 15:54:15] Subscribe with mesos master [email protected]:5050
INFO[2017-02-09 15:54:15] a0a64d26-8069-41d1-8b46-4946f8feae4a
INFO[2017-02-09 15:54:15] send response not 202 but 400
ERRO[2017-02-09 15:54:15] start node failed. Error: http got respose not 202
http got respose not 202

This phenomenon is incidental. When i restart it work good.

app 和 slot 的 dns 地址，去掉 gateway

eventbus start failed

INFO[2017-03-23 11:07:01] starting eventBus in leader. raft_id=50fd0d12dc5d14be
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14725a8]

goroutine 66 [running]:
github.com/Dataman-Cloud/swan/src/event.AddListener(0x1addc80, 0xc420176bd0)
/usr/local/go/src/github.com/Dataman-Cloud/swan/src/event/event_bus.go:73 +0x28
github.com/Dataman-Cloud/swan/src/manager.(*Manager).handleLeadershipEvents.func2(0x2984128, 0xc420194440, 0xc4201a6010, 0xc420119860, 0xc4201a6000)
/usr/local/go/src/github.com/Dataman-Cloud/swan/src/manager/manager.go:250 +0xef
created by github.com/Dataman-Cloud/swan/src/manager.(*Manager).handleLeadershipEvents
/usr/local/go/src/github.com/Dataman-Cloud/swan/src/manager/manager.go:253 +0x7a4

Use or add `mesos` TaskID in swan app struct.

Currently, the task ID in swan API response is not the mesos TaskID, e.g. "0-2048test-supergroup-datamanmesos" vs "0-2048test-supergroup-datamanmesos-557b9894e3ef46bc9a86f8d7195ca7dd".
Without the mesos TaskID, it's not possible to talk to mesos with swan task info.

Would we use or add mesos task ID back to swan app?

http.Server has no method Close in golang1.6.2

github.com/Dataman-Cloud/swan/vendor/github.com/Dataman-Cloud/swan-janitor/src
vendor/github.com/Dataman-Cloud/swan-janitor/src/janitor_server.go:51: s.httpServer.Close undefined (type *http.Server has no field or method Close)

could swan to be compatible with the marathon's docker label

We use the docker labels to mark the information of the container created by Swan or Marathon, include cluster, user, app, task. But they use the different label key, that increase the difficulty when we handle the data from the containers. Following are the list of the different label's name, please think about how to fix.

Marathon:
cluster LABEL: "VCLUSTER"
app LABEL: "APP_ID"
task ENV: "MESOS_TASK_ID"

Swan:
cluster LABEL: "DM_CLUSTER, VCLUSTER"
app LABEL: "APP_ID, DM_APP"
user LABEL: "DM_USER, USER_NAME"
group LABEL: "GROUP_NAME"
task ENV: "MESOS_CONTAINER_NAME"

@cmingxu @xychu @upccup , we need make sure the name of label and ENV for the monitor tools.

panic: cannot use none as id

tiseAddr: Status: Labels:map[] Role:agent RaftID:0}

panic: cannot use none as id

goroutine 1 [running]:
github.com/Dataman-Cloud/swan/vendor/github.com/coreos/etcd/raft.newRaft(0xc420176200, 0xe1)
/usr/local/go/src/github.com/Dataman-Cloud/swan/vendor/github.com/coreos/etcd/raft/raft.go:268 +0xda4
github.com/Dataman-Cloud/swan/vendor/github.com/coreos/etcd/raft.RestartNode(0xc420176200, 0x0, 0x0)
/usr/local/go/src/github.com/Dataman-Cloud/swan/vendor/github.com/coreos/etcd/raft/node.go:215 +0x43
github.com/Dataman-Cloud/swan/src/manager/raft.(*Node).StartRaft(0xc420192000, 0x2b600f8, 0xc420122ec0, 0xc420176580, 0x1, 0x1, 0x1, 0x1b0b520, 0x1cb84b0)
/usr/local/go/src/github.com/Dataman-Cloud/swan/src/manager/raft/raft.go:196 +0xd13
github.com/Dataman-Cloud/swan/src/manager.(*Manager).start(0xc42010aea0, 0x2b600b8, 0xc420018250, 0xc420176580, 0x1, 0x1, 0x1, 0x0, 0x0)
/usr/local/go/src/github.com/Dataman-Cloud/swan/src/manager/manager.go:201 +0x21e
github.com/Dataman-Cloud/swan/src/manager.(*Manager).InitAndStart(0xc42010aea0, 0x2b600b8, 0xc420018250, 0x2b600b8, 0xc420018250)
/usr/local/go/src/github.com/Dataman-Cloud/swan/src/manager/manager.go:178 +0x100
main.StartManager(0xc420106500, 0x100, 0xc420106500)
/usr/local/go/src/github.com/Dataman-Cloud/swan/main.go:313 +0x214
github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli.HandleAction(0x16840a0, 0x179f130, 0xc420106500, 0xc420124200, 0x0)
/usr/local/go/src/github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli/app.go:485 +0xd4
github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli.Command.Run(0x1776c0d, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x177b94d, 0xd, 0x0, ...)
/usr/local/go/src/github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli/command.go:193 +0xb72
github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli.(*App).RunAsSubcommand(0xc42005eb60, 0xc420106280, 0x0, 0x0)
/usr/local/go/src/github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli/app.go:374 +0xacc
github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli.Command.startApp(0x177838b, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x177eaab, 0x12, 0x0, ...)
/usr/local/go/src/github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli/command.go:280 +0x824
github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli.Command.Run(0x177838b, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x177eaab, 0x12, 0x0, ...)
/usr/local/go/src/github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli/command.go:79 +0x14c7
github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli.(*App).Run(0xc42005e9c0, 0xc420010120, 0x6, 0x6, 0x0, 0x0)
/usr/local/go/src/github.com/Dataman-Cloud/swan/vendor/github.com/urfave/cli/app.go:250 +0x7d0
main.main()

should add VERSION file to repo, build swan need version include.

because go build need VERSION. we need standard the INPUT.

 go build -ldflags "-X github.com/Dataman-Cloud/swan/srv/version.BuildTime=`date -u +%Y-%m-%d:%H-%M-%S` -X github.com/Dataman-Cloud/swan/src/version.Version=0.011"  -v -o bin/swan main.go

and

Makefile

build-swan:
        go build  -ldflags "-X github.com/Dataman-Cloud/swan/srv/version.BuildTime=`date -u +%Y-%m-%d:%H-%M-%S` -X github.com/Dataman-Cloud/swan/src/version.Version=0.01-`git rev-parse --short HEAD`"  -v -o bin/swan main.go

So please add VERSION to standard the version spec.

"make docker-build" is very slow, always hang on the process of downloading packages.

Suggest put "sh -c "apk update && apk add make && apk add git && make"" into base image or change other build way. Seems waste time running update and add package when run docker-build every time.

@cmingxu , @pwzgorilla

delete app swan crash

time="2017-03-31 11:20:39" level=info msg="192.168.1.73 - - [31/Mar/2017:11:20:39 +0000] "DELETE /v_beta/apps/0002.nmg.nmgtest HTTP/1.1" 204 0"
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0xa9 pc=0xa7f373]

goroutine 493159 [running]:
panic(0xcbb380, 0xc820012060)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/Dataman-Cloud/swan/src/manager/framework/state.(*Slot).KillTask(0x0)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/state/slot.go:130 +0x23
github.com/Dataman-Cloud/swan/src/manager/framework/state.(*StateDeleting).Step(0xc82070b100)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/state/state_deleting.go:71 +0x33d
github.com/Dataman-Cloud/swan/src/manager/framework/state.(*StateMachine).Step(0xc820263740)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/state/app_state_machine.go:75 +0x39
github.com/Dataman-Cloud/swan/src/manager/framework/state.(*App).Step(0xc82054b740)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/state/app.go:373 +0x28
github.com/Dataman-Cloud/swan/src/manager/framework/state.(*Slot).SetHealthy(0xc82068c210, 0x0)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/state/slot.go:406 +0x5c
github.com/Dataman-Cloud/swan/src/manager/framework/scheduler.UpdateHandler(0xc82075c510, 0xc82075c510, 0x0, 0x0)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/scheduler/hdl_update.go:59 +0x1631
github.com/Dataman-Cloud/swan/src/manager/framework/scheduler.(*Handler).Process(0xc82075c510, 0x7fcc95e771d0, 0xc82075c540)
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/scheduler/handler.go:43 +0x301
created by github.com/Dataman-Cloud/swan/src/manager/framework/scheduler.(*HandlerManager).Handle
/go/src/github.com/Dataman-Cloud/swan/src/manager/framework/scheduler/handler_manager.go:58 +0x36e
[root@srymaster2 ~]#

Any plan of supporting configurable `MaxLength` for `utils/selection`?

Now they are just constant definitions like
const qualifiedNameMaxLength int = 63 and const LabelValueMaxLength int = 63.

When use swan as an upstream project, we'd like to use some of our own defined fields or labels with different length limits, so it would be nice for us that swan could make it configurable.

swan killPolicy doesn't function well for the time present

killPolicy in swan doesn't work well, should remove now

portName is not needed when healthCheck protocol is "CMD"

Currently, user can not deploy app when there is no portName defined in healthCheck.

容器网络如何选型

[Call for Discussion] Decouple interface `store` into small, separated interfaces base on usage/features

For now we have all data access layer into single one store interface, my concern is that it is going to be got explode when too many feature added in.

What I suggest is we are going to separated into small chunk of interfaces like AppStore, TaskStore, IPAMStore, etc.

Any suggestion?

leader change not stable

raft2017/01/05 11:47:45 INFO: 1 is starting a new election at term 728
raft2017/01/05 11:47:45 INFO: 1 became candidate at term 729
raft2017/01/05 11:47:45 INFO: 1 received MsgVoteResp from 1 at term 729
raft2017/01/05 11:47:45 INFO: 1 became leader at term 729
raft2017/01/05 11:47:45 INFO: raft.node: 1 elected leader 1 at term 729
INFO[2017-01-05 11:47:45] Now leader is change to 127.0.0.1:9999
raft2017/01/05 11:47:47 INFO: 1 [logterm: 729, index: 1129, vote: 1] rejected MsgVote from 3 [logterm: 671, index: 20] at term 729
raft2017/01/05 11:47:50 INFO: 1 [term: 729] received a MsgVote message with higher term from 3 [term: 730]
raft2017/01/05 11:47:50 INFO: 1 became follower at term 730
raft2017/01/05 11:47:50 INFO: 1 [logterm: 729, index: 1129, vote: 0] rejected MsgVote from 3 [logterm: 671, index: 20] at term 730
raft2017/01/05 11:47:50 INFO: raft.node: 1 lost leader 1 at term 730

add https support for API Gateway

swan-janitor could be confit with tls support, so swan user could proxy any https request to http backend,
leave for private.key config on swan.

Add `delaySecond` for health checks

delaySeconds should added to heath checks , which indicates how many seconds should be wait before first health check was trigger after transition to TASK_RUNNING.

multiple role support for swan

reuse mesos multi-role feature to
1, set weight for each role
2, limit resource allocation for individual role
3, turn "runAs" into "role"

mesos异步事件处理封装为同步调用

目前task创建都是基于mesos事件回调进行处理，开发不方便，建议用广播订阅模型封装为同步调用。
功能开发更便捷，也方便直观获取请求的处理过程和异常输出。形如

func Create() error
func CreateWithProgress() io.Reader

作为编排开发的基础

Support cfs_quota_us / cfs_period_us

Mesos has an option:

--[no]-cgroups_enable_cfs Cgroups feature flag to enable hard limits on CPU resources via the CFS bandwidth limiting subfeature. (default: false)

This is really intriguing, since it provides a way to hard-cap containers. such as docker run via the container.docker.parameters field, and docker has command line flags for this:

  --cpu-period                    Limit CPU CFS (Completely Fair Scheduler) period
  --cpu-quota                     Limit CPU CFS (Completely Fair Scheduler) quota

below is workable demo:

krall@dev5-devc:~/ $ docker run -d --cpu-period 100000 --cpu-quota 20000 busybox sh -c 'yes >/dev/null'
481e13359f87800a5979d09f6c0caa3f8b54754bdfe2d6103cd16563c6000366
krall@dev5-devc:~/ $ docker stats --no-stream 481e13359f87800a5979d09f6c0caa3f8b54754bdfe2d6103cd16563c6000366
CONTAINER                                                          CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O
481e13359f87800a5979d09f6c0caa3f8b54754bdfe2d6103cd16563c6000366   20.12%              409.6 kB / 135.2 GB   0.00%               648 B / 648 B       319.5 kB / 0 B
krall@dev5-devc:~/ $

DEFAULT_CPU_PERIOD = 100000 (100ms)
DEFAULT_CPU_BURST_PCT = 100

   def get_cpu_period(self):
        return self.config_dict.get('cfs_period_us', DEFAULT_CPU_PERIOD)
 
     def get_cpu_quota(self):
         cpu_burst_pct = self.config_dict.get('cpu_burst_pct', DEFAULT_CPU_BURST_PCT)
         return self.get_cpus() * self.get_cpu_period() * (100 + cpu_burst_pct) / 100

the formula:
cfs_quota_us = cpus * cfs_period_us * (100 + cpu_burst_pct) / 100

docker parameter setting:

 {"key": "cpu-period", "value": self.get_cpu_period()},
 {"key": "cpu-quota", "value": self.get_cpu_quota()},

`taskPortMappings` is null

[
{
"appID": "nginx-xcm-datamanmesos",
"appMode": "replicates",
"ip": "192.168.1.175",
"taskID": "0-nginx-xcm-datamanmesos",
"taskPortMappings": null,
"url": "0.nginx.xcm.datamanmesos"
}
]