projecteru2 / core

Eru, a simple, stateless, flexible, production-ready orchestrator designed to easily integrate into existing workflows. Can run any virtualization things in long or short time.

Eru

Eru is a stateless, flexible, production-ready resource scheduler designed to easily integrate into existing systems.

Eru can use multiple engines to run anything for the long or short term.

This project is Eru Core. The Core use for resource allocation and manage resource's lifetime.

Suggest use go 1.20 and above.

Testing

Run make test

Compile

• Run make build if you want binary.
• Run ./make-rpm if you want RPM for el7. However we use FPM for packing, so you have to prepare it first.

Developing

Run make deps for generating vendor dir.

You can use our footstone image for testing and compiling.

GRPC

Generate golang grpc definitions.

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
make grpc

Run it

$ eru-core --config /etc/eru/core.yaml.sample

or

$ export ERU_CONFIG_PATH=/path/to/core.yaml
$ eru-core

Dockerized Core manually

Image: projecteru2/core

docker run -d \
  --name eru_core_$HOSTNAME \
  --net host \
  --restart always \
  -v <HOST_CONFIG_DIR_PATH>:/etc/eru \
  projecteru2/core \
  /usr/bin/eru-core

Build and Deploy by Eru itself

After we implemented bootstrap in eru, now you can build and deploy eru with cli tool.

1. Test source code and build image

<cli_execute_path> --name <image_name> http://bit.ly/EruCore

Make sure you can clone code. After the fresh image was named and tagged, it will be auto pushed to the remote registry which was defined in config file.

2. Deploy core itself

<cli_execute_path> workloads deploy --pod <pod_name> [--node <node_name>] --entry core --network <network_name> --image <projecteru2/core>|<your_own_image> --file <core_config_yaml>:/core.yaml [--count <count_num>] [--cpu 0.3 | --mem 1024000000] http://bit.ly/EruCore

Now you will find core was started in nodes.

core's Issues

randomly generate server list

截获 server list 的时候随机化

https://eru.sh/docs/ is 404

inplace update failed

time="2019-06-20 09:45:02" level=info msg="[rpc] DeployStatus start virtual"
time="2019-06-20 09:45:02" level=info msg="[rpc] DeployStatus start redisproxy"
time="2019-06-20 09:45:02" level=info msg="[rpc] DeployStatus start redissingular"
time="2019-06-20 09:45:02" level=info msg="[rpc] DeployStatus start rediscluster"
time="2019-06-20 09:45:36" level=info msg="[rpc] DeployStatus stop redissingular"
time="2019-06-20 09:45:36" level=info msg="[rpc] DeployStatus stop virtual"
time="2019-06-20 09:45:36" level=info msg="[rpc] DeployStatus stop redisproxy"
time="2019-06-20 09:45:36" level=info msg="[rpc] DeployStatus stop rediscluster"
time="2019-06-20 09:46:28" level=debug msg="[task] ReplaceContainer added"
time="2019-06-20 09:46:28" level=debug msg="[withContainerLocked] Container 5baa5cbb981a154beca2407bbb1d0b7ee1b4e5212e53f273a19e1b50a7ca8b3e locked"
time="2019-06-20 09:46:28" level=warning msg="[ReplaceContainer] Skip not in pod container 5baa5cbb981a154beca2407bbb1d0b7ee1b4e5212e53f273a19e1b50a7ca8b3e"
time="2019-06-20 09:46:28" level=debug msg="[doUnlock] Unlock 5baa5cbb981a154beca2407bbb1d0b7ee1b4e5212e53f273a19e1b50a7ca8b3e"
time="2019-06-20 09:46:28" level=error msg="[ReplaceContainer] Replace and remove failed Not Support: container 5baa5cbb981a154beca2407bbb1d0b7ee1b4e5212e53f273a19e1b50a7ca8b3e not in pod slave, old container restarted"
time="2019-06-20 09:46:28" level=debug msg="[task] ReplaceContainer done"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xda3e6b]
goroutine 2568 [running]:
github.com/projecteru2/core/rpc.toRPCRemoveContainerMessage(...)
	/go/src/github.com/projecteru2/core/rpc/transform.go:325
github.com/projecteru2/core/rpc.toRPCReplaceContainerMessage(0xc000440560, 0xc000993c58)
	/go/src/github.com/projecteru2/core/rpc/transform.go:278 +0x4b
github.com/projecteru2/core/rpc.(*Vibranium).ReplaceContainer(0xc0001ea1c0, 0xc000a3ede0, 0x11dc1e0, 0xc000a3a0a0, 0x0, 0x0)
	/go/src/github.com/projecteru2/core/rpc/rpc.go:509 +0x1a1
github.com/projecteru2/core/rpc/gen._CoreRPC_ReplaceContainer_Handler(0x1023480, 0xc0001ea1c0, 0x11da680, 0xc0002486c0, 0x1adbfc8, 0xc000163400)
	/go/src/github.com/projecteru2/core/rpc/gen/core.pb.go:4929 +0x10c
github.com/projecteru2/core/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000514900, 0x11e00e0, 0xc000262300, 0xc000163400, 0xc0003f2000, 0x1a70b20, 0x0, 0x0, 0x0)
	/go/src/github.com/projecteru2/core/vendor/google.golang.org/grpc/server.go:1170 +0xacd
github.com/projecteru2/core/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000514900, 0x11e00e0, 0xc000262300, 0xc000163400, 0x0)
	/go/src/github.com/projecteru2/core/vendor/google.golang.org/grpc/server.go:1249 +0xcbe
github.com/projecteru2/core/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000b3c7b0, 0xc000514900, 0x11e00e0, 0xc000262300, 0xc000163400)
	/go/src/github.com/projecteru2/core/vendor/google.golang.org/grpc/server.go:685 +0x9f
created by github.com/projecteru2/core/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/projecteru2/core/vendor/google.golang.org/grpc/server.go:683 +0xa1
time="2019-06-20 09:46:29" level=info msg="[main] Cluster started successfully."

dissociate issue

time="2019-06-20 10:52:44" level=debug msg="[task] ReplaceContainer done"
time="2019-06-20 10:54:20" level=debug msg="[task] DissociateContainer added"
time="2019-06-20 10:54:20" level=debug msg="[withContainerLocked] Container 8203dfe19ba10d6eebf83905e26bdf66413a4615abaa9a6cb0d67140b244f354 locked"
time="2019-06-20 10:54:20" level=debug msg="[withNodeLocked] Node cachecloud-master-sg2-test-0.shopeemobile.com locked"
time="2019-06-20 10:54:20" level=info msg="[DissociateContainer] Container 8203dfe19ba10d6eebf83905e26bdf66413a4615abaa9a6cb0d67140b244f354 dissociated"
time="2019-06-20 10:54:20" level=debug msg="[Metrics] Update cachecloud-master-sg2-test-0.shopeemobile.com memory capacity gauge"
time="2019-06-20 10:54:20" level=debug msg="[UpdateNode] pod admin node cachecloud-master-sg2-test-0.shopeemobile.com cpu slots map[0:100 1:100 10:100 11:100 12:100 13:100 14:100 15:100 16:100 17:100 18:100 19:100 2:100 20:100 21:100 22:100 23:100 24:100 25:100 26:100 27:100 28:100 29:100 3:100 30:100 31:100 32:100 33:100 34:100 35:100 36:100 37:100 38:100 39:100 4:100 40:100 41:100 42:100 43:100 44:100 45:100 46:100 47:100 5:100 6:100 7:100 8:100 9:100] memory 131345006592"
time="2019-06-20 10:54:20" level=debug msg="[doUnlock] Unlock cachecloud-master-sg2-test-0.shopeemobile.com"
time="2019-06-20 10:54:20" level=debug msg="[doUnlock] Unlock 8203dfe19ba10d6eebf83905e26bdf66413a4615abaa9a6cb0d67140b244f354"
time="2019-06-20 10:54:20" level=debug msg="[task] DissociateContainer done"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xda0aac]
goroutine 1719 [running]:
github.com/projecteru2/core/rpc.toRPCDissociateContainerMessage(...)
/go/src/github.com/projecteru2/core/rpc/transform.go:335
github.com/projecteru2/core/rpc.(*Vibranium).DissociateContainer(0xc0001f01c0, 0xc0008f5400, 0x11dc000, 0xc000c0bed0, 0x0, 0x0)
/go/src/github.com/projecteru2/core/rpc/rpc.go:566 +0x19c
github.com/projecteru2/core/rpc/gen._CoreRPC_DissociateContainer_Handler(0x1023480, 0xc0001f01c0, 0x11da620, 0xc0009ce0c0, 0x1adbfc8, 0xc0008e0d00)
/go/src/github.com/projecteru2/core/rpc/gen/core.pb.go:4971 +0x109
github.com/projecteru2/core/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000532c00, 0x11e0080, 0xc000a66900, 0xc0008e0d00, 0xc0005506f0, 0x1a70b60, 0x0, 0x0, 0x0)
/go/src/github.com/projecteru2/core/vendor/google.golang.org/grpc/server.go:1170 +0xacd
github.com/projecteru2/core/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000532c00, 0x11e0080, 0xc000a66900, 0xc0008e0d00, 0x0)
/go/src/github.com/projecteru2/core/vendor/google.golang.org/grpc/server.go:1249 +0xcbe
github.com/projecteru2/core/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0009b4100, 0xc000532c00, 0x11e0080, 0xc000a66900, 0xc0008e0d00)
/go/src/github.com/projecteru2/core/vendor/google.golang.org/grpc/server.go:685 +0x9f
created by github.com/projecteru2/core/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/go/src/github.com/projecteru2/core/vendor/google.golang.org/grpc/server.go:683 +0xa1

AfterCreate hook doesn't work when opts.entrypoints.hook is nil

can't remove pod

lock timeout use same unit

Unsupported method returns no error, may cause misleading success result in grpc call

core/engine/virt/virt.go

Lines 185 to 189 in c5f33e0

	// VirtualizationCopyTo copies one.
	func (v *Virt) VirtualizationCopyTo(ctx context.Context, ID, target string, content io.Reader, AllowOverwriteDirWithFile, CopyUIDGID bool) (err error) {
	log.Warnf("VirtualizationCopyTo does not implement")
	return
	}

Just use eru-cli to call send cmd line and it shows result is successful, but actually it fails cuz this method did not implement.
Maybe return an error would be better to avoid misleading successful message?

Some concurrently printed messages are messed up

The strings are mixed with each other and not readable

`make mock` use huge memory

Eru-core loses track of running Lambda when eru-core is restarted

Eru-core tracks the running of Lambda, but when eru-core is restarted it loses the track of running Lambdas, this could cause long-running Lambda container leak and never get cleaned.

relloc在没操作CPU时，保持容器的CPU bind一致性

realloc 在设置memory时，假如没对cpu进行操作，应该保持cpu bind的一致。

`eru-cli container list` slow than before

Version: 830

Documentation for deploying strategies

Some strategies like FILL, AUTO have changed so the current documentation is outdated. Updating is needed to make it up-to-date.

FILL

In previous versions deployment with no effect will be taken as success.
Now it's taken as a failure.

Reproduce steps:

deploy(1, 1, FILL) -> succ, 1 new instance
deploy(1, 1, FILL) -> fail, 0 new instance
deploy(1, 3, FILL) -> succ, 2 new instance

AUTO

In previous versions node limit took no effect under AUTO strategy. But now it's used as the maximum instances per node besides similar implementation.

When Core start, send all metrics

Core will lost all metrics when restart

[Feature Request] Create Workload with Specific ulimit

docker api already has this arguments: docker run --ulimit <type>=<soft>:<hard>, let's just wrap it.

requested by @jason-joo

build image failed

inherit network incompatible with host network

[ReplaceContainer] Replace and remove failed Error response from daemon: user specified IP address is supported on user defined networks only, old container restarted

[feature request] enable uprobe in eru-core

Refer to containerd binary:

root@localhost:~# nm /usr/bin/containerd
nm: /usr/bin/containerd: no symbols

The symbol tables were dropped.

root@localhost:~# bpftrace -l 'u:/usr/bin/containerd' | grep -i cri | head
uprobe:/usr/bin/containerd:crypto/x509.(*UnhandledCriticalExtension).Error
uprobe:/usr/bin/containerd:crypto/x509.UnhandledCriticalExtension.Error
uprobe:/usr/bin/containerd:github.com/containerd/containerd.(*Client).Subscribe
uprobe:/usr/bin/containerd:github.com/containerd/containerd.(*eventRemote).Subscribe
uprobe:/usr/bin/containerd:github.com/containerd/containerd.(*eventRemote).Subscribe.func1
uprobe:/usr/bin/containerd:github.com/containerd/containerd/api/events.(*ContainerCreate).Descriptor
uprobe:/usr/bin/containerd:github.com/containerd/containerd/api/events.(*ContainerCreate_Runtime).Descriptor
uprobe:/usr/bin/containerd:github.com/containerd/containerd/api/events.(*ContainerDelete).Descriptor
uprobe:/usr/bin/containerd:github.com/containerd/containerd/api/events.(*ContainerUpdate).Descriptor
uprobe:/usr/bin/containerd:github.com/containerd/containerd/api/events.(*ContentDelete).Descriptor

However the uprobes were available!

This move could empower us to use BPF for tracing & debugging the eru-core in live without affecting any performance.

Better CPU Binding Implementation in Process-based Runtimes

现有的 cpu 配额和绑定的 runtime 实现是:

1. cpu-bind=true: 使用 cpuset.cpus + cpu.cfs_quota_us. 比方说请求 cpu=1.2, bind={1,2}, 落实到 OS 层面就 cpuset.cpus=1,2, cpu.cfs_quota_us=120000;
2. cpu-bind=false: 使用 cpu.cfs_quota_us. 比方请求 cpu=1.2, 落实到 OS 层面就是 cpu.cfs_quota_us=120000

这造成了几个问题:

1. cpuset.cpus + cpu.cfs_quota_us 的方案对性能影响非常大. 我做的简单测试, 三进程用这种方案分别绑定 1.2, 1.3, 1.5 cpu, 理论上希望能跑出 4 个 100% 的 cpu, 然而结果是 4 个 cpu 利用率波动在 85%~97%, 而且 cpu throttle 非常频繁.
2. cpu unbind 的进程没有限制 cpuset.cpus, 可能会影响绑核的进程. Kubernetes 的做法是做一个动态的 share cpu pool, 比如初始时是 0-7, 后来若有进程绑定了 cpu 2, 那么 share pool 变成 0-1,3-7, 并更新到已有的非绑定容器上.
3. shopee SRE 的需求, 希望 redis 进程能被分配两个 cpu, 一个专属一个共享, 平时 redis 只用专属, bgsave 的时候才使用共享; 同一个 host 上的所有 redis 进程们可以共享同一个 cpu.

因此新的方案是:

1. 照抄 Kubernetes 的 share pool 方案, 动态把已绑定的 cpu 从里面摘除.
2. cpu-bind=false 时, cpuset.cpus 指定 share pool, 同时设置 cpu.cfs_quota_us; share pool 更新的时候需要对已有进程的 cpuset.cpus 更新一遍
3. cpu-bind=true 时, cpuset.cpus 正常设置, 不再限制 cpu.cfs_quota_us, 而是根据碎片核的占比设置 cpu.shares

比如以下是连续请求:

1. 请求 cpu=1, bind=false, 最终 cpuset.cpus=0-7, cpu.cfs_quota_us=100000
2. 请求 cpu=1, bind=true, 最终 cpuset.cpus=0, cpu.shares=1024 (1024 是默认 share, 相当于没有修改); 之后请求1创建的进程的 cpuset.cpus 被调整为 1-7.
3. 请求 cpu=1.2, bind=true, 最终 cpuset.cpus=1,3, cpu.shares=20; 之后请求 1 创建的进程 cpuset.cpus 再次被调整为 2,4-7.
4. 请求 cpu=1.2, bind=false, 最终 cpuset.cpus=2,4-7, cpu.cfs_quota_us=120000

那么对于 Shopee SRE 的需求, 可以在 redis-zero 保证:

1. 用户在 web 请求 cpu=1, bind=true
2. redis-zero 发送给 eru-core 的请求里修改为 cpu=1.01, bind=true
3. redis 容器被绑上两核, 一个完整核一个碎片核, 对应需求“一个专属核一个共享核”

realloc support bind-cpu & release-cpu

for realloc API we can not bind-cpu if containers did not bind cpus when creating and vice versa.

so, I need realloc API support this.

the rpc layer should add bind-cpu or release-cpu interface, bool flag. then pass this to cluster layer.

In the mean time, dont forget unit-tests, I hope cover rate not decrease after this implementation.

[proposal] better lifecycle management

1. running, stopped, removed 都是由 agent 上报的远程实际状态, 只有 created 是 core 调用 API 成功后就认为成功.
2. running 细分为 healthy / unhealthy, 还是老设计
3. stopped, removed 都是靠监听 docker events
4. unknown 是 core 记录的 status 超时后的状态, agent 没有定时上报

直接动机是使资源管理更加清晰, created 成功即扣资源, 写元数据; 发现 removed 就还数据.

另外的好处是:

1. 改善 zero refresh. 比方说重启 agent 导致 overwatch 的 refresh 风暴, 在细分了生命期和状态机之后, overwatch 可以精准 refresh 具体的状态变化, 对于重启 agent 而言 watch 到的信息是 running -> running, 那就无视就好了.
2. 新增了 unknown 状态, 让我们能够处理节点下线的情况, 之前节点下线只会导致 key expire, 然后收到的是 not running & not healthy 的状态, 无法得知节点下线事件.
3. 新增 removed 状态让我们能够处理外部事件, 比如我手动 docker rm 一个 eru 容器; 在新设计下, remove 事件能被 agent 监听到, 然后 core 能够处理 agent 的上报, 归还资源.
4. 精细化资源管理, 之前只要调用 start api 成功就扣资源, remove api 成功就还资源, 现在完全依赖 agent 上报生命状态, 能够应对更多意外情况.

当然这是一个巨大而激进的改动, 是否走向这个方向还需讨论.

Support core-dump on Docker Container

replace failed if count > 1

safeSplit will discard quoted single word (w/o space)

core/utils/utils.go

Lines 271 to 298 in 30f925f

	func safeSplit(s string) []string {
	split := strings.Split(s, " ")
	var result []string
	var inquote string
	var block string
	for _, i := range split {
	if inquote == "" {
	if strings.HasPrefix(i, "'") || strings.HasPrefix(i, "\"") {
	inquote = string(i[0])
	block = strings.TrimPrefix(i, inquote) + " "
	} else {
	result = append(result, i)
	}
	continue
	}
	if !strings.HasSuffix(i, inquote) {
	block += i + " "
	} else {
	block += strings.TrimSuffix(i, inquote)
	inquote = ""
	result = append(result, block)
	block = ""
	}
	}
	return result
	}

If s has a single quoted word, block will not be appended to result correctly and resulting in absence of this word after split.

e.g. with eru-cli single quoted word:

eru-cli lambda --pod eru echo \"123\"
INFO[2020-11-10 17:07:16] [EruResolver] start sync service discovery
[0778c83]: &&

with non-single quoted word it will be correct:

eru-cli lambda --pod eru echo \"123 456\"
INFO[2020-11-10 17:07:06] [EruResolver] start sync service discovery
[63e1660]: && 123 && 456

Enhancement: Retry count should be 0 or omitted with "no" policy

Background

In business the restart policy needs to be set to no.

Result Expected

Success.

Result Actual

Fail with: maximum retry count could be used with policy "no".

Related Code

container.go:58

// set restart always
restartRetryCount := 3
if opts.RestartPolicy == restartAlways {
    restartRetryCount = 0
}

The retry count should be 0 when using no policy or just omit it.

Workaround

Currently a workaround works that is setting policy to empty instead of setting it to no explicitly. Better to fix it in future in core.

Systemd engine改进

Systemd engine改进

已经实现的API

1. Info
2. VirtualizationCreate
3. VirtualizationCopyTo
4. VirtualizationStart
5. VirtualizationStop
6. VirtualizationRemove
7. VirtualizationInspect

可以添加的API

1. VirtualizationLogs

部署改进 - 把Systemd执行内容打包成标准的分发内容（Docker Image, Tar &&）

• 如果使用Docker Image，镜像可以作为Container执行，也可以把文件抽离后用Systemd执行
• 通过Image分发内容（镜像内包含标准化的systemd执行内容）

1. 实现ImageLocalDigests
2. 实现ImagePull

• 通过Docker Image构建出systemd运行内容

1. 改进VirtualizationCreate

• 构建Systemd镜像,实现以下API

1. BuildRefs
2. BuildContent
3. ImageBuild
4. ImageBuildCachePrune
5. ImageBuildFromExist
6. ImageList
7. ImageRemove
8. ImagesPrune

其他可实现API

1. VirtualizationResize

其他讨论

1. 替换ssh client？在目标机器的容器内一定条件下逃逸掉systemctl指令？好处：统一到docker client实现

strange node set issue

1. user run node set to add resource
2. because cli not use KEEP as default, after set node marked as down.
   https://github.com/projecteru2/core/blob/master/cluster/calcium/node.go#L56
3. user add storage to a node without volume resource, but after set, you can get volume resource --> because cli version not same --> but why storage usage is 0?

stackoverflow when cpu-bind = true on QA env

zero-redis call eru to create a container(cpubind=true) will cause stackoverflow. Seems like the recursion does not end correctly.

version: core:v20.09.30
IP: 10.143.204.21

time="2021-01-16 06:59:02" level=debug msg="[ContainerStatusStream] container 0aace5eb6c387c446cb6a78d2596e749011853907d2bb826f6bbf5b0560bd163 status changed"
time="2021-01-16 06:59:08" level=debug msg="[ContainerStatusStream] container 0aace5eb6c387c446cb6a78d2596e749011853907d2bb826f6bbf5b0560bd163 status changed"
time="2021-01-16 06:59:08" level=debug msg="[ContainerStatusStream] container f437f7b7bed24b5cefd04c0dd3b2e5e38a0251e7c458b310e805059d0ae5cc0b status changed"
time="2021-01-16 06:59:48" level=debug msg="[task] CalculateCapacity added"
time="2021-01-16 06:59:48" level=debug msg="[withNodesLocked] Node redisslave locked"
time="2021-01-16 06:59:48" level=warning msg="[MakeDeployStatus] Deploy status not found redissingular.redis14783"
time="2021-01-16 06:59:48" level=info msg="[SelectCPUNodes] nodesInfo 1, need cpu: 1.000000 memory: 10485760"
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc05a7003c8 stack=[0xc05a700000, 0xc07a700000]
fatal error: stack overflow

runtime stack:
runtime.throw(0x14c11d5, 0xe)
        /usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.newstack()
        /usr/local/go/src/runtime/stack.go:1067 +0x78d
runtime.morestack()
        /usr/local/go/src/runtime/asm_amd64.s:449 +0x8f

goroutine 16443964 [running]:
runtime.heapBitsSetType(0xc099a8ad50, 0x30, 0x30, 0x1446380)
        /usr/local/go/src/runtime/mbitmap.go:911 +0xaa5 fp=0xc05a7003d8 sp=0xc05a7003d0 pc=0x4190a5
runtime.mallocgc(0x30, 0x1446380, 0x1, 0x0)
        /usr/local/go/src/runtime/malloc.go:1090 +0x5a5 fp=0xc05a700478 sp=0xc05a7003d8 pc=0x40f5a5
runtime.newobject(0x1446380, 0x8)
        /usr/local/go/src/runtime/malloc.go:1195 +0x38 fp=0xc05a7004a8 sp=0xc05a700478 pc=0x40fcd8
runtime.makemap_small(0x30)
        /usr/local/go/src/runtime/map.go:293 +0x2d fp=0xc05a7004c8 sp=0xc05a7004a8 pc=0x41090d
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7ff60, 0x1, 0x1, 0xc099a90078, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:189 +0x305 fp=0xc05a7005b0 sp=0xc05a7004c8 pc=0x9da725
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7ff40, 0x1, 0x1, 0xc099a90070, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700698 sp=0xc05a7005b0 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7ff20, 0x1, 0x1, 0xc099a90068, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700780 sp=0xc05a700698 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7ff00, 0x1, 0x1, 0xc099a90060, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700868 sp=0xc05a700780 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fee0, 0x1, 0x1, 0xc099a90058, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700950 sp=0xc05a700868 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fec0, 0x1, 0x1, 0xc099a90050, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700a38 sp=0xc05a700950 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fea0, 0x1, 0x1, 0xc099a90048, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700b20 sp=0xc05a700a38 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fe80, 0x1, 0x1, 0xc099a90040, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700c08 sp=0xc05a700b20 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fe60, 0x1, 0x1, 0xc099a90038, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700cf0 sp=0xc05a700c08 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fe40, 0x1, 0x1, 0xc099a90030, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700dd8 sp=0xc05a700cf0 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fe20, 0x1, 0x1, 0xc099a90028, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700ec0 sp=0xc05a700dd8 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fe00, 0x1, 0x1, 0xc099a90020, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a700fa8 sp=0xc05a700ec0 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fde0, 0x1, 0x1, 0xc099a90018, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a701090 sp=0xc05a700fa8 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fdc0, 0x1, 0x1, 0xc099a90010, 0x0, 0x1)
        /go/src/github.com/projecteru2/core/scheduler/complex/resource.go:202 +0x445 fp=0xc05a701178 sp=0xc05a701090 pc=0x9da865
github.com/projecteru2/core/scheduler/complex.(*host).getFullResult(0xc000a2f840, 0x1, 0xc099a7fda0, 0x1, 0x1, 0xc099a90008, 0x0, 0x1)

Weird types mapping

Type 1: pb.CapacityMessage
Type 2: types.CapacityMessage

map[string]int64 <====> map[string]int

So weird mapping. Who's kind enough to fix this?

@CMGS @tonicbupt

realloc cpu-unbind not work

due to docker's issue, we can bind cpu after call realloc API. However after CPU bond, can not unbind.

etcdlock.Mutex 的实现并不是一个行为上正确的分布式锁

我们这里使用的是 concurrency.NewSession(cli, concurrency.WithTTL(ttl)).
这个 session 是对 lease 的一个 wrapper, 会一直去 refresh lease 的 ttl, 只要这个 session 还存在, 没有断开或者没有过期, lease 就会一直被刷新.

这个就导致了 func New(cli *clientv3.Client, key string, ttl time.Duration) (*Mutex, error) 这个函数的 ttl 会迷惑用户, 这个 ttl 并不是非正确释放的情况下的自动释放的时间... 这个 ttl 是说 session 断开了或者锁被释放之后, 再过最多多久 lease 会被释放...

如下代码并不是预期结果:

m1, _ := New(cli, "test", 5*time.Second)
m2, _ := New(cli, "test", 5*time.Second)
m1.Lock(ctx)
m2.Lock(ctx)

m2 会超时退出, 而不是因为 5s 后 m1 被自动释放所以 m2 顺延拿到锁.

所以建议要么改成正确实现, 要么直接屏蔽掉这个 ttl, 默认给给最小值, 在一个持有者非正常退出之后会被很快释放.

realloc failed cause node resource diff

realloc container failed -> node resource got diff

opti: Stop is too slow

Description

We have frequently large bundle of nodes to create and destroy recently.
But I found that it's very inefficient in the stopping step(maybe also in removing but it's much more efficient)

Optimization

Is it possible to turn it from serializing into batches?

启动的时候会做一些一致性检查以及修复和清理相关的工作吗？

eru-core 声称是无状态的，那么如果异常退出的话，是不是有可能：

1. 出现资源被分配了，但是实际上容器没起来的情况？
2. 容器起来了，但是资源没被实际分配？

出于好奇扫了下启动相关的代码，貌似没看到对 etcd 数据进行重放并做一些修复或者清理相关的工作

artifacts should be refactor

1. no more unzip
2. consider raw artifacts, without clone bla bla

watch service status failed

bug: `eru-cli lambda` terminates without any error output when failed

Description

eru-cli would terminate without any outputs if it actually failed (infer from the logging of eru-core). For example use a wrong pod to run a lambda:

eru-cli lambda --pod masterasdf --name jason-test ls 

That the pod masterasdf actually doesn't exist.

Expected Result

The error like not enough nodes in the example above should be returned to eru-cli.

request: hope `GetWorkload` can return `Env` of `Workload`

Inside Eru core, the Workload.Env is visible, but via RPC, the Env is discarded and not visible to client, while some clients need this Env

avoid get node info blocking

ResourceOptions's CpuBind is useless in Realloc

type ReallocOptions struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    Id           string           `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
    BindCpuOpt   TriOpt           `protobuf:"varint,2,opt,name=bind_cpu_opt,json=bindCpuOpt,proto3,enum=pb.TriOpt" json:"bind_cpu_opt,omitempty"`
    ResourceOpts *ResourceOptions `protobuf:"bytes,3,opt,name=resource_opts,json=resourceOpts,proto3" json:"resource_opts,omitempty"`
}

type ResourceOptions struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    CpuQuotaLimit   float64  `protobuf:"fixed64,1,opt,name=cpu_quota_limit,json=cpuQuotaLimit,proto3" json:"cpu_quota_limit,omitempty"`
    CpuQuotaRequest float64  `protobuf:"fixed64,2,opt,name=cpu_quota_request,json=cpuQuotaRequest,proto3" json:"cpu_quota_request,omitempty"`
    CpuBind         bool     `protobuf:"varint,3,opt,name=cpu_bind,json=cpuBind,proto3" json:"cpu_bind,omitempty"`
    MemoryLimit     int64    `protobuf:"varint,4,opt,name=memory_limit,json=memoryLimit,proto3" json:"memory_limit,omitempty"`
    MemoryRequest   int64    `protobuf:"varint,5,opt,name=memory_request,json=memoryRequest,proto3" json:"memory_request,omitempty"`
    StorageLimit    int64    `protobuf:"varint,6,opt,name=storage_limit,json=storageLimit,proto3" json:"storage_limit,omitempty"`
    StorageRequest  int64    `protobuf:"varint,7,opt,name=storage_request,json=storageRequest,proto3" json:"storage_request,omitempty"`
    VolumesLimit    []string `protobuf:"bytes,8,rep,name=volumes_limit,json=volumesLimit,proto3" json:"volumes_limit,omitempty"`
    VolumesRequest  []string `protobuf:"bytes,9,rep,name=volumes_request,json=volumesRequest,proto3" json:"volumes_request,omitempty"`
}

BindCpuOpt is conflict with ResourceOptions's CpuBind, and the latter seems useless.

使用lambda执行容器命令时，希望可以取代dockerfile的CMD

如：
cli lambda --pod eru "sh -c \"ping yonghaoTCPServer\" " 时，希望不是执行dockerfile里写死的CMD，而是我的sh -c \"ping yonghaoTCPServer\"命令。

deploy strategy each/fill failed

improve: Refine error message during creation of container

Background

The error message during creation of the containers is implicit:

[CreateContainer] create container failed Error response from daemon: network cloud-live-sg4 not found

It's hard to locate the node complaining the error above. So maybe introduce the node name at least is better for locating the issue.

Proposal

Introduce more details into the error description to make it easy to locate the complaining node, for example:

[CreateContainer] create container failed Error response from daemon: network cloud-live-sg4 not found on sg4-node-10-10-33-34

Sure a more fine message is also acceptable.

exec 接口区分 stdout/stderr (docker engine)

The reliability on agent in GetWorkload()/GetWorkloads()

Let's focus on the publish property of a workload.

Generally the network info would be available only after agent detects and saves it into the store. So it won't be available if the agent hung.

But in CreateWorkload/ReplaceWorkload we could find that it's more like a realtime property:

			// inspect real meta
			var workloadInfo *enginetypes.VirtualizationInfo
			workloadInfo, err = workload.Inspect(ctx) // 补充静态元数据
			if err != nil {
				return errors.WithStack(err)
			}

			// update meta
			if workloadInfo.Networks != nil {
				msg.Publish = utils.MakePublishInfo(workloadInfo.Networks, opts.Entrypoint.Publish)
			}

And I remember in the previous implementations the returned structure of GetContainer() has such thing like inspect data enabling the users to do similar thing.
So is it possible to do similar thing in GetWorkload like CreateWorkload?

when nodename and appname are both given to list container, nodename is ignored

i thought this would be "listing containers on this node with this appname", but it's actually "listing containers with this appname, nodename? just ignore that" ...

resize container(has volume) failed

&types.DeployOptions{
  Name: "zerox",
  Entrypoint: &types.Entrypoint{
    Name: "rdssingular",
    Command: "zero-redis --task-id RedisClusterInstanceResize-1594698023 --user [email protected] --config /etc/zero-redis.yaml --name T0713001 cluster resize  --m
emory 1024MB ",
    Privileged: false,
    Dir: "/",
    Log: nil,
    Publish: nil,
    HealthCheck: nil,
    Hook: nil,
    RestartPolicy: "",
    Sysctls: map[string]string{
    },
  },
  Podname: "slave",
  Nodename: "",
  Image: "harbor.shopeemobile.com/cloud/zero-redis:latest",
  ExtraArgs: "",
  CPUQuota: 0.5,
  CPUBind: false,
  Memory: 52428800,
  Storage: 0,
  Count: 1,
  Env: nil,
  DNS: nil,
  ExtraHosts: nil,
  Volumes: nil,
  Networks: map[string]string{
  },
  NetworkMode: "host",
  User: "root",
  Debug: false,
  OpenStdin: false,
  Labels: map[string]string{
  },
  NodeLabels: map[string]string{
  },
  DeployMethod: "auto",
  Data: map[string]*bytes.Reader{
    "/etc/zero-redis.yaml": &bytes.Reader{},
    "/usr/bin/redis-cli": &bytes.Reader{},
  },
  SoftLimit: false,
  NodesLimit: 0,
  ProcessIdent: "gVSddWYgZVgEHhvn",
  IgnoreHook: false,
  AfterCreate: nil,
  RawArgs: nil,

err msg:

time="2020-07-14 15:26:55" level=error msg="[VirtualizationUpdateResource] docker engine not support rebinding volume resource: [/data/log/platform/redis-instance-test
/T0714001/slave/598df0c9-cea8-4a5b-806c-2f404bb9533d/:/data/log /data/log/platform/redis-instance-test/T0714001/slave/791dda80-1fe4-4e81-bc44-b8d6b89406fd/:/data/log]"
time="2020-07-14 15:26:55" level=error msg="[updateResource] When Realloc container, VirtualizationUpdateResource fd454a7a4a3ec4b3d7877a812e1ab51c19cfee72b09410f84f61f2105b953a2d failed Not Support"

the memory calculate algorithm cause core can distribute more memory than server memory

here:
if opts.Memory == 0 {
opts.Memory = info.MemTotal * 10 / 8 // use 80% real memory
}

Lambda's output should be readable