ehazlett / stellar Goto Github PK

Add the ability to issue an image pull for the cluster.

Does stellar support fail-over? I'm able to deploy containers to specific nodes, but it's unclear how or if these get replicated or scaled.

Btw - Your video was very informative. Where can I find part II?
https://www.youtube.com/watch?v=diRmI45Yhg4

I tried to specify an endpoint for a UDP application like this:

"endpoints": [
    {
        "service": "dns",
        "protocol": "udp",
        "port": 53
    }
]

But it doesn't seem to work. Is the feature implemented or do I've to install Radiant?

There is a race condition when an app is created, deleted and re-created quickly. There are tombstone records created for the delete which do not get resolved until after the re-create which then remove the valid records in the nameserver service.

Note: due to the eventual consistency this will get resolved on the next datastore replication but can cause issues in the meantime. A possible solution is to check for tombstone records when creating and remove the tombstone to prevent the delete.

Running into the following error on ARM when attempting to create an app:

sctl --addr 10.0.1.123:9000 apps create -f example.conf
FATA[0000] rpc error: code = Unknown desc = failed to find plugin "loopback" in path [/usr/local/bin /usr/local/bin/cni]

Here's my stellar.conf

{
    "ConnectionType": "local",
    "ClusterAddress": "10.0.1.123:7946",
    "AdvertiseAddress": "10.0.1.123:7946",
    "Debug": false,
    "NodeID": "rpi-cluster-03",
    "GRPCAddress": "10.0.1.123:9000",
    "TLSServerCertificate": "",
    "TLSServerKey": "",
    "TLSClientCertificate": "",
    "TLSClientKey": "",
    "TLSInsecureSkipVerify": false,
    "ContainerdAddr": "/run/containerd/containerd.sock",
    "Namespace": "default",
    "DataDir": "/var/lib/stellar",
    "StateDir": "/run/stellar",
    "Bridge": "stellar0",
    "UpstreamDNSAddr": "10.0.1.1:53",
    "ProxyHTTPPort": 80,
    "ProxyHTTPSPort": 443,
    "ProxyTLSEmail": "",
    "GatewayAddress": "127.0.0.1:9001",
    "EventsAddress": "10.0.1.123:4222",
    "EventsClusterAddress": "10.0.1.123:5222",
    "EventsHTTPAddress": "10.0.1.123:4322",
    "CNIBinPaths": [
        "/usr/local/bin",
        "/usr/local/bin/cni"
    ],
    "Peers": [],
    "Subnet": "172.16.0.0/12",
	"ProxyHealthcheckInterval": "5s"
}

The example config's CNIBinPaths appeared to specify containerd and cni binaries. I grabbed the latest release of cni for ARM and your ARM build of containerd. Here's my /usr/local/bin:

|-- cni
|   |-- cnitool
|   `-- noop
|-- containerd
|-- containerd-shim
|-- containerd-shim-runc-v1
|-- containerd-shim-runc-v2
|-- ctr
|-- sctl
|-- stellar
`-- stellar-cni-ipam

Am I missing a cni plugin for this to work? Thanks for all of your help so far!

Hi,

i tested stellar on 2 VMs before going on baremetal. Everything fine - now on baremetal i get the following error log

$ stellar config --nic enp3s0 > stellar.config
$ stellar -D server --config ./stellar.config
DEBU[0000] seed peers seedPeers="[]"
DEBU[0000] getPeersFromCache: []
DEBU[0000] cluster peers peers="[]" seed_peers="[]"
INFO[0000] registered service id=stellar.services.version.v1
INFO[0000] registered service id=stellar.services.node.v1
INFO[0000] registered service id=stellar.services.health.v1
INFO[0000] registered service id=stellar.services.cluster.v1
INFO[0000] registered service id=stellar.services.datastore.v1
INFO[0000] registered service id=stellar.services.gateway.v1
INFO[0000] registered service id=stellar.services.network.v1
INFO[0000] registered service id=stellar.services.application.v1
INFO[0000] registered service id=stellar.services.nameserver.v1
INFO[0000] registered service id=stellar.services.proxy.v1
INFO[0000] registered service id=stellar.services.events.v1
DEBU[0000] starting server agent
DEBU[0000] starting grpc server addr="172.16.0.6:9000"
DEBU[0000] initializing server
DEBU[0000] network init
DEBU[0000] allocating network subnet for node y
DEBU[0000] service.network allocating subnet
FATA[0000] invalid CIDR address:

The configfile sounds similar:

{
"NodeID": "y",
"GRPCAddress": "172.16.0.6:9000",
"TLSServerCertificate": "",
"TLSServerKey": "",
"TLSClientCertificate": "",
"TLSClientKey": "",
"TLSInsecureSkipVerify": false,
"ContainerdAddr": "/run/containerd/containerd.sock",
"Namespace": "default",
"DataDir": "/var/lib/stellar",
"StateDir": "/run/stellar",
"Bridge": "stellar0",
"UpstreamDNSAddr": "8.8.8.8:53",
"ProxyHTTPPort": 80,
"ProxyHTTPSPort": 443,
"ProxyTLSEmail": "",
"GatewayAddress": "172.16.0.6:9001",
"EventsAddress": "172.16.0.6:4222",
"EventsClusterAddress": "172.16.0.6:5222",
"EventsHTTPAddress": "172.16.0.6:4322",
"CNIBinPaths": [
"/opt/containerd/bin",
"/opt/cni/bin"
],
"ConnectionType": "local",
"ClusterAddress": "172.16.0.6:7946",
"AdvertiseAddress": "172.16.0.6:7946",
"Debug": false,
"Peers": [],
"Subnet": "172.16.0.0/12",
"ProxyHealthcheckInterval": "5s"
}

I tried to modify the subnet, reviewed the code where the output comes from but i cannot get the cause of this failure :-(

Thanks, Martin

In building the network service we need to decide how to handle IP assignment.

1. Should we assign an IP to every container or just certain ones?
2. When assigning do we only assign containers and applications that are launched through Stellar or any container in the namespace?

Currently it's not possible to check the logs of an application (unless we use ctr) launched by stellar, it would be great to add a command like this one to be able to fetch the logs about the application from any node: sctl apps logs <appname>.

The Makefile is hardcoded to use amd64. What are the chances this could be built for arm?

I am setting up stellar and configuring the first vm. I got this error when trying:

DEBU[0000] starting server agent
DEBU[0000] starting grpc server                          
DEBU[0000] initializing server
DEBU[0000] network init
DEBU[0000] allocating network subnet for node 
DEBU[0000] service.network allocating subnet
DEBU[0000] setting up subnet 172.16.0.0/22
DEBU[0000] setting up local gateway 172.16.0.1
DEBU[0000] setting up gateway
DEBU[0000] bind interface: eth0
FATA[0000] operation not permitted

Currently there is no way to clean up containerd automatically just like we would with Docker.
Is it possible to introduce a manual garbage collector for the images, containers, snapshots, content (ctr content) and more (?)?
The command could be like this: sctl gc

Occasionally the proxy service can report duplicate upstreams. The nameserver service appears to have the correct data so this looks like a bug in the local query to get applications:

Proxy

foo.com              http://172.16.8.4:80 (up: 479.478µs), http://172.16.0.4:80 (up: 947.151µs), http://172.16.4.5:80 (up: 826.847µs), http://172.16.8.4:80 (up: 2.05114ms)

Nameserver Records

blog.app00.stellar       A                   172.16.0.4
blog.app00.stellar       TXT                 node=ctr-00; updated=2018-10-24T23:42:16-04:00
blog.app00.stellar       SRV                 blog.app00.stellar                               service=web proto=http priority=0 weight=0 port=80
blog.app01.stellar       A                   172.16.8.4
blog.app01.stellar       TXT                 node=ctr-02; updated=2018-10-24T23:42:21-04:00
blog.app01.stellar       SRV                 blog.app01.stellar                               service=web proto=http priority=0 weight=0 port=80
blog.app02.stellar       A                   172.16.4.5
blog.app02.stellar       TXT                 node=ctr-01; updated=2018-10-24T23:42:28-04:00
blog.app02.stellar       SRV                 blog.app02.stellar                               service=web proto=http priority=0 weight=0 port=80

Does this support MacOS ? I would like to use it for Mac desktops in order to run workloads on our Macs :)

It seems contanerd suppors windows via the Windows hyper shim.

There is an issue with the update detection that causes the proxy service to reload often. There is no apparent negative effects as the reloads are cheap but it should be improved. Perhaps switch to a cluster event system for notifications.

Right now most of the tests are unit based. We need to add integration tests.

I wanted to built stellar myself with the built via Docker option (because there are no armhf binaries) and got this

-> building daemon e70a943-dev
../../vendor/github.com/mholt/caddy/caddytls/client.go:30:2: cannot find package "github.com/xenolf/lego/acme" in any of:
        /go/src/github.com/ehazlett/stellar/vendor/github.com/xenolf/lego/acme (vendor tree)
        /usr/local/go/src/github.com/xenolf/lego/acme (from $GOROOT)
        /go/src/github.com/xenolf/lego/acme (from $GOPATH)
make: *** [Makefile:50: daemon] Error 1
 -> Built dev version e70a943 (linux/amd64)

I now manually added a go get ... but now I get

# github.com/ehazlett/stellar/vendor/github.com/mholt/caddy/caddytls
../../vendor/github.com/mholt/caddy/caddytls/config.go:122:10: undefined: acme.KeyType
make: *** [Makefile:50: daemon] Error 2

Hi,

i love the idea of a lightweight orchestrator instead of an overblown kubernetes.
I just created a 2 node cluster within virtual machines to play around if i am able to switch a kubernetes cluster over to stellar....
I also ran over #21 :-D but got the example running.

The mount options is on the same level as docker is using right? Do you have a sample with nfs mounts? How do you handle this stuff?

Thanks Martin!

PS: I gonna prepare a ansible role to setup a node/cluster - i hate doing repeatable tasks manually..

At the part where Stellar tried to build the local route and get the error below:

DEBU[0005] services started                              duration=5.0005632s
ERRO[0005] timeout starting service stellar.services.proxy.v1
DEBU[0010] skipping local route 172.16.0.0/22
ERRO[0010] rpc error: code = Unknown desc = failed to dial "/run/containerd/containerd.sock": context deadline exceeded
DEBU[0020] skipping local route 172.16.0.0/22
DEBU[0030] skipping local route 172.16.0.0/22
DEBU[0040] skipping local route 172.16.0.0/22
DEBU[0050] skipping local route 172.16.0.0/22
DEBU[0060] skipping local route 172.16.0.0/22
DEBU[0060] pruning datastore

The last parts of the message just continue until I stop it.

Currently if I want to make a cluster of nodes connected using the internet (not trough a private network like a VPN) anyone can join my cluster because the AdvertiseAddress is exposed on the internet.
It would be great to add an authentication layer with a token or a certificate so that only trusted node can participate in the cluster.

I found here #8 that it's possible to interact with the API using HTTP but I don't really know the endpoints of that API, is it possible to help me how to find them or maybe add swagger for an easy use of the API?

When deleting an application the corresponding nameserver records are not removed.

Hi,
I have a "containerize apps" project and use a makefile to run cobra commands for building app binaries, then an image, then run a container.
Your makefile capability is impressive and I'm keen to adopt some of your design, especially the package target.
What is your intended use case for the package target?

Hi! I'm diving into the code base of stellar in efforts to learn more about containers and go, but in some places that I get stuck, I wonder if there is a chat room I can discuss in? If there is not one currently established, are there any plans to make one?

If wanted I can help with this....

Seems like building is not working via Docker make docker-generate. I'm not familar enough with protoc to know what it means by the plugin failure.

I ran this on two machines to ensure it can be replicated: AWS Instance and my MBP.

[snip]
go: finding github.com/mholt/caddy v0.0.0-20180907212407-13a54dbdda25
 -> building cli 42605bd-dev
 -> building cni-ipam 42605bd-dev
 -> Built dev version 42605bd (linux/amd64)
Removing intermediate container d25e59d175b5
 ---> 55835e7cb1df
Successfully built 55835e7cb1df
Successfully tagged stellar-dev:latest
protoc-gen-grpc-gateway: program not found or is not executable
--grpc-gateway_out: protoc-gen-grpc-gateway: Plugin failed with status code 1.
2018/11/06 17:29:13 protoc -I.:/go/src/github.com/ehazlett/stellar/vendor/github.com/gogo/protobuf:/go/src/github.com/ehazlett/stellar/vendor/github.com/gogo/googleapis:/go/src/github.com/ehazlett/stellar/vendor:/go/src:/usr/local/include:/usr/include --grpc-gateway_out=import_path=github.com/ehazlett/stellar/api/services/application/v1,Mgogoproto/gogo.proto=github.com/gogo/protobuf/gogoproto,Mgoogle/api/annotations.proto=github.com/gogo/googleapis,Mgoogle/api/http.proto=github.com/gogo/googleapis,Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/descriptor.proto=github.com/gogo/protobuf/protoc-gen-gogo/descriptor,Mgoogle/protobuf/duration.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/empty.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types:/go/src /go/src/github.com/ehazlett/stellar/api/services/application/v1/application.proto
make: *** [Makefile:38: generate] Error 123
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
make: *** [docker-generate] Error 1

On Nomad and on docker with watchtower it's possible to automatically update the app if a new image from the Docker hub comes up.
I would love to see that feature implemented and maybe an automatic revert to the previous working container if the update fail?

Hi Evan,

First of all, thank you so much for this project, this is just unbelievably great. I am an ops guys trying to learn go, been researching around containerd and ended-up on your awesome project.

I have installed the latest CNI version following a cillium project doc (https://cilium.readthedocs.io/en/v1.1/kubernetes/install/) as I did not have loopback before.

I thought of raising this issue to ask you what would be the recommended approach, like updating the plugin to use 0.3.1 or for me to downgrade the CNI (couldn't find the right key to download the 0.3.0 version yet) ?

jralmaraz@jralmaraz-vn:~/git/stellar$ sudo sctl --addr 127.0.0.1:9000 apps create -f ./example.conf
FATA[0000] rpc error: code = Unknown desc = incompatible CNI versions; config is "0.3.1", plugin supports ["0.1.0" "0.2.0" "0.3.0"]

Thanks in advance and happy new year !

Cheers,

Jose

Currently when using multiple nodes behind a frontend proxy (i.e. TCP) certificates can take a little while to retrieve as the acme challenge can bounce around to different nodes until it lands on the correct node. There are a couple things we could possibly do:

• Replicate the certificate storage from a single node
• Implement a custom TLS storage asset provider for the embedded Caddy