sonm-io / core Goto Github PK

View Code? Open in Web Editor NEW

358.0 75.0 75.0 39.39 MB

Decentralized Fog Computing Platform

Home Page: https://sonm.com

License: GNU General Public License v3.0

Makefile 0.29% Go 90.25% Shell 0.12% JavaScript 6.66% Solidity 2.28% Rust 0.40%

docker golang ethereum solidity sonm

core's Introduction

SONM Core

Official core client

Documentation

Full documentation and frequently asked questions are available here.

Install

You can download binaries from the releases page.

Packages for Debian and Ubuntu are available at packagecloud.

Build

If you want to build all components by yourself:

git clone https://github.com/sonm-io/core.git
make build

core's People

Contributors

Stargazers

Watchers

Forkers

gitter-badger antmat johndpope leanderdulac elhomosiguiente codeaudit amadeobrands lifego obitel-forks pdtmi sakartvelo crystalsilver w1r2p1 marplemr teotikalki keenanhutchens rue-foundation rhoytdesign xeaon jekamas dghadi1 neuroidss soros1321 lijiawei16 pradyumanraj14 skynet hfbustillo maxximo1 spespe zippynetworks lichardchew drasted alekseygrud fyesgo skudasov herodex abefimov anthonyakentiev raghu999 chen0031 reed-lau jakobsn edisonandrade chillalittle kmeaw sonmcommunity swc1992 oako-enterprise jackyin68 yanb25 avsigaev seif-abaza ashsaccade maxmalysh new-economy sokel mingard ceased-blockchain-element hutian280 wrfcoin nrjcs robertdigital gridio ibrokhim-command dragonxu oako-uconn rohansaphal97 fearblackcat passysosysmas crypto-zone ayoubjadouli pinkdiamond1 testwill rsivakov voronkovventures

core's Issues

Add environment variables into task.yaml

I am currently working on getting a CS:GO server up and running with Sonm. I have the docker container working, and it actually works in Sonm, but I can't properly pass in some variables to make it work 100%.

CS:GO server requires a Steam Token to work in non-lan mode. When I run docker normally, I used -e GSLT=<token> to pass it in. or I pass it in as part of the command docker run csgo /home/csgoserver/start.sh <token>

Either of these methods would work, but not currently possible with the tasks yaml. I looked through the code, and it seems you have tests for a command config value in the tasks yaml, but it is not hooked up or get passed into the container configs in insomnia. As for ENV the only place I see that is in Exec for the overseer or ExecCmd in the container. Neither of which seem to be used fro the tasks yaml method of launching.

Could we get ENV variables hooked up? I can try to submit a pull request myself, but I haven't had time to familiarize myself with the code, so it might be a while before I can comfortably provide a pull request.

Example env config:

task:
  container:
    # image name to start, requried
    name: daegalus/docker-csgo:latest
    env:
      - "GSLT=<token>"
      - "GAMEPORT=27015"
  resources:
      # number of CPU cores required by task, required param
      CPU: 1
      # amount of memory required by task, required param
      # You may use Kb, Mb and Gb suffixes
      RAM: 50Mb

https://docs.docker.com/engine/api/v1.29/#operation/ContainerCreate (Env)

Miner set restrictions on resource pool

Launch containers using parent cgroups
Setup parent cgroup, where mounted, permissions
Rebuild cgroup on restart
Report resources on handshake

Build failing on CentOS 7.4

Hi there, I'm trying to build on CentOS 7.4 but am hitting the follow error after 'make build':

[ira@localhost core]$ make build
+ build/bootnode
go build -tags "nocgo" -ldflags "-s -X main.version=v0.2.1.1-4a5f989" -o sonmbootnode ./cmd/bootnode
+ build/hub
go build -tags "nocgo" -ldflags "-s -X main.version=v0.2.1.1-4a5f989" -o sonmhub ./cmd/hub
# _/home/ira/repos/core/cmd/hub
cmd/hub/main.go:43: cannot use logger (type *"github.com/sonm-io/core/vendor/go.uber.org/zap".Logger) as type *"go.uber.org/zap".Logger in argument to ctxlog.WithLogger
make: *** [build/hub] Error 2

Any tips?

Something wrong with blockchain tests

TestDeployHubWallet fails with following errors:

TESTRPC="/Users/alex/gopath/src/github.com/sonm-io/core/blockchain/./node_modules/ethereumjs-testrpc/build/cli.node.js" go test ./tests
--- FAIL: TestDeployHubWallet (0.01s)
        Error Trace:    HubWallet_test.go:20
    Error:          Should be empty, but was failed to retrieve account nonce: json: cannot unmarshal hex number with leading zero digits into Go value of type hexutil.Uint64
    Messages:       should deploy SonmDummyToken successfully
        Error Trace:    HubWallet_test.go:30
    Error:          Should be empty, but was failed to retrieve account nonce: json: cannot unmarshal hex number with leading zero digits into Go value of type hexutil.Uint64
    Messages:       should deploy HubWallet successfully
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x43b5d81]

goroutine 35 [running]:
testing.tRunner.func1(0xc4202500d0)
    /usr/local/Cellar/go/1.8.3/libexec/src/testing/testing.go:622 +0x29d
panic(0x4439560, 0x474a640)
    /usr/local/Cellar/go/1.8.3/libexec/src/runtime/panic.go:489 +0x2cf
github.com/sonm-io/core/blockchain/tests.TestDeployHubWallet(0xc4202500d0)
    /Users/alex/gopath/src/github.com/sonm-io/core/blockchain/tests/HubWallet_test.go:32 +0x3d1
testing.tRunner(0xc4202500d0, 0x44fa748)
    /usr/local/Cellar/go/1.8.3/libexec/src/testing/testing.go:657 +0x96
created by testing.(*T).Run
    /usr/local/Cellar/go/1.8.3/libexec/src/testing/testing.go:697 +0x2ca
FAIL    github.com/sonm-io/core/blockchain/tests    1.091s
make[1]: *** [test] Error 1
make: *** [test] Error 2

I have no idea how to fix it and it is a reason why I am opening this issue :)

Metrics collection from miner (CPU, Memory)

Collect metrics collection from miner (CPU, Memory), report to hub.

Metrics from Docker (used CPU, Memory).
Metrics from OS (total CPU, Memory).
Communication through gRPC.
Hub requests report.
Miner provide report.

Refactor proto-files

I think we must refactor proto-files to make structures names more evident

Fix types

Fix types
Correct readme

Enable GPU detection

Done in #64.

Basic scheduling

This includes miner's resources aggregation on a hub.

After this issue is done it's no longer required to specify a miner (still, sometimes handy) to start processing a userland load.

Refactor smart contracts using registry pattern

It is possible to replace bunch of our contracts with single registry contract.
This will greatly reduce gas usage and (I hope so) will make code more concise, clear and easier to test.

Hub panics when starting with dev consul

⇒  ./sonmhub
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x436c9f5]

goroutine 1 [running]:
testing.(*T).Log(0x0, 0xc42027ac10, 0x1, 0x1)
	<autogenerated>:9 +0x5
github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/retry.run.func1()
	/Users/esafronov/go/src/github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/retry/retry.go:116 +0x110
github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/retry.(*Timer).NextOr(0xc42023b4a0, 0xc42021eaa0, 0xc42027ab50)
	/Users/esafronov/go/src/github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/retry/retry.go:192 +0x120
github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/retry.run(0x4ba4f80, 0xc42023b4a0, 0x4bab480, 0x0, 0xc42021ea60)
	/Users/esafronov/go/src/github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/retry/retry.go:120 +0xd4
github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/retry.Run(0x4bab480, 0x0, 0xc42021ea60)
	/Users/esafronov/go/src/github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/retry/retry.go:85 +0x74
github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil.NewTestServerConfigT(0x0, 0xc42021ea40, 0x4693f60, 0xc420051901, 0xc42021ea40)
	/Users/esafronov/go/src/github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/server.go:210 +0xcf
github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil.NewTestServerConfig(0xc42021ea40, 0xc42021ea40, 0xc420051930, 0x449d2e8)
	/Users/esafronov/go/src/github.com/sonm-io/core/vendor/github.com/hashicorp/consul/testutil/server.go:195 +0x34
github.com/sonm-io/core/insonmnia/hub.newDevConsul(0x4baf100, 0xc42001b770, 0xc42001b770, 0x4baf100, 0xc42001b770)
	/Users/esafronov/go/src/github.com/sonm-io/core/insonmnia/hub/consul.go:52 +0x89
github.com/sonm-io/core/insonmnia/hub.New(0x5a612c0, 0xc42001b770, 0xc420095b80, 0x48b1630, 0x10, 0x0, 0x40116d2, 0x4c8fb00)
	/Users/esafronov/go/src/github.com/sonm-io/core/insonmnia/hub/server.go:729 +0x9b0
main.main()
	/Users/esafronov/go/src/github.com/sonm-io/core/cmd/hub/main.go:46 +0x1aa

Feature: Allow users to connect to the network through a proxy (e.g. socks5)

Hub still hangs forever on SIGINT

Probably, in the consul.

Add container name to miner info

It will be better to show container name like "username/containername:tag" as docker ps do;

travis build do not run tests

I am suggesting to change travis script from make build to make all.

Cleanup vendoring and repair `make install`

After merging, build instructions is outdated;

Remove broken dependencies
Add vendor libs to repo
Repair make make install
Write instructions: how to use vendoring
Write instructions: how to build core (hub + miner + cli)

Auto select miner based on devices wish-list

This requires some specification attached to the application. Like (simplified):

hardware:
  cpu: [Intel Core i5, Intel Core i7]
  gpu: [NVIDIA Tesla]

Mac setup

Does the alpha support mac? if so how? I tried setting it up but i found no download on the release page.

Allow to explicitly set hub address for miner

Disabling discovery and use statically assigned address will be very useful for local debugging

Take into consideration user bandwidth limits

From running the Alpha release it looks like there is a significant amount of network traffic that is generated by the miner. Some users have very limited bandwidth capabilities (e.g. 1Mbs/1Mbs), witch might cause issues like lag, timeouts...

Network accounting

The following metrics are need to be collected:

Interface used.
Total tx/rx bytes.
Dropped tx/rx packets.

Realtime status from miner to hub (started, stopped)

Create realtime status from miner to hub (started, stoped):

transfer of summary (status of miner, tasks, ports for tasks)
on connect, on change, on request
full report, diffs not required for now

Emit miner's OS limits with the handshake

SONM Logo Not Appearing on README.md

When I try to follow the link to the logo I get a response which reads Error Fetching Resource.
Coming from North-American IP.

No deals matched

In the current MVP phase, is there any statistics available on how long it takes before a deal can be matched on average? I set up all the components, created an ask bid and waited for long, yet no deal was struck.

BTW, is there any telegram channel for dev?

OpenCL support

Actions

detect items needed for support OpenCL by nvidia cards
detect items needed for support OpenCL by ATI cards
detect OpenCL API version
add OpenCL support to nvidia-docker (or create new daemon)
add ATI support to nvidia-docker (or create new daemon)
add task API to point which card/version/interface required for a task

How it works

Thing which must be binded inside a containers depend on vendor, API.

Device itself might be detected via special vendor libraries (e.g. NVLM for NVIDIA or OpenCL library itself)
OpenCL vendor files. Usually located in /etc/OpenCL/vendor/
Libraries for OpenCL, vendor bounded libraries for OpenCL. A nice way to detect libraries on a host system is prodided by nvidia-docker. Each library is looked up via ld.so cache.
Add docker storage plugin to pass libraries inside a container. nvidia's way just create a hadrlink to the libraries inside a volume (plain copy as fallback). Volumes contain version in their names.

packages

dpkg -L ocl-icd-libopencl1
/usr/share/doc-base/ocl-icd-libopencl1

/usr/share/lintian/overrides
/usr/share/lintian/overrides/ocl-icd-libopencl1

/usr/share/doc/ocl-icd-libopencl1
/usr/share/doc/ocl-icd-libopencl1/README
/usr/share/doc/ocl-icd-libopencl1/NEWS.Debian.gz
/usr/share/doc/ocl-icd-libopencl1/html
/usr/share/doc/ocl-icd-libopencl1/html/libOpenCL.html
/usr/share/doc/ocl-icd-libopencl1/copyright
/usr/share/doc/ocl-icd-libopencl1/changelog.Debian.gz
/usr/share/doc/ocl-icd-libopencl1/README.Debian.gz

/usr/share/man/man7/libOpenCL.7.gz

/usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0
/usr/lib/x86_64-linux-gnu/libOpenCL.so.1
/usr/share/man/man7/libOpenCL.so.7.gz

dpkg -L mesa-opencl-icd
/usr/lib/x86_64-linux-gnu/libMesaOpenCL.so.1.0.0

/usr/lib/x86_64-linux-gnu/gallium-pipe/pipe_vmwgfx.so
/usr/lib/x86_64-linux-gnu/gallium-pipe/pipe_r600.so
/usr/lib/x86_64-linux-gnu/gallium-pipe/pipe_r300.so
/usr/lib/x86_64-linux-gnu/gallium-pipe/pipe_radeonsi.so
/usr/lib/x86_64-linux-gnu/gallium-pipe/pipe_swrast.so
/usr/lib/x86_64-linux-gnu/gallium-pipe/pipe_nouveau.so

/usr/share/doc/mesa-opencl-icd
/usr/share/doc/mesa-opencl-icd/copyright
/usr/share/doc/mesa-opencl-icd/changelog.gz

/usr/share/bug/mesa-opencl-icd
/usr/share/bug/mesa-opencl-icd/control
/usr/share/bug/mesa-opencl-icd/script

/etc/OpenCL/vendors/mesa.icd

/usr/lib/x86_64-linux-gnu/libMesaOpenCL.so.1
/usr/lib/x86_64-linux-gnu/libMesaOpenCL.so

dpkg -L nvidia-opencl-icd-304
etc/OpenCL/vendors/nvidia.icd

/usr/lib/i386-linux-gnu/libnvidia-opencl.so.1
/usr/lib/i386-linux-gnu/libnvidia-opencl.so.304.135

/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.304.135

/usr/share/doc/nvidia-opencl-icd-304/changelog.Debian.gz
/usr/share/doc/nvidia-opencl-icd-304/copyright

Useful tool to understand gathering information:
clinfo

Subject to investigate

OpenCL API/ABI compatibility. Does a task need to specify API version? Or just bring your own libraries inside the container and get an access to a GPU device?

Extended task publishing with settings

NAT bypassing

After struggling for a couple of days I've came to the idea how we can allow miners without public IPs to be able to work.

I've made a prototype and it works! Multiple PRs incoming ...

Le plan:

Detect NAT type on the miner.
- If there is no NAT type or it's weak like symmetric UDP firewall - that's fine. Allow them to publish themselves on their white IPs.
- If there is NAT or some other firewall which restricts public access - then we're in trouble.
Provide this information to the Hub.
Hub can organise virtual private network with such miners, allowing requests to be reachable at least from the Hub's endpoint. Either directly or using NAT piercing strategy with further ipip tunnelling.
For miners with public IPs there will be direct access with containers. That information is provided using start and info requests.
For miners that can't be reachable from the WAN we can provide dynamic routing using IPVS configured as NAT. Then provide our own public IP and remapped port for virtual services as an endpoint(s) to the task.

Sudden profits:

Smart and configurable (rr, lwc, wrr, etc.) load balancing between similar containers from the same task. For now we can have only 1 container per task, however it is possible to perform auto-scaling.
1:1 port mapping is also becomes possible, especially for applications that can't be configured.
Both TCP/UDP routing.

Pros:

Fast, as it is L3 forwarding and is done in the kernel.
Fully optional. Some kind as extension for miners that can't obtain public IP for some reasons.

Cons:

Miner should set the Hub as a default gateway (or precisely for a given restricted port range using iptables), otherwise packets will be dropped by the kernel as unexpected, even if the client is reachable directly from miners.
Routed traffic must be additionally accounted (however it's possible to). Not sure it's cons :).
Increased timings (not much), as it's an additional hop between clients and miners.
Hubs are restricted to be run on linux machine, because of netlink and lvs.

Authenticate Hub/Miner via WhiteList smart contract

Hub and miner should be able to:

authenticate each other using ETH-addresses and ECDSA;
get some info (e.g. deposit, status) about each other from the WhiteList smart contract.

Features:

blockchain-related configuration section in config file
Eth JSON-RPC provider to use
Eth network id
Eth credentials (private key / path to json wallet / etc)
allow to run hub/miner in trust all mode, i.e. without blockchain-related configuration
basic challenge-response protocol to authenticate each other
get peer info from WhiteList contract
- there is no use for this info currently, so it is enough to drop it to log

Hub detects different miners if *.uuid file has been removed for some reasons

And if we start several miners on the same machine.

For sure we need a different mechanism for miners deduplication.

Miner sometimes hangs during discovery start up

Here: https://github.com/sonm-io/core/blob/master/insonmnia/miner/server.go#L388.

Hub does not clear detached miners

When a miner dies for whatever reason, Hub still shows it as alive.

Keep all protobuf related stuff in the same package

This allows to avoid the following:

import(
    pb "github.com/sonm-io/core/proto/hub"
    pbminer "github.com/sonm-io/core/proto/miner"
)

Also it simplifies cross-hub-miner-types transformation without much pain as it takes place now.

cc @antmat you'd like to do this.

Multiple wallets for login

The keystore is a folder under which at max 1 wallet will be read.

Wrong Indentation

In the latest release zip file, the last line of hub.yaml has the wrong indentation:

https://github.com/sonm-io/core/blob/master/hub.yaml#L81

When keystore file is switched, refresh state in boltdb

steps to reproduce:

sonmcli login
replace keystore file(remove the original one and added keystore file from web wallet)
without changing any settings, sonmhub, sonmnode, sonmworker

worker:

2017-12-26T02:31:56.119-0500    INFO    miner/server.go:618     connecting to hub       {"address": "127.0.0.1:10002"}
2017-12-26T02:31:56.119-0500    DEBUG   util/certs.go:97        start certificate rotation loop {"every": "1h20m0s"}
2017/12/26 02:31:56 grpc: Server.Serve failed to complete security handshake from "127.0.0.1:10002": authorization failed: expected 5�B90�M/Ĭ, actual ��]V`S�u���
^C2017-12-26T02:31:58.218-0500  INFO    miner/server.go:724     closing miner
2017-12-26T02:31:58.218-0500    INFO    miner/server.go:724     closing miner
2017-12-26T02:31:58.218-0500    INFO    miner/overseer.go:298   event listening has been cancelled
2017-12-26T02:31:58.218-0500    ERROR   miner/main.go:80        Server stop     {"error": "inmemory listener closed"}

hub:

Using /home/g/.sonm/keystore as KeyStore directory
2017-12-26T02:38:47.358-0500    INFO    gateway/gateway_linux.go:47     initializing IPVS context
2017-12-26T02:38:47.358-0500    DEBUG   util/certs.go:97        start certificate rotation loop {"every": "1h20m0s"}
2017-12-26T02:38:47.360-0500    INFO    hub/cluster.go:536      creating store  {"store": {"Store":{"Type":"boltdb","Endpoint":"/tmp/sonm/boltdb","Bucket":"sonm","KeyFile":"","CertFile":""},"Failover":false,"Endpoint":"0.0.0.0:10001","AnnounceEndpoint":"","LeaderKey":"sonm/hub/leader","MemberListKey":"sonm/hub/list","SynchronizableEntitiesPrefix":"sync_prefix","LeaderTTL":20,"AnnouncePeriod":10,"AnnounceTTL":20,"MemberGCPeriod":60}}
2017-12-26T02:38:47.362-0500    INFO    hub/cluster.go:606      fetched endpoints of new member {"endpoints": ["10.0.0.31:10001", "[2601:143:500:2af2::6927]:10001", "[2601:143:500:2af2:80bb:ea49:916e:7347]:10001", "[2601:143:500:2af2:b396:4946:bdab:9008]:10001", "172.17.0.1:10001"]}
2017-12-26T02:38:47.362-0500    INFO    hub/server.go:1384      listening for connections from Miners   {"address": "[::]:10002"}
2017-12-26T02:38:47.362-0500    INFO    hub/server.go:1393      listening for gRPC API connections      {"address": "[::]:10001"}
2017-12-26T02:38:47.362-0500    INFO    hub/server.go:1426      fetched entities        {"tasks": {}, "device_properties": {}, "acl": ["0xbea75d566053cFDCe0C3D0068F14f2DA75fba3A8"], "slots": {}}
2017-12-26T02:38:47.363-0500    DEBUG   hub/server.go:1497      fetching closed deals from the Blockchain
2017-12-26T02:38:47.363-0500    INFO    hub/cluster.go:179      runnning in dev single-server mode
2017-12-26T02:38:47.363-0500    INFO    hub/cluster.go:474      subscribing on sync folder
2017-12-26T02:38:47.363-0500    INFO    hub/server.go:1561      received cluster event  {"event": {"Id":"ced5e61a-ea0f-11e7-aa3c-309c235f2e66"}}
2017-12-26T02:38:47.363-0500    DEBUG   hub/server.go:1450      fetching accepted deals from the Blockchain
2017-12-26T02:38:47.363-0500    INFO    hub/server.go:1761      got cluster members for locator announcement    {"members": [{"Id":"ced5e61a-ea0f-11e7-aa3c-309c235f2e66"}]}
2017-12-26T02:38:47.363-0500    INFO    hub/server.go:1761      got cluster members for locator announcement    {"members": [{"Id":"ced5e61a-ea0f-11e7-aa3c-309c235f2e66"}]}
2017-12-26T02:38:47.363-0500    INFO    hub/server.go:1774      announcing Hub address  {"eth": "0xbea75d566053cFDCe0C3D0068F14f2DA75fba3A8", "addr": ["10.0.0.31:10001", "[2601:143:500:2af2::6927]:10001", "[2601:143:500:2af2:80bb:ea49:916e:7347]:10001", "[2601:143:500:2af2:b396:4946:bdab:9008]:10001", "172.17.0.1:10001"]}
2017-12-26T02:38:47.363-0500    INFO    hub/server.go:1774      announcing Hub address  {"eth": "0xbea75d566053cFDCe0C3D0068F14f2DA75fba3A8", "addr": ["10.0.0.31:10001", "[2601:143:500:2af2::6927]:10001", "[2601:143:500:2af2:80bb:ea49:916e:7347]:10001", "[2601:143:500:2af2:b396:4946:bdab:9008]:10001", "172.17.0.1:10001"]}
2017-12-26T02:38:55.463-0500    INFO    hub/server.go:1632      miner connected {"remote": "127.0.0.1:56156"}
2017-12-26T02:38:55.473-0500    INFO    hub/miner_ctx.go:92     grpc.Dial successfully finished
2017-12-26T02:38:55.473-0500    INFO    hub/miner_ctx.go:127    sending handshake to a Miner    {"addr": "127.0.0.1:56156"}
2017/12/26 02:38:55 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: failed to write initial settings frame: write tcp 127.0.0.1:10002->127.0.0.1:56156: write: broken pipe"; Reconnecting to {miner <nil>}
2017-12-26T02:38:55.474-0500    ERROR   hub/miner_ctx.go:130    failed to receive handshake from a Miner        {"addr": "127.0.0.1:56156", "error": "rpc error: code = Unavailable desc = grpc: the connection is unavailable"}
github.com/sonm-io/core/vendor/go.uber.org/zap.Stack
        /home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/field.go:209
github.com/sonm-io/core/vendor/go.uber.org/zap.(*Logger).check
        /home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/logger.go:273
github.com/sonm-io/core/vendor/go.uber.org/zap.(*Logger).Error
        /home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/logger.go:176
github.com/sonm-io/core/insonmnia/hub.(*MinerCtx).handshake
        /home/ubuntu/go/src/github.com/sonm-io/core/insonmnia/hub/miner_ctx.go:130
github.com/sonm-io/core/insonmnia/hub.(*Hub).createMinerCtx
        /home/ubuntu/go/src/github.com/sonm-io/core/insonmnia/hub/miner_ctx.go:95
github.com/sonm-io/core/insonmnia/hub.(*Hub).handleInterconnect
        /home/ubuntu/go/src/github.com/sonm-io/core/insonmnia/hub/server.go:1634
2017-12-26T02:38:55.474-0500    WARN    hub/server.go:1636      failed to create miner context  {"error": "rpc error: code = Unavailable desc = grpc: the connection is unavailable"}
github.com/sonm-io/core/vendor/go.uber.org/zap.Stack
        /home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/field.go:209
github.com/sonm-io/core/vendor/go.uber.org/zap.(*Logger).check
        /home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/logger.go:273
github.com/sonm-io/core/vendor/go.uber.org/zap.(*Logger).Warn
        /home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/logger.go:168
github.com/sonm-io/core/insonmnia/hub.(*Hub).handleInterconnect
        /home/ubuntu/go/src/github.com/sonm-io/core/insonmnia/hub/server.go:1636
2017/12/26 02:38:55 Failed to dial miner: context canceled; please retry.

Always "green" master

It's not-so-commonly-bad idea to keep master branch in always "green" state. For example it's extremely useful for new users to start working on in just git clone & make pattern.

I see travis, so let it be the tracking issue until it is repaired.

If accepted, it may also be useful to configure repo settings by protecting master branch from merging without CI passing and/or review.

Extend info to provide a miner's pub IP and ports exposed for a task

At a first look without NAT piercing.

Support `nvidia-docker-plugin` for nvidia-docker2

Allow JSON output in CLI

This is very helpful for monitoring robots, as they can easily parse JSON while having problems with the current human-readable output.

Possibly using optional argument --output=[plain(default), json]

cc @sshaman1101.

Add enviroments to develop and integrate blockchain into core

Move smart contracts into core repository:

For blockchain develpment:

make abigen and go-codegeneration script and tasks
wrote wrappers/api for usage in repo

@jorpic Need to research tests implementation in golang and update this article with your decides.

Integration and Unit testing

Miner crashes without "hub.endpoint" option set

Crash log

2017-08-22T18:27:46.036+0300	DEBUG	miner/builder.go:75	building a miner	{"config": {"HubConfig":null,"FirewallConfig":null,"GPUConfig":null,"SSHConfig":null,"LoggingConfig":{"Level":-1}}}
2017-08-22T18:27:46.036+0300	DEBUG	miner/builder.go:83	discovering public IP address ...
2017-08-22T18:27:46.333+0300	INFO	miner/builder.go:109	discovered public IP address	{"addr": "46.148.198.134", "nat": "Not behind a NAT"}
2017-08-22T18:27:46.333+0300	INFO	miner/overseer.go:207	subscribe to Docker events	{"since": "1503415666"}
2017-08-22T18:27:46.347+0300	INFO	miner/builder.go:140	collected Hardware info	{"hardware": {"CPU":[{"cpu":0,"vendorId":"GenuineIntel","family":"6","model":"61","stepping":4,"physicalId":"","coreId":"","cores":2,"modelName":"Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz","mhz":2700,"cacheSize":256,"flags":["fpu","vme","de","pse","tsc","msr","pae","mce","cx8","apic","sep","mtrr","pge","mca","cmov","pat","pse36","clfsh","ds","acpi","mmx","fxsr","sse","sse2","ss","htt","tm","pbe","sse3","pclmulqdq","dtes64","mon","dscpl","vmx","est","tm2","ssse3","fma","cx16","tpr","pdcm","sse4.1","sse4.2","x2apic","movbe","popcnt","aes","pcid","xsave","osxsave","seglim64","tsctmr","avx1.0","rdrand","f16c","smep","erms","rdwrfsgs","tsc_thread_offset","bmi1","avx2","bmi2","invpcid","smap","rdseed","adx","ipt","fpu_csds","syscall","xd","1gbpage","em64t","lahf","lzcnt","prefetchw","rdtscp","tsci"],"microcode":""}],"Memory":{"total":8589934592,"available":2134122496,"used":6455812096,"usedPercent":75.15554428100586,"free":422670336,"active":2102169600,"inactive":1711452160,"wired":2229387264,"buffers":0,"cached":0,"writeback":0,"dirty":0,"writebacktmp":0,"shared":0,"slab":0,"pagetables":0,"swapcached":0},"GPU":[]}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x4651728]

goroutine 1 [running]:
github.com/sonm-io/core/insonmnia/miner.(*config).HubResources(0xc420380ae0, 0x0)
	/Users/alex/gopath/src/github.com/sonm-io/core/insonmnia/miner/config.go:46 +0x8
github.com/sonm-io/core/insonmnia/miner.(*MinerBuilder).Build(0xc4201e1ef8, 0x4d1ef80, 0xc420381020, 0x5761180)
	/Users/alex/gopath/src/github.com/sonm-io/core/insonmnia/miner/builder.go:144 +0x458
main.main()
	/Users/alex/gopath/src/github.com/sonm-io/core/cmd/miner/main.go:39 +0x237```

Append Discovery to hub/miner

append discovery logic to hub
append discovery logic to miner
test it

Metrics are bound to a container id, instead of task

It shows the real world's state more informative, because containers can auto-restart.

Appveyor CI for CLI

Miner/hub daemon config

Create miner/hub daemon configurable ability

common package for common parsers/options, hub/miner have their own interfaces
json (yaml?)
shipped params: all existing params and keys fom CLI (IP, ports, etc)

Feature: Allow user(s) to set device (CPU/GPU) usage cap/upper limit.

Allowing a user to set a resource usage upper limit in percentage allows for example running the miner in the background. This also allows users to control the impact the miner has on system resources and tune it accordingly to the user's wishes.

Unable to start SONM Hub

Problem description:

When running ./sonmhub, encountered IPVS error:

g@g-desktop:~/sonm/hub$ ./sonmhub_linux64 
Using /home/g/.sonm/keystore as KeyStore directory
2017-12-26T01:40:17.624-0500	INFO	gateway/gateway_linux.go:47	initializing IPVS context
2017-12-26T01:40:17.624-0500	DEBUG	util/certs.go:97	start certificate rotation loop	{"every": "1h20m0s"}
2017-12-26T01:40:17.625-0500	INFO	gateway/gateway_linux.go:269	shutting down IPVS context
2017-12-26T01:40:17.625-0500	ERROR	hub/main.go:62	failed to create a new Hub	{"error": "error while calling into IPVS"}
github.com/sonm-io/core/vendor/go.uber.org/zap.Stack
	/home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/field.go:209
github.com/sonm-io/core/vendor/go.uber.org/zap.(*Logger).check
	/home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/logger.go:273
github.com/sonm-io/core/vendor/go.uber.org/zap.(*Logger).Error
	/home/ubuntu/go/src/github.com/sonm-io/core/vendor/go.uber.org/zap/logger.go:176
main.main
	/home/ubuntu/go/src/github.com/sonm-io/core/cmd/hub/main.go:62

Linux g-desktop 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

g@g-desktop:~/sonm/hub$ sudo docker -v
Docker version 17.09.1-ce, build 19e2cf6

UDP port rude capturing

Currently tested on OS X.

Sometimes Hub rejects to start because of:

⇒  ./sonmhub
INFO [08-17|01:51:40] Starting P2P networking
2017-08-17T01:51:40.105+0300	ERROR	hub/main.go:51	Server stop	{"error": "listen udp 192.168.1.227:30343: bind: address already in use"}

LSOF shows nothing about 30343:

⇒  sudo lsof -i udp
Password:
COMMAND     PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
launchd       1           root   24u  IPv4 0x7aa3f720f59e3dc5      0t0  UDP *:netbios-ns
launchd       1           root   26u  IPv4 0x7aa3f720f59e3b65      0t0  UDP *:netbios-dgm
UserEvent    54           root  175u  IPv4 0x7aa3f7210ba2bc05      0t0  UDP *:*
configd      67           root    8u  IPv4 0x7aa3f720f59e3445      0t0  UDP *:*
configd      67           root    9u  IPv4 0x7aa3f720f59e57e5      0t0  UDP *:*
configd      67           root   10u  IPv4 0x7aa3f720f59e23a5      0t0  UDP *:*
airportd     79           root   10u  IPv6 0x7aa3f7210c15cc85      0t0  UDP *:*
airportd     79           root   20u  IPv4 0x7aa3f7210c15ca25      0t0  UDP *:*
airportd     79           root   22u  IPv4 0x7aa3f7210ba2adc5      0t0  UDP *:*
airportd     79           root   23u  IPv4 0x7aa3f720f59e17c5      0t0  UDP *:*
airportd     79           root   24u  IPv4 0x7aa3f720f59e1565      0t0  UDP *:*
airportd     79           root   25u  IPv4 0x7aa3f720f59e4285      0t0  UDP *:*
airportd     79           root   26u  IPv4 0x7aa3f720f59e10a5      0t0  UDP *:*
airportd     79           root   29u  IPv4 0x7aa3f720f59e3905      0t0  UDP *:*
airportd     79           root   32u  IPv4 0x7aa3f720f59e2605      0t0  UDP *:*
locationd   110     _locationd    4u  IPv4 0x7aa3f7210c1607e5      0t0  UDP *:*
blued       114           root    3u  IPv4 0x7aa3f7210c15fc05      0t0  UDP *:*
loginwind   121      esafronov    9u  IPv4 0x7aa3f7210bcf13a5      0t0  UDP *:*
AirPlayXP   127           root    3u  IPv4 0x7aa3f7210c15e1e5      0t0  UDP *:*
AirPlayXP   127           root    5u  IPv4 0x7aa3f7210c15eb65      0t0  UDP *:*
AirPlayXP   127           root    9u  IPv4 0x7aa3f720f59e6fa5      0t0  UDP *:*
AirPlayXP   127           root   10u  IPv4 0x7aa3f720f59e2d25      0t0  UDP *:*
mDNSRespo   173 _mdnsresponder    7u  IPv4 0x7aa3f7210c15dd25      0t0  UDP *:mdns
mDNSRespo   173 _mdnsresponder    8u  IPv6 0x7aa3f7210c15dac5      0t0  UDP *:mdns
mDNSRespo   173 _mdnsresponder   23u  IPv4 0x7aa3f7210bcf2b65      0t0  UDP *:54468
mDNSRespo   173 _mdnsresponder   24u  IPv6 0x7aa3f7210bcf5d45      0t0  UDP *:54468
mDNSRespo   173 _mdnsresponder   36u  IPv4 0x7aa3f7210bcf0565      0t0  UDP *:61748
mDNSRespo   173 _mdnsresponder   47u  IPv6 0x7aa3f7210bcf5885      0t0  UDP *:61748
mDNSRespo   173 _mdnsresponder   48u  IPv4 0x7aa3f7210c15bbe5      0t0  UDP *:57338
mDNSRespo   173 _mdnsresponder   49u  IPv6 0x7aa3f7210c161165      0t0  UDP *:57338
mDNSRespo   173 _mdnsresponder   51u  IPv4 0x7aa3f7210c160a45      0t0  UDP *:64373
mDNSRespo   173 _mdnsresponder   52u  IPv6 0x7aa3f7210c15e445      0t0  UDP *:64373
mDNSRespo   173 _mdnsresponder   54u  IPv4 0x7aa3f7210c161fa5      0t0  UDP *:59601
mDNSRespo   173 _mdnsresponder   56u  IPv6 0x7aa3f7210c161885      0t0  UDP *:59601
ntpd        248           root   20u  IPv6 0x7aa3f720f59e5ca5      0t0  UDP *:ntp
ntpd        248           root   21u  IPv4 0x7aa3f720f59e6165      0t0  UDP *:ntp
ntpd        248           root   22u  IPv4 0x7aa3f720f59e63c5      0t0  UDP localhost:ntp
ntpd        248           root   23u  IPv6 0x7aa3f720f59e1a25      0t0  UDP localhost:ntp
ntpd        248           root   24u  IPv6 0x7aa3f720f59e6d45      0t0  UDP [fe80:1::1]:ntp


ntpd        248           root   25u  IPv6 0x7aa3f7210ba2b745      0t0  UDP [fe80:b::cacc:d501:2fea:6bf7]:ntp
ntpd        248           root   26u  IPv6 0x7aa3f7210ba2b025      0t0  UDP esafronov-osx3.local:ntp
ntpd        248           root   27u  IPv6 0x7aa3f7210bcf0a25      0t0  UDP esafronov-osx3.local:ntp
ntpd        248           root   29u  IPv4 0x7aa3f7211c95c3c5      0t0  UDP 192.168.1.227:ntp

ntpd        248           root   30u  IPv6 0x7aa3f7210ba2c0c5      0t0  UDP [fe80:c::40e0:113c:c515:e007]:ntp
ntpd        248           root   31u  IPv4 0x7aa3f7211c959dc5      0t0  UDP 37.9.111.23-vpn.dhcp.yndx.net:ntp
ntpd        248           root   32u  IPv6 0x7aa3f7211c95a285      0t0  UDP [fe80:f::f65c:89ff:fe94:7187]:ntp
ntpd        248           root   33u  IPv6 0x7aa3f7210c15c305      0t0  UDP [fe80:a::5339:4709:ef4d:86a1]:ntp
ntpd        248           root   34u  IPv6 0x7aa3f7211c95b325      0t0  UDP [2a02:6b8:b010:d006::1:16]:ntp
ntpd        248           root   35u  IPv6 0x7aa3f7210bcf1605      0t0  UDP [fe80:10::4f9c:421e:9f3d:48b9]:ntp
symptomsd   259      _networkd   23u  IPv4 0x7aa3f7210c15c0a5      0t0  UDP *:*
UserEvent   425      esafronov    4u  IPv4 0x7aa3f7210bcf0ee5      0t0  UDP *:*
WirelessR   439           root    3u  IPv4 0x7aa3f7210ba2a1e5      0t0  UDP *:*
WirelessR   439           root    4u  IPv4 0x7aa3f7210ba29d25      0t0  UDP *:*
identitys   440      esafronov   20u  IPv4 0x7aa3f7210ba2cf05      0t0  UDP *:*
sharingd    443      esafronov    4u  IPv4 0x7aa3f7210bcef725      0t0  UDP *:*
sharingd    443      esafronov    8u  IPv4 0x7aa3f720f59e0e45      0t0  UDP *:*
sharingd    443      esafronov    9u  IPv4 0x7aa3f720f59e44e5      0t0  UDP *:*
sharingd    443      esafronov   10u  IPv4 0x7aa3f720f59e4745      0t0  UDP *:*
sharingd    443      esafronov   12u  IPv4 0x7aa3f7210c15b985      0t0  UDP *:*
SystemUIS  1036      esafronov   11u  IPv4 0x7aa3f7210c15d3a5      0t0  UDP *:*
SystemUIS  1036      esafronov   12u  IPv4 0x7aa3f7210c15d145      0t0  UDP *:*
SystemUIS  1036      esafronov   15u  IPv4 0x7aa3f7210c160325      0t0  UDP *:52551
SystemUIS  1036      esafronov   16u  IPv4 0x7aa3f7210c15cee5      0t0  UDP *:*
SystemUIS  1036      esafronov   19u  IPv4 0x7aa3f7210c15f4e5      0t0  UDP *:*
WiFiProxy  1084      esafronov    5u  IPv4 0x7aa3f7210c15f9a5      0t0  UDP *:*
WiFiAgent  1270      esafronov    5u  IPv4 0x7aa3f7210bcf34e5      0t0  UDP *:*
wifiveloc  1302           root    3u  IPv4 0x7aa3f7210bcf5165      0t0  UDP *:*
wifiveloc  1302           root    4u  IPv4 0x7aa3f7210bcf26a5      0t0  UDP *:*
Yandex    73569      esafronov   53u  IPv4 0x7aa3f7210ba2d3c5      0t0  UDP *:*
netbiosd  86599       _netbios    3u  IPv4 0x7aa3f720f59e3dc5      0t0  UDP *:netbios-ns
netbiosd  86599       _netbios    4u  IPv4 0x7aa3f720f59e3b65      0t0  UDP *:netbios-dgm
vpnkit    90186      esafronov   26u  IPv4 0x7aa3f7210ba2c585      0t0  UDP *:61079
openvpn   91876           root    5u  IPv4 0x7aa3f720f59e50c5      0t0  UDP *:54947

Seems like it's only reproducible if the firewall is enabled (which is) and geth was previously running with the same port. The firewall seems to hold on to the port for some reason.

Macohipster problems, nuff said.