open-horizon / anax Goto Github PK

See step 5 of https://staging.bluehorizon.network/documentation/adding-your-device for the instructions that the user sees.

Though anax api /horizondevice supports token update, the update does not get carried to the other places of the code that talk to the exchange.

{
"name": "Policy for location merged with location amd64",
"archived": true,
"current_agreement_id": "0f5406af533819c7f8f2c825e4dee804fe85e824f8d2b56618436aaa5cb8f00a",
"consumer_id": "stg-agbot-lon02-01.bluehorizon.hovitos.engineering",
"agreement_creation_time": 1487754299,
"agreement_accepted_time": 1487754311,
"agreement_finalized_time": 1487754406,
"agreement_data_received_time": 1487754393,
"agreement_terminated_time": 1487754481,
"agreement_execution_start_time": 1487754331,
"terminated_reason": 205,
"terminated_description": "agreement bot discovered cancellation from producer"
}

Here is timeline for this event:
agbot:
09:04:51 send proposal
09:05:04 got reply, send reply ack
09:05:50 recored agreement on bc
09:06:12 send dataverification to exchange
09:06:34 got bc event (agreement created)
09:06:38 got dataverification ack
09:06:44 cannot find agreement on the bc (governance 150 "discovered terminated agreement")
09:06:44 terminate agreement
09:08:05 got bc event (agreement canceled)

device
09:04:58 received proposal
09:04:59 send reply
09:05:11 received replyack
09:06:23 received dataverification
09:06:45 got bc event (agreement created)
09:08:01 got bc event (agreement canceled)
09:08:01 terminate agreement

scmon
agreement terminated by consumer

Currently anax uses default http.Client to make request to the exchange APIs. The default timeout for the http.Client is 0 which means not default. This means if it take 1 hour for the server to return the result, it will wait for an hour. We can add timeout to the http request and add retry logic if it is returns error.

On the device setup page and the service selection pages, there are many links that links to other websites. Clicking the links and then click the back button on the browser, it brings to a page which says "Your device is not configured to execute any services." and there is a button "Begin Setup". It does not bring back to the previous page.

The launchcontext has both a structured configure message member and an unstructured one. The unstructured one is used to write the raw configure description inside the workload so it can read facts about how it was set up. It also is used to build iptables network isolation rules.

It looks like we're not writing the raw config to the launchcontext so these container features won't work (the written config file is 0-length and no one is using the network isolation feature so we're not encountering errors there).

We either need to write the configuration facts to the launch context as a byte array or rework the container code to use the structure configuration data.

What can be done now with the current tools is documented in https://repo.hovitos.engineering/MTN/mtn/wikis/horizon_v2_on_rpi#update-and-re-registration-scenarios . But when implemented in re-reg for real (and someone does the equivalent in the reg ui), it should get simpler from the user’s perspective, because reg_all/anax will programmatically compare the new registration params with the previous and figure out what agreements need to be added/modified/redone/deleted

Currently in the device agreement cancellation process, it terminates the agreement on exchange, blockchain and then archive the agreement before sending the message to the container to cleanup the workload. The order should be changed so that the archiving happens after the workload cleanup.

New registration GUI does not work on staging. It only talks to the production exchange api.

1. When using an existing username, they shouldn't be asked to fill in the email field
2. Reset Password should take them to a page in which the reset token is emailed to the user and this page waits for them to enter the token and new pw. Or if they click on the link in the email, it should take them to this page with the token already filled in.
3. There should also be a simpler option to change their pw when they know the current pw.

Happens when 1 agreement is in the process of being terminated. Is this the proper design? Should there be another state called pending-termination

Machine-based reputation system: https://drive.google.com/open?id=1uTlwDpMrjqhwVcAoQsWQWx3puJ84uSNySnH3zkMEhZ8

• install snap and register services on a test nodes (carl)
• monitor the progress of the test nodes
• diagnose if something is wrong with the progress
• setup a management node and a few test nodes (carl)
• collect agreement performance data of the test nodes
• enable retest (doing os reload) (carl)
• simple, documented cmd that anyone can use to start/stop the test

Registration dashboard shows the all the agreement names as "Compute Resources". No way to tell which one is sdr, which is netspeetc. etc.

Currently the egbot just logs the error message when the blockchain write fails. In the mean time the edge device keeps setting up the workload. The agreement will get canceled by the device after AgreementTimeoutS reached.
Can agbot retry blockchain writting or cancel the agreement right there?

Once we do the MS/workload split properly, then anax will be in control of the MSs (know about them, start them, etc. Currently anax is at the mercy of the workloads to start the firmware containers.) So once anax knows about the gps MS, and knows how to start it, anax can (depending on the user registration choices) start the gps MS and call its api to get the obfuscated location, and include that with the other device registration info it puts in the exchange.

curl http://localhost/status did not return anything. The log file has the following errors.
Jan 11 21:04:05 vs2 snap[31395]: I0111 21:04:05.794682 31444 status.go:82] returned: map[jsonrpc:2.0 id:1 result:false]
Jan 11 21:04:05 vs2 snap[31395]: I0111 21:04:05.795695 31444 status.go:82] returned: map[jsonrpc:2.0 id:1 result:0x19b0a1]
Jan 11 21:04:05 vs2 snap[31395]: I0111 21:04:05.797107 31444 status.go:82] returned: map[jsonrpc:2.0 id:1 result:0x4]
Jan 11 21:04:05 vs2 snap[31395]: I0111 21:04:05.798215 31444 status.go:82] returned: map[jsonrpc:2.0 id:1 error:map[message:invalid or missing value for params[0] code:-32602]]
Jan 11 21:04:05 vs2 snap[31395]: 2017/01/11 21:04:05 http: panic serving 127.0.0.1:56322: interface conversion: interface is nil, not string
Jan 11 21:04:05 vs2 snap[31395]: goroutine 661 [running]:
Jan 11 21:04:05 vs2 snap[31395]: net/http.(*conn).serve.func1(0xc420214480)
Jan 11 21:04:05 vs2 snap[31395]: #11/usr/local/go/src/net/http/server.go:1491 +0x12a
Jan 11 21:04:05 vs2 snap[31395]: panic(0x8535a0, 0xc42036f500)
Jan 11 21:04:05 vs2 snap[31395]: #11/usr/local/go/src/runtime/panic.go:458 +0x243
Jan 11 21:04:05 vs2 snap[31395]: github.com/open-horizon/anax/api.WriteGethStatus(0xc4202138e0, 0x15, 0xc42036ee40, 0x0, 0xc4203a3710)
Jan 11 21:04:05 vs2 snap[31395]: #11/tmp/anax-gopath/src/github.com/open-horizon/anax/api/status.go:114 +0x33d
Jan 11 21:04:05 vs2 snap[31395]: github.com/open-horizon/anax/api.(*API).status(0xc420213c20, 0xad8600, 0xc420360000, 0xc4201e6ff0)
Jan 11 21:04:05 vs2 snap[31395]: #11/tmp/anax-gopath/src/github.com/open-horizon/anax/api/api.go:774 +0x310
Jan 11 21:04:05 vs2 snap[31395]: github.com/open-horizon/anax/api.(*API).(github.com/open-horizon/anax/api.status)-fm(0xad8600, 0xc420360000, 0xc4201e6ff0)
Jan 11 21:04:05 vs2 snap[31395]: #11/tmp/anax-gopath/src/github.com/open-horizon/anax/api/api.go:84 +0x48
Jan 11 21:04:05 vs2 snap[31395]: net/http.HandlerFunc.ServeHTTP(0xc420297220, 0xad8600, 0xc420360000, 0xc4201e6ff0)
Jan 11 21:04:05 vs2 snap[31395]: #11/usr/local/go/src/net/http/server.go:1726 +0x44
Jan 11 21:04:05 vs2 snap[31395]: github.com/open-horizon/anax/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc4201b2b40, 0xad8600, 0xc420360000, 0xc4201e6ff0)
Jan 11 21:04:05 vs2 snap[31395]: #11/tmp/anax-gopath/src/github.com/open-horizon/anax/vendor/github.com/gorilla/mux/mux.go:103 +0x255
Jan 11 21:04:05 vs2 snap[31395]: github.com/open-horizon/anax/api.(*API).listen.func1.1(0xad8600, 0xc420360000, 0xc4201e6ff0)
Jan 11 21:04:05 vs2 snap[31395]: #11/tmp/anax-gopath/src/github.com/open-horizon/anax/api/api.go:69 +0x1e1
Jan 11 21:04:05 vs2 snap[31395]: net/http.HandlerFunc.ServeHTTP(0xc420300e60, 0xad8600, 0xc420360000, 0xc4201e6ff0)
Jan 11 21:04:05 vs2 snap[31395]: #11/usr/local/go/src/net/http/server.go:1726 +0x44
Jan 11 21:04:05 vs2 snap[31395]: net/http.serverHandler.ServeHTTP(0xc420215400, 0xad8600, 0xc420360000, 0xc4201e6ff0)
Jan 11 21:04:05 vs2 snap[31395]: #11/usr/local/go/src/net/http/server.go:2202 +0x7d
Jan 11 21:04:05 vs2 snap[31395]: net/http.(*conn).serve(0xc420214480, 0xad8e40, 0xc42036eb80)
Jan 11 21:04:05 vs2 snap[31395]: #11/usr/local/go/src/net/http/server.go:1579 +0x4b7
Jan 11 21:04:05 vs2 snap[31395]: created by net/http.(*Server).Serve
Jan 11 21:04:05 vs2 snap[31395]: #11/usr/local/go/src/net/http/server.go:2293 +0x44d
Jan 11 21:04:06 vs2 kernel: [5430253.005455] audit_printk_skb: 66 callbacks suppressed

In the repetitive cancellation tests, there are a lot of agreement got canceled because of "workload terminated". Checking the syslog, found that the workload could not start because there are no enough ip in the ip pool for it.
The related errors are:
Jan 20 13:20:36 vs3 snap[21575]: E0120 13:20:36.818865 21622 container.go:943] Error starting containers: API error (500): {"message":"could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network"}

The network could not be removed for previous workload containers because of this:
Jan 20 12:56:35 vs3 snap[21575]: I0120 12:56:35.406252 21622 container.go:1092] Freeing agreement net: d0ca332359a73f5c0637121cca28169a8173f17403dbc19726b47c6db723b484
Jan 20 12:56:35 vs3 snap[21575]: E0120 12:56:35.411960 21622 container.go:1101] Failure removing network: 6ca3d3c42a6cf25edb6896e0f805808d36a3e5160895d58363371b34a5f34954. Error: API error (500): {"message":"network d0ca332359a73f5c0637121cca28169a8173f17403dbc19726b47c6db723b484 has active endpoints"}
Jan 20 12:56:35 vs3 snap[21575]: I0120 12:56:35.442092 21622 container.go:1120] Removing iptables isolation rules for agreement d0ca332359a73f5c0637121cca28169a8173f17403dbc19726b47c6db723b484

Jan 20 12:56:02 vs3 snap[21575]: E0120 12:56:02.384070 21622 container.go:1070] Service pvp in agreement d0ca332359a73f5c0637121cca28169a8173f17403dbc19726b47c6db723b484 could not be removed. Error: API error (500): {"message":"Unable to remove filesystem for 11ab08e88c742e6de498b28c6e3fbd0486fe72c3b111200bf1dd16b886c6f11b: remove /var/lib/docker/containers/11ab08e88c742e6de498b28c6e3fbd0486fe72c3b111200bf1dd16b886c6f11b/shm: device or resource busy"}

I did a cancellation tests on an x86 machine over the weekend (66 hours) where the script canceled the agreements every 10 minutes. There were 2 services registered at any time (location and netspeed). However, when I came back and checked the archived agreements, I saw there were 313 agreements got canceled due to "agreement never appeared on the blockchain", which is every 12 minutes.

The dashboard gui shows the agreement status from "registered" to "in agreement" as soon as the agreement is created. It does not show other status like: negotiating, downloading, running workloads.

anax restarted due to geth out of memory. Then one agreement got into the following state and no new agreement can be formed for netspeed.

  "name": "Policy for netspeed merged with netspeed arm",
  "archived": false,
  "current_agreement_id": "ce788fff87553d1ed101ebd1be709e2b1c0345404a5c2bac480b53879457b490",
  "consumer_id": "stg-agbot-lon02-01.bluehorizon.hovitos.engineering",
  "agreement_creation_time": 1488789542,
  "agreement_accepted_time": 1488789564,
  "agreement_finalized_time": 1488789715,
  "agreement_data_received_time": 0,
  "agreement_execution_start_time": 1488789586,
  "agreement_terminated_time": 1488790593,
  "terminated_reason": 203,
  "terminated_descriptioni": null,
  "agreement_protocol_terminated_time": null,
  "workload_terminated_time": null
}```
The daemon.log shows this line: 
```Mar  6 08:57:14 horizon-00000000217597c7 snap[2547]: I0306 08:57:14.927297    2644 agreement.go:369] AgreementWorker agreement ce788fff87553d1ed101ebd1be709e2b1c0345404a5c2bac480b53879457b490 missing from exchange, adding it back in.```

"Blue Horizon map" link for the "Personal Weather Station" service is wrong. there is a typo there.
http://bluehorizeon.network/map
And another issue: should it point to the staging map if the rpi is setup for staging?

This is a feature request. Please provide a way for the gui to show all the agreement attributes and their values plus all the status details.

The archived agreements returned by /agreeement API are not sorted by the timestamp which makes it very hard to understand what's going on if there are a lot of failure.

This is a feature request. Please provide a button on the dashboard so that it will show all the terminated agreements and their reasons when clicked.

It is currently at 3 minutes (but configurable), but this delay can contribute to the device timing out waiting for the BC write (depending on what this is configured to), or just make the user concerned.

Add some more code to go-solidity lib to check on pending transactions, and maybe we can shorten the retry interval time (but that will be tricky code)

I have the most recent kernel from Ubuntu Core for the TX1 and all of the hardware, will start work on the image today.

Change prefix for workload environmental variables from "MTN_" to "HORIZON_". This needs to be done after all the workloads get changed.

Needs to be done by end of april

This is a feature request: Anax support DELETE and PATCH for /service and service/attribute. This will enable user to remove a service and modify a service agreement.

In the repetitive cancellation test (cancel agreements every 5 minutes), all the agreements got the into the state like this:
{
"name": "Policy for netspeed merged with netspeed amd64",
"current_agreement_id": "11be06f12475f01f10edc871d623b2021da7a613d2730df7433c04f2d7200f48",
"consumer_id": "agbot-lon02-01.bluehorizon.hovitos.engineering",
"agreement_creation_time": 1486653685,
"agreement_accepted_time": 1486653749,
"agreement_finalized_time": 0,
"agreement_data_received_time": 1486653837,
"agreement_execution_start_time": 1486653772,
"agreement_terminated_time": 0,
"terminated_reason": 0,
"terminated_description": ""
}
where agreement_finalized_time is not set. Further debugging found that the agreement has been written on the blockchain but anax stops receiving any blockchain events at certain point. The blochchain number that geth got is updated on the node. geth seemed working because whisper was working.

Thought Anax /service/attributes API mentions that if the service_url is null for the input attribute the attributes will apply to all, some attributes do not go by this rule. For example:
If POST /service/attribute is called with the following data,
{
"id": "compute",
"short_type": "compute",
"label": "Compute Resources",
"publishable": true,
"mappings": {
"ram": 128,
"cpus": 1
}
}
Then call POST /service with:
{
"sensor_url": "https://bluehorizon.network/documentation/cputemp-device-api",
"sensor_name": "cputemp",
"attributes": [
{
"id": "app",
"short_type": "mapped",
"label": "app",
"publishable": true,
"mappings": {
"MTN_CPU_TEMP": "true"
}
}
]
}

The GET /service will show
{
"sensor_url": "https://bluehorizon.network/documentation/cputemp-device-api",
"sensor_name": "cputemp",
"attributes": [
{
"id": "app",
"short_type": "mapped",
"label": "app",
"publishable": true,
"mappings": {
"MTN_CPU_TEMP": "true"
}
},
{
"id": "compute",
"short_type": "compute",
"label": "Compute Resources",
"publishable": true,
"mappings": {
"ram": 0,
"cpus": 1
}
}
]
}

Sometimes the anax wrongly reported that the agreement termination reason was "workload terminated", but in fact the agreement was terminated by user.

In the daemon.log, I saw geth was running out of memory and cause itself to restart. And sometimes it caused other processes such as sanpd, python, anax etc. to restart.

Mar  5 15:53:34 horizon-00000000217597c7 kernel: [649556.201395] Out of memory: Kill process 26255 (apt-check) score 45 or sacrifice child
Mar  5 15:53:43 horizon-00000000217597c7 kernel: [649565.931482] Out of memory: Kill process 26424 (geth) score 62 or sacrifice child
Mar  6 00:00:58 horizon-00000000217597c7 kernel: [678800.559754] Out of memory: Kill process 26465 (geth) score 137 or sacrifice child
Mar  6 01:46:21 horizon-00000000217597c7 kernel: [685123.677986] Out of memory: Kill process 13387 (geth) score 142 or sacrifice child
Mar  6 01:48:21 horizon-00000000217597c7 kernel: [685243.267560] Out of memory: Kill process 6388 (geth) score 49 or sacrifice child
Mar  6 01:49:25 horizon-00000000217597c7 kernel: [685307.211020] Out of memory: Kill process 6938 (geth) score 86 or sacrifice child
Mar  6 01:49:25 horizon-00000000217597c7 kernel: [685307.222264] Out of memory: Kill process 1767 (python) score 15 or sacrifice child
Mar  6 01:49:25 horizon-00000000217597c7 kernel: [685307.235810] Out of memory: Kill process 5608 (anax) score 10 or sacrifice child
Mar  6 01:50:11 horizon-00000000217597c7 kernel: [685353.815619] Out of memory: Kill process 7409 (geth) score 77 or sacrifice child
Mar  6 01:50:11 horizon-00000000217597c7 kernel: [685353.832414] Out of memory: Kill process 7338 (python) score 16 or sacrifice child
Mar  6 01:50:11 horizon-00000000217597c7 kernel: [685353.845622] Out of memory: Kill process 17519 (snapd) score 10 or sacrifice child
Mar  6 01:52:03 horizon-00000000217597c7 kernel: [685465.427976] Out of memory: Kill process 7867 (geth) score 93 or sacrifice child
Mar  6 01:56:33 horizon-00000000217597c7 kernel: [685735.102476] Out of memory: Kill process 8549 (geth) score 106 or sacrifice child
Mar  6 01:56:33 horizon-00000000217597c7 kernel: [685735.112650] Out of memory: Kill process 7760 (python) score 15 or sacrifice child
Mar  6 01:57:30 horizon-00000000217597c7 kernel: [685792.970986] Out of memory: Kill process 9687 (geth) score 85 or sacrifice child
Mar  6 01:58:09 horizon-00000000217597c7 kernel: [685831.800181] Out of memory: Kill process 10101 (geth) score 45 or sacrifice child
Mar  6 01:58:09 horizon-00000000217597c7 kernel: [685831.809562] Out of memory: Kill process 9619 (python) score 17 or sacrifice child
Mar  6 01:58:46 horizon-00000000217597c7 kernel: [685868.979759] Out of memory: Kill process 10471 (geth) score 42 or sacrifice child
Mar  6 01:58:47 horizon-00000000217597c7 kernel: [685868.992343] Out of memory: Kill process 10366 (python) score 17 or sacrifice child
Mar  6 01:59:19 horizon-00000000217597c7 kernel: [685901.905018] Out of memory: Kill process 10908 (geth) score 59 or sacrifice child
Mar  6 01:59:51 horizon-00000000217597c7 kernel: [685933.832290] Out of memory: Kill process 11214 (geth) score 57 or sacrifice child
Mar  6 01:59:51 horizon-00000000217597c7 kernel: [685933.841674] Out of memory: Kill process 10809 (python) score 18 or sacrifice child
Mar  6 02:00:24 horizon-00000000217597c7 kernel: [685966.589475] Out of memory: Kill process 11549 (geth) score 56 or sacrifice child
Mar  6 02:01:03 horizon-00000000217597c7 kernel: [686005.260840] Out of memory: Kill process 11881 (geth) score 41 or sacrifice child
Mar  6 02:01:03 horizon-00000000217597c7 kernel: [686005.269099] Out of memory: Kill process 11944 (geth) score 43 or sacrifice child
Mar  6 02:01:41 horizon-00000000217597c7 kernel: [686043.796878] Out of memory: Kill process 12133 (geth) score 45 or sacrifice child
Mar  6 02:01:41 horizon-00000000217597c7 kernel: [686043.812232] Out of memory: Kill process 11460 (python) score 17 or sacrifice child
Mar  6 02:02:15 horizon-00000000217597c7 kernel: [686077.752381] Out of memory: Kill process 12474 (geth) score 60 or sacrifice child
Mar  6 02:02:54 horizon-00000000217597c7 kernel: [686116.937272] Out of memory: Kill process 12774 (geth) score 42 or sacrifice child
Mar  6 02:02:54 horizon-00000000217597c7 kernel: [686116.952997] Out of memory: Kill process 12407 (python) score 17 or sacrifice child
Mar  6 02:02:54 horizon-00000000217597c7 kernel: [686116.969856] Out of memory: Kill process 7229 (anax) score 12 or sacrifice child
Mar  6 02:03:35 horizon-00000000217597c7 kernel: [686157.907090] Out of memory: Kill process 13171 (geth) score 49 or sacrifice child
Mar  6 02:03:35 horizon-00000000217597c7 kernel: [686157.919953] Out of memory: Kill process 13051 (python) score 18 or sacrifice child
Mar  6 02:04:19 horizon-00000000217597c7 kernel: [686201.295913] Out of memory: Kill process 13588 (geth) score 43 or sacrifice child
Mar  6 02:05:08 horizon-00000000217597c7 kernel: [686250.553078] Out of memory: Kill process 13928 (geth) score 44 or sacrifice child
Mar  6 02:05:08 horizon-00000000217597c7 kernel: [686250.561549] Out of memory: Kill process 13966 (geth) score 46 or sacrifice child
Mar  6 02:05:20 horizon-00000000217597c7 kernel: [686262.478269] Out of memory: Kill process 14251 (geth) score 46 or sacrifice child
Mar  6 02:05:20 horizon-00000000217597c7 kernel: [686262.490812] Out of memory: Kill process 13488 (python) score 17 or sacrifice child
Mar  6 02:06:02 horizon-00000000217597c7 kernel: [686304.092069] Out of memory: Kill process 14361 (geth) score 48 or sacrifice child
Mar  6 02:06:02 horizon-00000000217597c7 kernel: [686304.101823] Out of memory: Kill process 14355 (python) score 15 or sacrifice child
Mar  6 02:06:02 horizon-00000000217597c7 kernel: [686304.110071] Out of memory: Kill process 12991 (anax) score 12 or sacrifice child
Mar  6 02:06:44 horizon-00000000217597c7 kernel: [686346.701968] Out of memory: Kill process 14840 (geth) score 58 or sacrifice child
Mar  6 02:06:44 horizon-00000000217597c7 kernel: [686346.708838] Out of memory: Kill process 14892 (geth) score 60 or sacrifice child
Mar  6 02:07:39 horizon-00000000217597c7 kernel: [686401.040659] Out of memory: Kill process 15156 (geth) score 47 or sacrifice child
Mar  6 02:07:39 horizon-00000000217597c7 kernel: [686401.050280] Out of memory: Kill process 15192 (geth) score 49 or sacrifice child
Mar  6 02:08:41 horizon-00000000217597c7 kernel: [686463.601240] Out of memory: Kill process 15476 (geth) score 44 or sacrifice child
Mar  6 02:08:41 horizon-00000000217597c7 kernel: [686463.609119] Out of memory: Kill process 15538 (geth) score 46 or sacrifice child
Mar  6 02:08:41 horizon-00000000217597c7 kernel: [686463.616638] Out of memory: Kill process 14769 (python) score 17 or sacrifice child
Mar  6 02:09:42 horizon-00000000217597c7 kernel: [686524.700182] Out of memory: Kill process 15913 (geth) score 46 or sacrifice child
Mar  6 02:09:42 horizon-00000000217597c7 kernel: [686524.713396] Out of memory: Kill process 15840 (python) score 15 or sacrifice child
Mar  6 02:10:42 horizon-00000000217597c7 kernel: [686584.270918] Out of memory: Kill process 16432 (geth) score 48 or sacrifice child
Mar  6 02:10:42 horizon-00000000217597c7 kernel: [686584.285043] Out of memory: Kill process 16358 (python) score 17 or sacrifice child

Design: http://drive.google.com/open?id=1jr8X0x6k_ahRvt4gOWQss73ba26-npkl3aGlQj87T7A