nerves-hub / nerves_hub Goto Github PK
View Code? Open in Web Editor NEWNervesHub client for devices
NervesHub client for devices
Problem
The current docs and README shows shows examples of pre-mix target
implementation.
Acceptance Criteria
mix target
usageRequested by: @fhunleth
nerves_team_device $ 1 mix nerves_hub.deployment update "Test 1" is_active true
NervesHub server: api.nerves-hub.org:443
NervesHub organization: aselder
Local NervesHub user password:
Failed to update deployment.
Reason: {:error, %{"errors" => %{"detail" => "Not Found"}}}
See this link: https://github.com/arduino/arduino-cvd-policy
This is merely a proposal right now. It will be important to review the details to make sure that we can commit to them.
This happens for you if you leverage NervesHub.Supervisor, but in the case of not wanting a WebSocket connection (so no supervisor) it does not, and isn't documented. Simply adding NervesHub.Connection to your own supervisor is sufficient.
At the risk of introducing a painful project name change, I'm having lots of trouble verbally communicating with people about this library. "NervesHub", the server, and nerves_hub
, this package, sound the same. To limit confusion, I've been calling this "NervesHub device" or "nerves underscore hub".
I want to start this issue so that the decision process on the name, if any, is logged and I can point people to it when they ask. I'm not in any rush to commit to a new name and I'm not set on nerves_hub_device
if there are other better ones. If we're going to switch, I'd like it to be before 1.0.
Just an idea, but depending on the device's connection it might be interesting to not only have a iex shell to the device, but also a tunnel to send http requests through. E.g. a device I currently use has a small phoenix app for "device admin" tasks, which I gathered would be quite nice to be able to access through nerves_hub as well.
A few times i've had a device open a stream connection then the connection gets dropped. The stream never exits requiring a reboot to try again.
It's currently untested.
For devices that require firmware to be marked valid by application software, it would be useful to have a callback or way to trigger that to happen on a successful connection to NervesHub. The logic being that if a device can connect to NervesHub, it's working well enough to receive a firmware update.
In nerves_hub
, the NERVES_HUB_HOME
directory is being set to nerves-hub
while in nerves_hub_cli
it is being set to .nerves-hub
in dev and test configs.
Its not really a huge deal, just odd to me that behavior would change so minimally yet enough to be bothersome to someone switching between working in nerves_hub
and nerves_hub_cli
.
Before I go changing things, is there a specific reason its that way and I just don't know about it?
The ability to run the client behind an HTTP proxy would be nice, because Nerves devices are likely to be deployed inside of corporate networks where use of a proxy is required to get to the web.
It looks like :hackney.request/5 allows specification of a proxy to use. I also saw :hackney_http_proxy.connect_proxy/5
I haven't been able to find a way to do this with websocket_client yet, but perhaps I just haven't dug far enough into it.
See #16 for failure log.
A missing firmware UUID is a pretty bad situation, but we shouldn't crash. I think the right answer is to connect to NervesHub anyway and provide an empty UUID. Given how rare this situation should be in deployed devices, it seems fine for NervesHub to just log that the device is running unknown firmware. We may eventually want to force a firmware download to the device for a last-ditch attempt to remotely recover it. However, no remote recovery is possible if the device doesn't connect.
Right now we're spewing Logger messages multiple times all over the place which leads to basically the same log messages over and over cluttering the logs. And each message is basically the same, but just different enough that it can cause some confusion troubleshooting what might be going on.
We also send pretty much every error to a Client
and the Client.Default
logs with a "Firmware stream error: #{error}"
but not every error is fully related to the firmware stream.
It would be nice to clean this up a little bit and make greping through logs a little easier.
Running firmware on a RPI0 with nerves_hub (mix nerves_hub.device burn IDENTIFIER), resulted in an early exit of the applicaion
[
url: “wss://device.nerves-hub.org/socket/websocket”,
serializer: Jason,
ssl_verify: :verify_peer,
socket_opts: [
cert: “”,
key: {:ECPrivateKey, “”},
cacerts: [
<<48, 130, ...>>,
<<48, 130, ...>>,
<<48, 130, ...>>
],
server_name_indication: ‘device.nerves-hub.org’
]
10:47:43.365 [info] Application nerves_hub exited: exited in: NervesHub.Application.start(:normal, [])
** (EXIT) an exception was raised:
** (ArgumentError) argument error
:erlang.byte_size(nil)
(nerves_hub) lib/nerves_hub/firmware_channel.ex:8: NervesHub.FirmwareChannel.topic/0
(nerves_hub) lib/nerves_hub/application.ex:17: NervesHub.Application.start/2
(kernel) application_master.erl:273: :application_master.start_it_old/4
10:47:43.366 [info] Application phoenix_channel_client exited: :stopped
10:47:43.366 [info] Application jason exited: :stopped
10:47:43.366 [info] Application websocket_client exited: :stopped
10:47:43.366 [info] Application nerves_time exited: :stopped
10:47:43.367 [info] Application muontrap exited: :stopped
10:47:43.367 [info] Application nerves_leds exited: :stopped
10:47:43.367 [info] Application nerves_init_gadget exited: :stopped
10:47:43.367 [info] Application one_dhcpd exited: :stopped
10:47:43.440 [info] Application mdns exited: :stopped
10:47:43.458 [info] Application dns exited: :stopped
10:47:43.467 [info] Application ring_logger exited: :stopped
10:47:43.513 [info] Application nerves_firmware_ssh exited: :stopped
10:47:43.650 [info] Application ssh exited: :stopped
10:47:43.770 [info] Application nerves_network exited: :stopped
10:47:43.838 [info] Application nerves_network_interface exited: :stopped
10:47:43.950 [info] Application nerves_wpa_supplicant exited: :stopped
10:47:43.954 [info] Application elixir_ale exited: :stopped
10:47:43.961 [info] Application instream exited: :stopped
10:47:43.965 [info] Application poolboy exited: :stopped
10:47:43.971 [info] Application poison exited: :stopped
10:47:44.059 [info] Application hackney exited: :stopped
10:47:44.093 [info] Application metrics exited: :stopped
10:47:44.100 [info] Application ssl_verify_fun exited: :stopped
10:47:44.103 [info] Application certifi exited: :stopped
10:47:44.115 [info] Application mimerl exited: :stopped
I assume this is because the Wifi connection is not established yet.
Right now there is only one FirmwareChannel
open between NervesHub and NervesHubWeb. I think as more features are implemented, it's going to be hard to wedge documentation of the messages passed back and forth over the socket and we should consider adopting a common message format/RPC format.
In other apps i usually use a format along the lines of:
@typedoc "Common RPC format"
@type t %__MODULE__{
kind: atom(),
args: map()
}
@type firmware_update_rpc :: %__MODULE__{
kind: :firmware_update,
args: %{required(:url) => String.t()}
}
@type some_other_rpc :: %__MODULE__{
kind: :some_other,
args: %{optional(:data) => number}
}
@typedoc "All possible RPCs that need to be implemented"
@type rpc :: firmware_update_rpc | some_other_rpc
I find that i frequently need to do one time things such as upgrade embedded firmware connected to my devices. I think allowing nerves_hub
to execute these one off tasks could be beneficial in a lot of cases. It could also allow a "status" of some sort to be updated on the
nerves hub web app if an error happens.
defmodule SomeUpgradeHook do
use NervesHub.PostUpgradeHook
def upgrade(old_version, new_version) do
Logger.info "upgrading from #{old_version} to #{new_version}"
case SomeEmbeddedDevice.upgrade() do
:ok -> :ok:
{:error, reason} -> {:error, "Failed to upgrade: #{reason}"}
end
end
end
** (CaseClauseError) no case clause matching: {:ok, %{"data" => %{"update_available" => false}}}
(nerves_hub) lib/nerves_hub.ex:13: NervesHub.update/0
Fix coming.
not 100% sure what is going on here..
** (EXIT from #PID<0.16173.6>) shell process exited with reason: an exception was raised:
** (FunctionClauseError) no function clause matching in RingLogger.Client.handle_info/2
(ring_logger) lib/ring_logger/client.ex:155: RingLogger.Client.handle_info({:io_reply, #Reference<0.2340004039.277086209.158264>, "Supervisor.which_children(FarmbotExt.Bootstrap.Supervisor)"}, %RingLogger.Client.State{colors: %{debug: :cyan, enabled: true, error: :red, info: :normal, warn: :yellow}, format: ["\n", :time, " ", :metadata, "[", :level, "] ", :levelpad, :message, "\n"], index: 0, io: :stdio, level: :debug, metadata: [], module_levels: %{}})
(stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:711: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Passing --framing
to fwup makes it send status back in an easy to part form. See https://github.com/fhunleth/fwup#integration-with-applications. This make it possible to send warnings, errors, and progress to the server. Currently, fwup's output is going to stdout and getting lost unless a console cable is attached.
Reproduction:
NervesHub.update/0
Explanation
The root of the issue is here https://github.com/nerves-hub/nerves_hub/blob/v0.4.0/lib/nerves_hub/http_client.ex#L47
We pass the private key to Certificate.pem_to_der()
here: https://github.com/nerves-hub/nerves_hub/blob/v0.4.0/lib/nerves_hub/certificate.ex#L25
which uses X509.Certificate.from_pem/1
in all cases, even though as in this case it is possible it will be called with a key, not a cert.
Solution
In this case, we should be calling X509.PrivateKey.from_pem/1
like Socket
does here: https://github.com/nerves-hub/nerves_hub/blob/v0.4.0/lib/nerves_hub/socket.ex#L63
--
Edit: In the wild this leads to a relatively cryptic error message:
** (exit) exited in: :gen_statem.call(#PID<0.1835.0>, {:start, 8000}, :infinity)
** (EXIT) an exception was raised:
** (MatchError) no match of right hand side value: {:error, {:asn1, {{:invalid_value, 0}, [{:asn1rt_nif, :decode_ber_tlv, 1, [file: 'asn1rt_nif.erl', line: 85]}, {:"OTP-PUB-KEY", :decode, 2, [file: 'OTP-PUB-KEY.erl', line: 1109]}, {:public_key, :der_decode, 2, [file: 'public_key.erl', line: 280]}, {:ssl_config, :init_private_key, 5, [file: 'ssl_config.erl', line: 121]}, {:ssl_config, :init, 2, [file: 'ssl_config.erl', line: 38]}, {:ssl_connection, :ssl_config, 4, [file: 'ssl_connection.erl', line: 602]}, {:tls_connection, :init, 1, [file: 'tls_connection.erl', line: 132]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}}
(public_key) public_key.erl:284: :public_key.der_decode/2
(ssl) ssl_config.erl:121: :ssl_config.init_private_key/5
(ssl) ssl_config.erl:38: :ssl_config.init/2
(ssl) ssl_connection.erl:602: :ssl_connection.ssl_config/4
(ssl) tls_connection.erl:132: :tls_connection.init/1
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
(stdlib) gen.erl:177: :gen.do_call/4
(stdlib) gen_statem.erl:598: :gen_statem.call_dirty/4
(ssl) ssl_connection.erl:2202: :ssl_connection.call/2
(ssl) ssl_connection.erl:122: :ssl_connection.handshake/2
(ssl) tls_connection.erl:90: :tls_connection.start_fsm/8
(ssl) ssl_connection.erl:91: :ssl_connection.connect/8
(ssl) ssl.erl:125: :ssl.connect/4
(hackney) /Users/daniel/dev/git/very-possible/some_client/some_repo_name/deps/hackney/src/hackney_connect.erl:279: :hackney_connect.do_connect/5
After burning my firmware with identifier on a RPI0
mix nerves_hub.device burn 0001
I'm not able to start my RPi0. It crashes:
15:29:17.647 [info] Application nerves_hub exited: exited in: NervesHub.Application.start(:normal, [])
** (EXIT) an exception was raised:
** (FunctionClauseError) no function clause matching in :pubkey_pem.join_entry/2
(public_key) pubkey_pem.erl:180: :pubkey_pem.join_entry([], [])
(public_key) pubkey_pem.erl:120: :pubkey_pem.decode_pem_entries/2
(nerves_hub) lib/nerves_hub/certificate.ex:6: NervesHub.Certificate.pem_to_der/1
(nerves_hub) lib/nerves_hub/socket.ex:15: NervesHub.Socket.configure/1
(nerves_hub) lib/nerves_hub/application.ex:13: NervesHub.Application.start/2
(kernel) application_master.erl:277: :application_master.start_it_old/4
I did discover that the contents of the nerves_hub_key and nerves_hub_cert in Nerves.Runtime.KV. only contain the first line of the specific files:
iex(9)> Nerves.Runtime.KV.get_all
%{
"a.nerves_fw_application_part0_devpath" => "/dev/mmcblk0p3",
"a.nerves_fw_application_part0_fstype" => "ext4",
"a.nerves_fw_application_part0_target" => "/root",
"a.nerves_fw_architecture" => "arm",
"a.nerves_fw_author" => "The Nerves Team",
"a.nerves_fw_description" => "",
"a.nerves_fw_misc" => "",
"a.nerves_fw_platform" => "rpi0",
"a.nerves_fw_product" => "zero_i2c",
"a.nerves_fw_uuid" => "14b1bcd7-b618-5cf9-dc77-3b286006628e",
"a.nerves_fw_vcs_identifier" => "",
"a.nerves_fw_version" => "0.1.0",
"nerves_fw_active" => "a",
"nerves_fw_devpath" => "/dev/mmcblk0",
"nerves_hub_cert" => "-----BEGIN CERTIFICATE-----",
"nerves_hub_key" => "-----BEGIN EC PRIVATE KEY-----",
"nerves_serial_number" => "0001"
}
I was able to reproduce the same error with only the first line of the certificate:
iex(14)> NervesHub.Certificate.pem_to_der("-----BEGIN CERTIFICATE-----")
** (FunctionClauseError) no function clause matching in :pubkey_pem.join_entry/2
The following arguments were given to :pubkey_pem.join_entry/2:
# 1
[]
# 2
[]
(public_key) pubkey_pem.erl:180: :pubkey_pem.join_entry/2
(public_key) pubkey_pem.erl:120: :pubkey_pem.decode_pem_entries/2
(nerves_hub) lib/nerves_hub/certificate.ex:6: NervesHub.Certificate.pem_to_der/1
I can confirm that the whole contents of the cert and key where put in the environment variables before the burn in r176 of Mix.Tasks.NervesHub.Device
def burn(identifier, opts) do
...
Shell.info("Burning firmware")
System.put_env("NERVES_SERIAL_NUMBER", identifier)
System.put_env("NERVES_HUB_CERT", File.read!(cert_path))
System.put_env("NERVES_HUB_KEY", File.read!(key_path))
IO.puts System.get_env("NERVES_HUB_KEY")
IO.puts System.get_env("NERVES_HUB_CERT")
Mix.Task.run("firmware.burn", [])
end
mix nerves_hub.device burn 0001
Nerves environment
MIX_TARGET: rpi0
MIX_ENV: dev
NervesHub org: dovadi
Burning firmware
-----BEGIN EC PRIVATE KEY-----
[the real contents of the key]
-----END EC PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
[the real contents of the cert]
-----END CERTIFICATE-----
Use 3.69 GiB memory card found at /dev/rdisk2? [Yn] y
|====================================| 100% (31.79 / 31.79) MB
Success!
and this is where I'm now ......
(BTW1: I'm using the latest version of nerves 1.3.0 and nerves_system_rpi0 1.4.0)
(BTW2: awesome work on nerves hub!)
when adding nerves_hub
as a dep to a new project having never created a nerves_hub account and signed in via the mix task, one cant make an account to create public keys because public keys don't exist yet.
https://github.com/nerves-hub/nerves_hub/blob/master/lib/nerves_hub/certificate.ex#L2
I'll have a PR for this tonight.
When a FWUP error occurs, the whole channel process is restarted, joins the channel in nerves-hub.org again, gets told to update, then fails etc etc. I'm not 100% sure why, but I definitely think we should not restart the whole channel process and instead decide if the error is recoverable and can be restarted, or just let it crash, keep the channel connected, and wait for the next update message.
This is definitely a big contributor to a device attempting crazy amounts of updates just from rejoining over and over. I'm attempting to adjust a bit on the nerves-hub.org side, but I think this would be a quick win here as well
I just ran into a case where I get stuck in a loop trying to install a fw update from nerves-hub.org, but it constantly fails with:
22:39:17.656 [info] [NervesHub] Downloading firmware: http://nerves-hub.org:4000/firmware/3/e9f997f4-0be5-5c10-a28c-a22260639dc4.fw
22:39:17.656 [debug] FWUP PROG: 0%
22:39:17.755 [debug] Stream Start: [{'cache-control', 'public'}, {'date', 'Thu, 02 May 2019 22:39:16 GMT'}, {'accept-ranges', 'bytes'}, {'etag', '3D15AF6'}, {'server', 'Cowboy'}, {'content-length', '28359000'}, {'content-type', 'application/octet-stream'}]
22:39:17.827 [error] FWUP ERROR: Please validate the running firmware before upgrading it again.
The case is I built and installed a firmware manually (mix firmware && ./upload.sh my_device
). On boot, it quickly connected to nerves-hub.org (my local instance) and was told to update but apparently didn't quite finish validating the firmware, so it got this error.
This seems normal except the error crashes the whole channel process, it rejoins, gets told to update again, and is still not validated (see #95)
This seems like an error we could handle on the nerves_hub
side to either wait for validation, or report an error, etc pending the fix of #95
this commit
tries to pull the ca certs from the wrong application, OR the certs in nerves_hub_core
are wrong.
The following fixes the problem
env NERVES_HUB_CA_CERTS=$PWD/deps/rpi/nerves_hub/priv/ca_certs MIX_TARGET=rpi mix do deps.compile nerves_hub --force, firmware
Just a few questions about the ssh certificate that is being generated upon registering for NervesHub.
Most users should be familiar with GitHub's way of setting up an ssh certificate that's used by (commandline) git to authenticate the user. There are guides here: https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
Therefore:
Maybe also refer to https://help.github.com/en/articles/working-with-ssh-key-passphrases ?
Right now Toolshed
doesn't work with the remote IEx sessions. In fact, it basically crashes it if you try to send use Toolshed
as a command from NervesHub.
This would be a "nice have" for the remote IEx work.
A device key generated like so:
$ openssl ecparam -out key_to_fargo -name prime256v1 -genkey
Produces:
-----BEGIN EC PARAMETERS-----
BggqhkjOPQMBBw==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIGMAZLmpZ6fHm5aqk0fr4Fuo8SOkQsN8bEOa5CyzuU7MoAoGCCqGSM49
AwEHoUQDQgAEArRjNE8YmfVlhyRhI17ErmL98ozkUu6n1WSVPhZfRhrhZYg1Agi4
52xddjspxauP7vbvoiOxN7Xg5/IaX74NRw==
-----END EC PRIVATE KEY-----
Which chokes here:
https://github.com/nerves-hub/nerves_hub/blob/master/lib/nerves_hub/certificate.ex#L27
Like this:
** (MatchError) no match of right hand side value: [{:EcpkParameters, <<6, ...>>, :not_encrypted}, {:ECPrivateKey, <<48, ...>>, :not_encrypted}]
(nerves_hub) lib/nerves_hub/certificate.ex:27: NervesHub.Certificate.pem_to_der/1
(nerves_hub) lib/nerves_hub/http_client.ex:98: NervesHub.HTTPClient.ssl_options/0
(nerves_hub) lib/nerves_hub/http_client.ex:87: NervesHub.HTTPClient.opts/0
(nerves_hub) lib/nerves_hub/http_client.ex:31: NervesHub.HTTPClient.request/3
(nerves_hub) lib/nerves_hub.ex:13: NervesHub.update/0
It would be nice if we supported this, but I imagine there are a lot of things that could be in that key to account for.
Dropping that section from the key fixes the issue, and key generate can add -noout
to the command to ensure that section isn't output into the generated key:
$ openssl ecparam -out key_to_fargo -name prime256v1 -genkey -noout
Currently, when a new firmware is published, nerves_hub will always try to download and apply it. This isn't always a ideal behavior. It's be nice to have defined hooks that can influence the decision to download/apply the firmware. Something like this: (but with a better name)
defmodule FirmwareHandler do
@typedoc "Map of meta information about a firmware"
@type meta() :: map()
@callback should_download_firmware(meta) :: boolean
@callback should_apply_firmware(meta) :: boolean
end
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.