thethingsnetwork / lorawan-stack Goto Github PK

View Code? Open in Web Editor NEW

945.0 945.0 300.0 160.74 MB

The Things Stack, an Open Source LoRaWAN Network Server

Home Page: https://www.thethingsindustries.com/stack/

License: Apache License 2.0

Go 74.05% Makefile 0.02% JavaScript 24.15% Shell 0.01% Dockerfile 0.01% HTML 0.01% Stylus 1.70% Lua 0.03% CSS 0.03%

iot lora lorawan ttn

lorawan-stack's People

Contributors

Stargazers

Watchers

Forkers

irnas schulti mfalkvidd iothie adamsondelacruz thing12 marceluphd blueff thirdgen88 digeshkapadia mbrukman data-alliance rvolosatovs susiloharjo sypheos jhpark041 madjack101 bourne-h silentcld m-gregoire rongatrabbit tzuchiangshen hubertxie jlcoleman utkarshns cortex-media gqadonis hristijanstojchevski jastegmann creativeone86 mutiande rugirundao sebastianbuettrich twowinter zero2eight kevinstronger hishamjaffar98 amoryin tiger902 mmichel6 huei8888 carmenconekta nathanscarfi97 psjb2008 isgasho virtualguy peterhadac tacruc knowgis ledx iotfab kschiffer charuga hgqian guardianrg danielizquierdo ivancho523 backwardn gabrielastudillo swattor bzuellyp kurtmc mattdean-digicatapult sebastienchevalier neoaggelos kulpreet05 emilppeter homescriptone smartiotmaker yanwu12138 johnthomasjtk foxxnuaa jasdeepgill kat1627 needlecandy mikephung63 codingkeeper nj-ramadhan jinmin-chen halter-corp waterofbigriver cybereyeresearchlabs thanamate codyshao oliv4945 durgaprasadwebdev raghav-aggarwal mansour1991 requea fsaf11 xbotninja veghead czorgormez mihirr93 johanpot2 xjc01288 jancee atuld01 a-tsuchiya1323 jpmeijers

lorawan-stack's Issues

Come up with a proper way to display errors in frontend

Summary:
At the moment there is no real error handling on the frontend. We should make use of React's Error Boundaries here to scope error messages to their concerning components, as long as this is possible.
Additionally, we need a proper error page that will be shown when fatal errors occur that prevents the app from loading. This error page should match the static ones that are rendered by the backend.

Why do we need this?
Handling errors gracefully is essential for a UX that minimizes frustration. We should if possible also provide hints to the users as to how specific errors might be resolved.

What is already there? What do you see now?
There is some fragmented error handling already in place eg. for the login screen. Also we already have the necessary backend functionality to get error messages that have all the info that we need, as well as i18n capabilities. The <ErrorMessage />-component is specifically for this.

If the backend encounters an error in HTTP under /console or /oauth, it now renders HTML (if requested by browser in Accept header) with the error JSON in PAGE_DATA.error.

A request curl -H "Accept: text/html" -XPOST http://localhost:1885/oauth/api/auth/login results in the following (formatted for readability):

<!doctype html>
<html lang="en">

<head>
    <!-- ... -->
</head>

<body>
    <div id="app"></div>
    <script>
        window.APP_ROOT = "/oauth";
        window.ASSETS_ROOT = "http://localhost:1885/assets";
        window.APP_CONFIG = { "language": "en" };
        window.PAGE_DATA = {
            "error": {
                "code": 2,
                "message": "error:pkg/errors/web:http (HTTP error: code=403, message=invalid csrf token)",
                "details": [
                    {
                        "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
                        "namespace": "pkg/errors/web",
                        "name": "http",
                        "message_format": "HTTP error: {message}",
                        "attributes": {
                            "message": "code=403, message=invalid csrf token"
                        }
                    }
                ]
            }
        };
    </script>
    <script type="text/javascript" src="http://localhost:1885/assets/oauth.b4ebca19adbb1b50d911.js"></script>
</body>

</html>

Errors are structured in the same way as the errors you receive from the API, so with the details array, where you need to find the object with "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails" to get TTN error details.

We probably need a broader error message strategy. Four error message scenarios come to mind:

A full page error message -> #233
A component error overlay
An in-place error message (e.g. for invalid login credentials; after submit, not field validation), should probably be part of the component (this is more or less implemented via the <Notification /> component which can take error objects for display)
Error notifications/toasts ( -> this was implemented via #139 )

I propose the following designs:
The component error overlay I could envision like this:

In page example:

So it would be sort of an enhanced form of <Notification />, which spans the whole width of the errored component and displays a specific support message. This example shows an error of the device list, which I don't know whether we should provide an error boundary for. But I think it's good to show how the error overlay would work.

What is missing? What do you want to see?
A clear strategy for implementing this.

How do you propose to implement this?
By adding Error Boundaries via adding componentDidCatch() to all components in the webui that display fetched data and possibly adding a flag that will render the component as errored, which we should already add when we implement new components. This should also be put into storybook.

What can you do yourself and what do you need help with?
I can do this.

cc @bafonins

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1308 by @kschiffer

Use Mage instead of Make and clean development tooling

Summary:

Our development, build and CI tooling is getting a bit out of hand. Makefiles are complicated and annoying, so we decided to switch to Mage which allows us to write our development, build and CI tooling in Go.

What is already there? What do you see now?

We have simply written too many Make rules that aren't used, or give unexpected results. The dependency graph has more resemblance to spaghetti than to a directed acyclic graph. Variables are overwritten in multiple places in different files.

What is missing? What do you want to see?

I'd like to see a clearer separation in build targets, so that we can split our CI jobs in properly scoped containers:

Go quality: Testing, Linting, etc.
JS quality: Testing, Linting, etc.
Build: Full build with clear dependency flow (first JS, then Go).
Checking the repo: Regenerate protos, messages, etc. and check for diffs.

The last job ("Checking") is added so that the other jobs can run in parallel. Even though Go tests use .pb.go files, we don't need to regenerate them, because we can assume that any changes would already be built and committed to the repo. The "Checking" job will make sure that this is indeed the case before PRs are merged.

Obviously, the Go tests should not invoke yarn or webpack, because our Go tests can run just fine without the webui hashes.

The other way around could be a bit harder, because Mage requires Go. Therefore we may have to find a nice solution for running our JS quality builds in a container without Go.

How do you propose to implement this?

For a smooth transition we should probably start migrating Make rules to Mage one by one. The old Make rules can just call Mage until everyone is used to Mage.

I think it's important to consider the differences between development and CI. In development environments, both Go and NodeJS are installed, but on CI this may not be the case, as Go and JS jobs could run in different containers (with different images). The mage binary could be built separately of the actual jobs (see below), but it should still avoid calling programs that are specific to one environment (generating protos, calling node_modules/.bin/json, etc.). The simplest way to avoid these kind of problems is to avoid dependencies (so Go testing should not depend on the version/ttn.go being generated by a task that also calls node_modules/.bin/json). We can add more higher-level tasks for development that combine multiple lower-level tasks.

Running JS quality jobs in a node container could be a challenge. Perhaps a multi-stage build with Github Actions instead of Travis could help here. The first action could generate the .mage/run binary, which is then used by the subsequent js quality action.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/712 by @htdvisser

Remove otto module import on private fork

Blocked by robertkrimen/otto#283

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/452 by @johanstokking

Send uplink messages from gateways

Summary:

Send uplink messages from gateways, i.e. messages with invert polarization off.

Why do we need this?

For network performance testing, i.e. hear other gateways, check each other's link quality, etc.

What is already there? What do you see now?

Nothing, only a NS driven downlink flow.

What is missing? What do you want to see?

Two APIs to start with;

To send messages to a given gateway identifiers, with given TxSettings. We may chose to provide an UplinkMessage for simplicity, including DevAddr that is dedicated to testing
A streaming API to receive such messages, already filtering on a given DevAddr and NwkSKey for MIC check

How do you propose to implement this?

We may build a specific monitoring service that implements GsNs to receive messages from the GS automatically for a dedicated testing DevAddr. I don't think we should use NsGs for sending such messages, as we need fine grained control over TxSettings and this would be a separate flow.

Would be nice to build this in in the CLI to see the echo from the gateways that received the message with their RX metadata.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1385 by @johanstokking

Console logs expected errors on SIGINT

Summary:

When exiting the stack, it logs expected errors.

Steps to Reproduce:

Start the binary
Stop the binary

What do you see now?

  INFO Received signal, exiting...              signal=interrupt
ERROR transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing" namespace=grpc
 ERROR Could not read incoming UDP packets      error=read udp [::]:1700: use of closed network connection
 ERROR Error in Listener [::]:8885              error=accept tcp [::]:8885: use of closed network connection
 ERROR transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing" namespace=grpc
 ERROR Error serving HTTP on [::]:8885          error=mux: listener closed
 ERROR Error in Listener [::]:1884              error=accept tcp [::]:1884: use of closed network connection
 ERROR Error serving gRPC on [::]:1884          error=mux: listener closed
 ERROR Error in Listener [::]:8884              error=accept tcp [::]:8884: use of closed network connection
 ERROR Error serving gRPC/tls on [::]:8884      error=mux: listener closed
 ERROR Error in Listener [::]:1885              error=accept tcp [::]:1885: use of closed network connection
 ERROR Error serving HTTP on [::]:1885          error=mux: listener closed
 ERROR transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing" namespace=grpc
  WARN transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:51073->127.0.0.1:51074: use of closed network connection

What do you want to see instead?

  INFO Received signal, exiting...              signal=interrupt

Environment:

Any

What can you do yourself and what do you need help with?

...

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/841 by @htdvisser

Cache unwrapping keys to reduce load on key vault

Summary:

Cache unwrapping keys to reduce load on key vault

What do you see now?

Unwrapping keys on-the-fly, also inside loops.

What do you want to see instead?

Keys should not be stored (long) in memory, but they can be cached briefly within the same function.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1218 by @johanstokking

Support downlink to multiple antennas

Summary:

Support multiple antennas for downlink.

Why do we need this?

To have more downlink paths.

What is already there? What do you see now?

The Gateway Server currently only provides uplink tokens for antennas with index 0 and sets the downlink path constraint "never" on other antennas.

What is missing? What do you want to see?

Treat all antennas equal; provide uplink token and inherit the gateway's downlink path constraint.

How do you propose to implement this?

We need answers to the following questions;

Should duty-cycle be enforced per antenna or per physical gateway?
How to specify the antenna index in the UDP protocol?

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1406 by @johanstokking

Allow use of unauthorized OAuth clients by collaborators

Summary:

We should allow collaborators of OAuth clients to use those clients before they are approved.

Why do we need this?

Would be very helpful for development.

What is already there? What do you see now?

Regular OAuth flow that rejects requests for "REQUESTED" clients.

What is missing? What do you want to see?

Before rejecting the request based on REQUESTED state, check if the current user collaborates on the OAuth client and then still allow the authorization.

How do you propose to implement this?

As described above, altough we should probably discuss the consequences first. Especially considering that you could add anyone as collaborator on the Client without them having to approve their membership.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1414 by @htdvisser

Join server lookup and authorization

Summary:

Join server lookup and authorization for external join servers.

To support this inbound, we need #717

Why do we need this?

A device may use a JS external to the cluster
AS may need to fetch the AppSKey from the JS

What is already there? What do you see now?

Cluster-local JS with cluster authentication

What is missing? What do you want to see?

Support for looking up an external JS and using TLS client authentication.

How do you propose to implement this?

If there's a cluster-local JS, try that JS. If that works, great, we're done. If it can't handle the join request (for NS) or it can't find the AppSKey (for AS), it returns an error and go to 2. This way, the cluster-local JS overrides an external JS
The NS and AS lookup by DNS the JS and retry the operation. The DNS lookup is <eui>.joineuis.lora-alliance.org, but we may make that domain suffix configurable and support a list on which we attempt lookup in order. The <eui> is the reverse hex representation with dots on each nibble, i.e. 0.0.0.0.0.0.0.d.e.7.5.d.3.b.0.7.joineuis.lora-alliance.org

Once we have an IP address, we need to figure out the protocol to use. Proposing a manual DNS lookup;

Lookup the SRV records of the record. If there are V3 services (#138), use gRPC. If the record is a CNAME, go to step 2. Otherwise, use Backend Interfaces
Try 1 again with the canonical name

Console support for collaborators for entities

Summary:

The console should provide functionality to manage collaborators of top-level entities:

applications (https://github.com/TheThingsIndustries/lorawan-stack/issues/1065)
OAuth clients (#4300 )
gateways (https://github.com/TheThingsIndustries/lorawan-stack/issues/1068)
organizations (#1205)

This includes:

Listing collaborators
Adding collaborators
Updating collaborator rights
Removing collaborators

What is already there? What do you see now?

Screen design for the collaborators page within the applications section

What is missing? What do you want to see?

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1069 by @bafonins

Implement rejoin flow

Summary:

Rejoin flow should be supported

Why do we need this?

To be compliant with LoRaWAN1.1 spec

What is already there? What do you see now?

No rejoin flow handling

What is missing? What do you want to see?

Rejoin flow handling

How do you propose to implement this?

It should be in big part similar to Join-Request handling

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/557 by @rvolosatovs

Console support for gateways

Summary:

The console should provide functionality to manage gateways, namely:

listing gateways (paginated)
- list all gateways the caller has access to
- list all gateways for a user
- list all users for an organization
getting a single gateway
creating (adding) gateways
deleting (removing) gateways
updating gateways
display connection statistics per gateway (if the gateway_server_address matches the Gateway Server API config of the current console)

What is already there? What do you see now?

The Gateway entity:

Standard fields ids, created_at, updated_at, name, description, attributes, contact_info
version_ids: optional object with identifiers of the gateway's hardware and firmware
gateway_server_address: string with the hostname of the GS the gateway should connect to
auto_update: boolean to indicate whether to automatically update the gateway firmware
update_channel: string of the update channel for the gateway, can be URL or identifier (stable, beta, ...)
frequency_plan_id: string of the frequency plan that the gateway uses. For now manually entered, but in the future we should add some API for getting a list of possible values for this and turn this into a dropdown.
antennas: optional (@johanstokking?) list of objects representing gateway antennas and their locations
status_public: boolean indicating whether the status of this gateway is publicly visible
location_public: boolean indicating whether the locations of this gateway's antennas are publicly visible
schedule_downlink_late: boolean indicating whether downlinks should be queued by the gateway server; needed if the gateway does not have a JIT queue
enforce_duty_cycle: boolean indicating whether regulatory duty cycles should be enforced for this gateway. Typically true.

The Identity Server's Gateway Registry API:

Create(CreateGatewayRequest)
POST /api/v3/users/{collaborator.user_ids.user_id}/gateways
POST /api/v3/organizations/{collaborator.organization_ids.organization_id}/gateways
Get(GetGatewayRequest)
GET /api/v3/gateways/{gateway_ids.gateway_id}
List(ListGatewaysRequest)
GET /api/v3/gateways
GET /api/v3/users/{collaborator.user_ids.user_id}/gateways
GET /api/v3/organizations/{collaborator.organization_ids.organization_id}/gateways
Update(UpdateGatewayRequest)
PUT /api/v3/gateways/{gateway.ids.gateway_id}
Delete(GatewayIdentifiers)
DELETE /api/v3/gateways/{gateway_id}

The Identity Server's Gateway Access API:

CreateAPIKey(CreateGatewayAPIKeyRequest)
POST /api/v3/gateways/{gateway_ids.gateway_id}/api-keys
ListAPIKeys(GatewayIdentifiers)
GET /api/v3/gateways/{gateway_id}/api-keys
UpdateAPIKey(UpdateGatewayAPIKeyRequest)
PUT /api/v3/gateways/{gateway_ids.gateway_id}/api-keys/{api_key.id}
SetCollaborator(SetGatewayCollaboratorRequest)
PUT /api/v3/gateways/{gateway_ids.gateway_id}/collaborators
ListCollaborators(GatewayIdentifiers)
GET /api/v3/gateways/{gateway_id}/collaborators

The Gateway Server's API:

GetGatewayConnectionStats(GatewayIdentifiers)
GET /api/v3/gs/gateways/{gateway_id}/connection/stats

and the connection stats object containing information about the current connection of the gateway (this changes often):

connected_at timestamp
protocol string
last_status_received_at timestamp
last_status object
last_uplink_received_at timestamp
uplink_count number
last_downlink_received_at timestamp
downlink_count number

What is missing? What do you want to see?

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1068 by @bafonins

Improve gateway sorting algorithm in NS

Summary:

Gateway sorting algorithm in NS can be improved

Why do we need this?

To make better gateway choices

What is already there? What do you see now?

A very simplistic gateway sorting algorithm

What is missing? What do you want to see?

A more efficient gateway sorting algorithm

How do you propose to implement this?

Based on experience with v2:

SNR isn't the only metric; RSSI is also important.
We may need to filter "incorrect data" (SNR=0, RSSI=0 for example)
We may want to include gateway uplink utilization to avoid Tx on "valuable uplink gateways"
We may want to include gateway downlink utilization to be a little bit smarter about downlink gtw
selection than always selecting one with a full duty cycle
We may benefit from a little bit of randomness

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/729 by @rvolosatovs

Make duty-cycle window configurable

Summary:

Make duty-cycle window configurable.

Why do we need this?

Because users may want to prefer balancing in time over allowing short-term bursts that are even out over long windows.

What is already there? What do you see now?

Fixed duty-cycle window of 1 hour.

What is missing? What do you want to see?

Gateway Server configuration setting.

How do you propose to implement this?

Add time.Duration to Gateway Server component config. Best would be to make this a parameter all the way down to the sub-band scheduler. Would be nice to add a struct with scheduler settings for that.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1386 by @johanstokking

Encrypt session keys

Summary:

Pass and store keys encrypted in the key envelopes.

Why do we need this?

For secure transfer and encryption at rest.

What is already there? What do you see now?

We have key envelopes everywhere that can contain wrapped keys with a key encryption key (KEK) label. We also have key vaults with wrapping and unwrapping functionality in the base component. Finally, we have configuration for key vaults.

What is missing? What do you want to see?

We need to wrap the keys. The scope of this issue is the transfer of session keys from JS to NS and AS.

For this issue, we assume that KEKs are exchanged out-of-band. We may come up with bilateral KEK handshake using Diffie-Hellman or something later.

In any case, the server that wraps the key needs to know which KEK the client has. The server chooses the KEK based on the connection context.

How do you propose to implement this?

Proposed order of preference;

If there is cluster authentication, use a cluster KEK
If there is network authentication, use a network KEK
If there is TLS client authentication (i.e. external AS and NS), use a mapping of client certificate to pre-shared KEK
If there is only TLS server authentication, do not wrap keys and send warnings
If there is neither cluster, network, TLS client nor server authentication, do not return keys at all

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/271 by @rvolosatovs

Use race detector in Go tests

Summary:

There are quite some race conditions in our Go codebase, adding of which could be prevented by enabling the Go race detector. Possibly, it could run only on PRs not to slowdown CI too much.

Packages to fix(on https://github.com/TheThingsIndustries/lorawan-stack/commit/b00bbf3df9d2aaa23a2bd41c9c0864eb37f2804f):

go.thethings.network/lorawan-stack/pkg/component
go.thethings.network/lorawan-stack/pkg/gatewayserver/scheduling
go.thethings.network/lorawan-stack/pkg/networkserver
go.thethings.network/lorawan-stack/pkg/util/test/assertions
go.thethings.network/lorawan-stack/pkg/web/middleware

Steps to Reproduce:

make go.test should specify -race flag to go test

What do you see now?

It does not happen

What do you want to see instead?

It happening

Environment:

...

What can you do yourself and what do you need help with?

I created the feature/race-test branch, where the tests(because of race conditions) are failing due to race conditions.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/745 by @rvolosatovs

Add repo-scoped linters

Summary:

Guideline conformity verification should be automated as much as possible. However, we currently only use global Go linters for our Go code - which means that those linters do not reflect repo-specific rules, such as our guidelines.

We may also want to inverstigate protocol buffer linting(e.g. https://github.com/ckaznocha/protoc-gen-lint)

Why do we need this?

To ensure consistency and do so efficiently

What is already there? What do you see now?

~~gometalinter generic linter rules~~ Generic revive rules

What is missing? What do you want to see?

Custom linting rules

How do you propose to implement this?

~~Investigate https://github.com/mgechev/revive~~
Investigate https://github.com/ckaznocha/protoc-gen-lint (last time I checked, MHDR and FHDR fields did not "pass" the lint checks)

Ideas of guidelines to lint:

Variable names: e.g. single-word entities
Conventions on struct usage
Event definitions: no capitalizing, event names in snake_case.with.dots.allowed
- Ensure the event name follows the component.entity.action(=verb) format
Error definitions: no capitalizing, no could_not, no failed, error names in snake_case
- Give a warning on errors.New
Formatting that isn't handled by regular linters
Comments

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/900 by @egourlao

Add optional IP whitelisting for API Keys

Summary:

Add a whitelist of IP address ranges to API keys.

Why do we need this?

API keys can be quite sensitive, especially considering that they do not expire. A whitelist of IP address ranges that are allowed to use a specific API key could improve security.

What is already there? What do you see now?

Components query the Identity Server for the rights that an API key gives the caller.

What is missing? What do you want to see?

A field with IP address ranges in the API key.
Components should include the caller's IP address when requesting rights info.

How do you propose to implement this?

Add a list of IP ranges (prefixes) to the APIKey model
We can use the X-Forwarded-For header, but that means that this header needs to be forwarded by the rights hook.
After the IS fetches the API key from the DB, it can first check the IP ranges, so that we don't have to hash the key if the IP address already doesn't match.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/86 by @romeovs

Enable test generation for customized protos

Summary:

We should enable test generation for protos, for which we define MarshalJSON, MarshalText and similar

Why do we need this?

Custom marshaling logic we add may break things.

What is already there? What do you see now?

No tests

What is missing? What do you want to see?

Tests for customized protocol buffers

How do you propose to implement this?

Enable tests in gogoproto, for customized protocol buffers

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1104 by @rvolosatovs

Console support for applications

Summary:
The console should provide functionality to manage applications, namely:

listing applications (short and full overview)
- list applications that the user has rights on (paginated) by default
adding applications
removing applications
updating applications

What is already there? What do you see now?
Screen design for the application page.

What is missing? What do you want to see?
Components:

Pages:

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1065 by @bafonins

AS linking events and statistics rpc

Summary:

Publish link/unlink events and get statistics of the AS-NS link.

Why do we need this?

If the given API key is or has become invalid, if the host can't be found, etc, it can only be learned from the logs which are not accessible in a hosted network.

What is already there? What do you see now?

Get and set the link, but no info whether it works.

What is missing? What do you want to see?

Publish link/unlink events
An endpoint to get statistics, much like the GS connection info statistics

How do you propose to implement this?

Publish events from AS. Also, a new rpc in As specifically for link statistics. Requires link rights.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1432 by @johanstokking

Two-factor Authentication

Summary:
Add two-factor authentication

Why do we need this?
Better security

What is already there? What do you see now?
Only single-factor authentication

What is missing? What do you want to see?
Support Authenticator app and u2f + recovery codes

How do you propose to implement this?
As suggested on Slack and on the Forum: add support for FIDO U2F. This library seems to implement it in Go: https://github.com/tstranex/u2f

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/99 by @htdvisser

Tracking consumed uplink/downlink Airtime

Summary:

We should be able to track the consumed airtime for messages, devices and applications.

What is already there? What do you see now?

We have it in v2, but nothing in v3

What is missing? What do you want to see?

Add google.protobuf.Duration consumed_uplink_airtime and google.protobuf.Duration consumed_downlink_airtime to the EndDevice proto message
Add a google.protobuf.Duration consumed_airtime to ApplicationUplink
Add a google.protobuf.Duration consumed_airtime to ApplicationDownlink (or whatever we send in the ApplicationUp.downlink_sent field

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1198 by @htdvisser

Retry failed registry transactions

Summary:

Components should retry failed registry transactions if possible

Why do we need this?

To gracefully handle errors

What is already there? What do you see now?

The RPC fails

What is missing? What do you want to see?

Retries of the transaction

How do you propose to implement this?

Depends on the concrete case

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1163 by @rvolosatovs

Version check and release notes in Console

Summary:

We should have release notes for new versions
These release notes cannot be linked to the public repository GitHub releases, as some functionality is in this private repository
We should make it available easily, I would propose the Console

Why do we need this?
It makes sense to use the console for this as it is where most of the people this would concern should be. Also then it's available in any deployment model (i.e. on-premises and SaaS)

What is already there? What do you see now?

The release notes (extracted from our issues)

What is missing? What do you want to see?

A screendesign and concept for where the release notes should appear
Can we fetch GitHub releases and their notes from this private repository in Markdown format and display them in the Console? cc @htdvisser @kschiffer

Class B support

Summary:

Class B devices should be supported

Why do we need this?

To support Class B devices and make end customers happy

What is already there? What do you see now?

No class B support

What is missing? What do you want to see?

Class B support

How do you propose to implement this?

According to the spec

Notes:

In class C, there is a timeout within which the device can respond to downlink traffic (i.e. MAC answer, downlink ack). During this time, NS may not send other downlink. This is not specified for class B; i.e. there is no requirement for the device to answer downlink traffic in ping slots before the next ping slot, and there is no requirement for the NS to hold traffic while the device has not responded. When implementing this, we may want to consider confirmed downlink (so that the device responds per spec, before the next ping slot), like we do with class C. Background here: #412 (comment)

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/833 by @rvolosatovs

Find a solution for concurrency-safe pseudorandom RNG

Summary:

Read method of rand.Rand is not safe for concurrent use even while using a locked source, see implementation.

Why do we need this?

"pseudorandom" RNG is good enough for ULID generation and faster

What is already there? What do you see now?

crypto/rand is used for ULID generation

What is missing? What do you want to see?

use "pseudorandom" RNG for ULID generation

How do you propose to implement this?

https://www.godoc.org/golang.org/x/exp/rand exists, which exports the LockedSource and provides thread-safe Read on the Rands created using it, however there currently is no way to construct a LockedSource except from inside the package. We may want to keep an eye on that package and use it once this changes.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/718 by @rvolosatovs

ADR profiles

Summary:

ADR profiles give fine grained control to which strategy is being followed per end device.

What is already there? What do you see now?

Basic ADR algorithm. This is essentially the "low power" profile.

What is missing? What do you want to see?

I'd suggest three profiles, something like this:

Low power: highest data rate, still downlink path available (even if it were RX2). Useful for applications that use battery powered sensors (i.e. uplink oriented)
Downlink availability: reach gateways that have a downlink path available in RX1. Useful for applications that use quite some downlink. Parameters:
- target number of gateways with downlink available in RX1 (i.e. 2)
- minimum allowed data rate (i.e. SF10)
Reach: reach gateways for more critical applications or geolocation. Useful for applications that need geolocation (tags that enable ADR when the device becomes stationary) or if you want to reduce packet loss as much as possible. Parameters:
- target number of gateways with decent reception (i.e. 3)
- minimum allowed data rate (i.e. SF10)

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1258 by @johanstokking

Retry downlink on Tx failure

Summary:

Retry downlink on Tx failure.

Why do we need this?

To perform a best-effort of (un)confirmed downlink.

What is already there? What do you see now?

The GS scheduler already detects and avoids downlink conflicts.

We already have a retry on the end device to NS level for confirmed downlink.

What is missing? What do you want to see?

Regardless of our amazing GS downlink scheduler, the gateway may still send a Tx acknowledgment with error code. In that case, even for unconfirmed downlink, the GS should try again.

How do you propose to implement this?

Once a downlink fail acknowledgement is received, unset all pending MAC state fields and:

call updateDataDownlinkTask

lorawan-stack/pkg/networkserver/downlink.go

Line 138 in 36d9597

func (ns *NetworkServer) updateDataDownlinkTask(ctx context.Context, dev *ttnpb.EndDevice, earliestAt time.Time) error {

for data downlink fail. If possible, recreating MACState.QueuedResponses would be best

recreate QueuedJoinAccept and add a downlink task

lorawan-stack/pkg/networkserver/grpc_gsns.go

Lines 1185 to 1193 in 36d9597

    
           // TODO: Extract this into a utility function shared with mac.HandleRejoinRequest. (https://github.com/TheThingsNetwork/lorawan-stack/issues/8) 
        
           downAt := up.ReceivedAt.Add(-infrastructureDelay/2 + phy.JoinAcceptDelay1 - macState.DesiredParameters.Rx1Delay.Duration()/2 - nsScheduleWindow()) 
        
           if earliestAt := time.Now().Add(nsScheduleWindow()); downAt.Before(earliestAt) { 
        
           	downAt = earliestAt 
        
           } 
        
           logger.WithField("start_at", downAt).Debug("Add downlink task") 
        
           if err := ns.downlinkTasks.Add(ctx, stored.EndDeviceIdentifiers, downAt, true); err != nil { 
        
           	logger.WithError(err).Error("Failed to add downlink task after join-request") 
        
           }

for join accept fail

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/794 by @egourlao

Consider moving to babel 7

Summary:
Babel 7 has been released and there introducing quite many changes that can affect us as well.

stage presets are deprecated. we include babel-preset-stage-0.
scoped packages, babel-* -> @babel/*

At the moment the only package requiring the newest babel version is @storybook/react, but I can image that later there will be more. This issue is rather low priority and we can using babel 6 for quite a while.

For reference, the announcement post.
Especially the major breaking changes, to quote:

Drop support for un-maintained Node versions: 0.10, 0.12, 4, 5 (details)

Move us to the @babel namespace by switching to using "scoped" packages (details). This helps differentiate official packages, so babel-core becomes @babel/core (and no squatting)

Remove (and stop publishing) any yearly presets (preset-es2015, etc) (details). @babel/preset-env replaces the need for these as it includes all yearly additions as well as the ability to target a specific set of browsers

Also drop the "Stage" presets (@babel/preset-stage-0, etc) in favor of opting into individual proposals. Similarly remove proposals from @babel/polyfill by default (details). Please consider reading the whole post on this for more explanation.

Some packages have renames: any TC39 proposal plugin will now be -proposal instead of -transform (details). So @babel/plugin-transform-class-properties becomes @babel/plugin-proposal-class-properties.

Introduce a peerDependency on @babel/core for certain user-facing packages (e.g. babel-loader, @babel/cli, etc) (details)

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1186 by @bafonins

Integrate location solvers

Summary:

After handling an uplink message, solve the location of the message.

What is already there? What do you see now?

Nothing.

What is missing? What do you want to see?

Configuration in application link on which location solvers to use
Interface for calling the location solvers, like message processors
Send results in ApplicationUp; message already exists

Discussion:

We need to figure out if the AS, which in principle only has link rights, may update the end device's location in the Entity Registry.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1221 by @johanstokking

Redis task deduplication strategy

Summary:

Currently the start_at of the first task with payload X is used as the score of task X, however we may want to use the minimum of the two or maybe come up with some other way to determine, which task "wins"

Why do we need this?

We usually want to be able to execute tasks ASAP

What is already there? What do you see now?

First wins

What is missing? What do you want to see?

Minimum wins or custom

How do you propose to implement this?

Do a ZSCORE in a watch block, if nil, just ZADD NX, otherwise, either:

Always update the score to reflect the minimum
Let the caller decide, which wins(e.g. by passing a function like func(time.Time, time.Time) bool)

Environment:

Everything

What can you do yourself and what do you need help with?

I can implement it

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1380 by @rvolosatovs

Validate the field mask per component in API implementation

Summary:

Validate the field mask per component in API implementation.

What is already there? What do you see now?

No checks.

What is missing? What do you want to see?

This issue is specifically about enforcing in the RPC service implementation that the given field mask is valid for the concerning component.

This is really per gRPC call. We can't use component-level checks, because the component may work with fields that it doesn't expose through the API. We can't use service-level checks, because some fields are valid for getters and not for setters.

I'm proposing the following:

// Get implements ttnpb.AsEndDeviceRegistryServer.
func (as *ApplicationServer) Get(ctx context.Context, req *ttnpb.GetEndDeviceRequest) (*ttnpb.EndDevice, error) {
	if err := fieldmask.Validate(ctx, req.FieldMask, ttnpb.AsEndDeviceRegistryServerGetMask); err != nil {
		return nil, err
	}
...

Here, fieldmask.Validate() would return an error if the second parameter is not a subset of the second.

In this case, ttnpb.AsEndDeviceRegistryGetMask would be []string{"formatters"}.

CC @htdvisser

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1226 by @johanstokking

Code generation for calling API endpoints

Summary:
Currently, api methods for the http connector in the js-sdk are generated at runtime from a json schema that contains all api defintions.

What is already there? What do you see now?
All api methods are generated at runtime when the main instance gets instantiated on the client. The api-defintions.json gets parsed and for each entry a method is created in the api module.

What is missing? What do you want to see?
A set of generated es6 methods (during build) that will delegate all api calls to axios and can be published within the sdk, without the need to include redundant json files and generate the methods at runtime.

This shouldnt be too hard, since all methods have the same structure, only the arguments differ.

Worth noting, that the current implementation should stay until the draft sdk is finished, as well as the backend api, and can be tested.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1137 by @bafonins

JavaScript SDK

Summary:
Since we would need abstractions on the stack API for the Console anyway, we decided to start building the JavaScript SDK alongside developing the Console. Here, it is planned to combine usage for the browser and Node.JS in a single SDK. The usage of the HTTP Bridge or gRPC directly can then be chosen as a config during instantiation of the SDK API object, depending on the capabilities of the given environment.
The SDK will implement a stack API abstraction that provides the same normalized interface for both configurations. For HTTP, a mapping from rpc name to HTTP request is generated by the build process which is used to generate function definitions. For gRPC the generated JS stubs are used.
The user will be able to use either the higher-level SDK API which provides simplified access to entities (like ttn.Applications.getById(id)) or call the underlying rpcs directly, resolved to HTTP or gRPC by the SDK.
Following our mono-repo idea, the SDK will be part of this repo and published isolated via npm, using the old (to be updated) SDK repo for issues.

What is already there? What do you see now?

Groundwork
(Rough) Draft of SDKs API / Architecture
Generating HTTP API interface from a protobuf generated JSON mapping

What is missing? What do you want to see?
Note: this is still incomplete / wip

What can you do yourself and what do you need help with?
After a initial PR for this by @kschiffer , we can see how work here can be split between @bafonins and @kschiffer . I think it's best to add functionality here as we need it in the console.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1087 by @kschiffer

Add utility for testing events publishing from stack packages

Summary:

In #989 I added a little utility for testing that the NS publishes expected events under certain circumstances. I later removed it due to multiple nasty import cycles (as the events package also test with the test package). This cycle can probably be solved by doing all testing in _test packages.

What is already there? What do you see now?

See pkg/networkserver/observability_util_test.go

What is missing? What do you want to see?

It would be nice if we could put the "expect events" in the test package.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1008 by @htdvisser

Demodulation Floor Validation

Summary:

The values in the demodulation floor table (in pkg/networkserver/adr.go) need to be verified.

What do you see now?

The current table is based on what we have in v2, which in turn is based on the ADR issue TheThingsArchive/ttn#265 (comment), and some other sources.

What do you want to see instead?

We're pretty confident that the current values are correct (especially the ones already there in v2) but to be sure, we should validate the ones for SF6 data rates (if those will ever be used), and the ones for BW250/BW500.

What can you do yourself and what do you need help with?

We based the table on the information that we have. It would be helpful if people had an authoritive source of these values.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/876 by @rvolosatovs

Consistent end device settings and defaults

Summary:

Help CLI and Console users a bit by filling boilerplate for devices.

Sane defaults, ideally version ids is sufficient
OTAA with option to generate root keys
ABP with option to generate a session (with session keys and DevAddr in NetID 0)

Why do we need this?

The CLI and Console would be very unfriendly if (for example) you'd had to generate keys with openssl and then copy-paste them. Better to let the CLI and Console generate those and other boilerplate.

Also, we need a consistent set of defaults in the CLI and Console to avoid user confusion.

What is already there? What do you see now?

The CLI sets some sane defaults; i.e. ADR on, 32-bit frame counters enabled, OTAA by default. See the up-to-date settings in cmd/ttn-lw-cli/commands/end_devices.go.

What is missing? What do you want to see?

The same defaults as CLI in the Console
Potentially the NS to generate a DevAddr in the DevAddr range (or NetID)

How do you propose to implement this?

Document the defaults
Implement this in CLI and Console, referring to the documentation

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1392 by @htdvisser

Observability with PProf, Trace and Prometheus

Summary:

Setting PProf labels and Trace regions on RPCs (perhaps in interceptor)

What is already there? What do you see now?

We have pprof, but we can't filter by RPC.
We do not have tracing.

What is missing? What do you want to see?

{
  defer pprof.SetGoroutineLabels(ctx)
  ctx = pprof.WithLabels(ctx, pprof.Labels("grpc.method", "ttn.lorawan.v3/XXX"))
  pprof.SetGoroutineLabels(ctx)
  // Execute RPC
}

{
  defer trace.StartRegion(ctx, "ttn.lorawan.v3/XXX").End()
}

Additionally, we could set these on functionality, sub-packages etc.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1072 by @htdvisser

Harmonize testing of RPC servers

Summary:

Harmonize testing of RPC servers.

Why do we need this?

To be consistent across the codebase.

What is already there? What do you see now?

Two classes of inconsistencies;

Direct invocation of RPC exposed methods vs creating a client to call RPC services
Overriding an RPC server by a test sink vs spawning mock cluster peers

What is missing? What do you want to see?

In general, I prefer a more "black box testing" approach; i.e. outside testing over inside hooks. This is still unit testing, but then the component from outside.

Regarding the two inconsistencies described above, I prefer;

Creating a client to call RPC servers, because this will include the interceptors for security, validation, etc. This may be or become functionally or non-functionally important to include in testing.
Spawning mock cluster peers, because this will include the asserting the outgoing context, again for security, and does not skip client RPC code that is relevant for testing

How do you propose to implement this?

The occurrences of 1 and 2 are only in the NS and JS. @rvolosatovs if you agree with this approach, please list the tests that are relevant, and work with @M-Gregoire or yourself to harmonize this.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1361 by @johanstokking

Use skeleton screens instead of loading spinners where applicable

Summary:
Spinners often result in suboptimal UX, because they leave the user in the dark about what to anticipate from a loading event (among other things, see e.g. here for more info)
It's often times better to provide a skeleton of the component/content/view that is being loaded, possibly in conjunction with a spinner or similar animation that indicates loading.

What is already there? What do you see now?

A loading spinner

What is missing? What do you want to see?

Skeleton graphic/markup for components that load frequently
A flag on these components that will result in the skeleton being rendered (for loading events)

cc @bafonins

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1223 by @kschiffer

Use perl instead of sed

Summary:

Use perl instead of sed

Why do we need this?

Predictable behavior across platforms (i.e. -i flag)
More powerful regex
perl is installed by default on Mac and Unix-like operating systems. (https://www.perl.org/get.html)

What is already there? What do you see now?

sed invocations in scripts and .make

What is missing? What do you want to see?

sed invocations replaced by perl

How do you propose to implement this?

Replace the sed invocations by perl

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/224 by @rvolosatovs

Add CSS styles to HTML emails

Summary:

Add CSS styles to HTML emails

Why do we need this?

What is already there? What do you see now?

Plaintext

What is missing? What do you want to see?

Inline css for email markup.

How do you propose to implement this?

Inline the CSS from https://github.com/TheThingsIndustries/account-server/blob/master/src/email/email.css and test.

What can you do yourself and what do you need help with?

I can create a PR

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/778 by @gomezjdaniel

Console support for event streams

Summary:

The console should support displaying real-time event streams.

All entities trigger events when they are created/updated/deleted:

Applications
Clients
EndDevices
Gateways
Organizations
Users
API Keys and Collaborations of above entities

LoRaWAN traffic also triggers messages while being processed by the stack. This is specifically useful for debugging EndDevices.

Why do we need this?

The purpose of event streams is mostly to let users trace messages through the stack, typically for debugging. Users will go to a single entity page and navigate to the events section. The console will then display some historical events if available (history is not stored by default) and add events as they come in. Users will be able to inspect event details. Users will be able to filter events by event name, and some other fields.

What is already there? What do you see now?

We have the Events API (see api doc). The Events.Stream RPC is mapped to HTTP on a POST /api/v3/events (note the POST). The request contains the identifiers of entities (plural) that the caller is interested in. All event names are selected, and it is not (yet) possible to filter event names server-side. Matching events are streamed as newline-delimited JSON objects in the response.

NOTE: In the future we may add a websocket version of this endpoint, but for now we're only working with this.

We also have the ttn-lw-cli events command that subscribes to the events stream (over gRPC).

The Event message contains the following fields:

name (string) of the event. An example is application.api-key.create. This name has a translation with key event:application.api-key.create.
time (timestamp) of the event.
identifiers one or more entity identifiers of entities related to the event (for example, an OAuth authorize event involves a user and an OAuth client)
data (optional object) details of the event, can be different for each event type
correlation_ids (list of strings) used for correlating this event to other events. In the future we may define an API for getting historical events by correlation ID, but that's currently not possible yet.
origin (string) origin of the event (typically the hostname of the instance/container), so may not say much.

What is missing? What do you want to see?

SDK support for setting up event streams
- Note: Although a typical page focuses on a single entity, the API supports subscribing to multiple entities (for example, a gateway and an application), which could be very powerful.
A re-usable component for displaying event streams
- Keep some history on the client side
- Filtering by name (after events are received)
- Filtering by identifiers (after events are received)
- Filtering by correlation_ids (after events are received)
Section in the applications page to display messages related to the current application, see https://github.com/TheThingsIndustries/lorawan-stack/issues/1065
Section in the gateways section to display messages related to the current gateway, see https://github.com/TheThingsIndustries/lorawan-stack/issues/1068
Section in the end devices section to display messages related to the current end device, see https://github.com/TheThingsIndustries/lorawan-stack/issues/1067

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1071 by @bafonins

Console support for devices

Summary:
The console should provide functionality to manage devices per application (see https://github.com/TheThingsIndustries/lorawan-stack/issues/1065), namely:

listing devices
- list all devices (paginated) per application
- search devices by ids per application
add devices
deleting devices
updating devices
adding locations for devices

What is already there? What do you see now?
Wireframes for the devices page

What is missing? What do you want to see?

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1067 by @bafonins

Support Private Network Connection on The Things Gateway

Summary:

In ttnv2, the configuration sent by the account server to The Things Gateway cannot be configured to support private networks with custom Router IPs.

References #68

Why do we need this?

This feature would add value to The Things Gateway making it a useful tool for new customers to evaluate our V3 service.

What is already there? What do you see now?

The gateway is always hard-coded to PCN routers.

What is missing? What do you want to see?

The gateway should connect to the gatewayserveraddress from the gateway settings.

How do you propose to implement this?

The Gateway connects to the hard-coded https://account.thethingsnetwork.org to fetch configuration.
The IS sends the frequency plans as well including the gateway-server address configured in its settings.
The Gateway now connects to the gatewayserver-IP:mqtt-port to send uplinks/receive downlinks.

What can you do yourself and what do you need help with?

I can create a PR.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/770 by @KrishnaIyer

Retry failed/timed-out tasks

Summary:

Failed tasks(like the ones added in #1372 ), should be retried on failure.

Why do we need this?

To not drop tasks

What is already there? What do you see now?

Successfully completed tasks are acked

What is missing? What do you want to see?

Unacked tasks, which are pending for too long should be retried

How do you propose to implement this?

https://redis.io/commands/xpending and https://redis.io/commands/xclaim
We want a generic helper function, in pkg/redis, which would use those to claim the timed-out tasks from other consumers.

What would be great, is ability to contact the consumers, which have the tasks pending and verify that they're down.

Environment:

Anything

What can you do yourself and what do you need help with?

I can implement it

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1381 by @rvolosatovs

Adapt gateway registration process to reflect new workflow

Summary:

In v3, we will change some important details in the logic of gateway information. To reflect those changes, it will be better if we change the gateway registration experience.

Gateway EUI and legacy protocols
Each gateway can have, on a very high logic level, two identifiers. It has a gateway ID (e.g. laurens-rockstart-roof), to identify it in TTN's database. It can also have a gateway EUI, made of 8 bytes (e.g. 0000024B08030C5F in hex) - indeed, in the legacy non-TTN protocol, this was how gateways identified themselves. Gateway EUIs are usually attributed by manufacturers, but since there is no way to verify it, we have always accepted entering arbitrary EUIs as long as they were unique.

In v2, IDs and EUIs were intertwined: if a user wanted to use legacy protocols with their gateway, the ID was generated from the ID. For example, laurens-rockstart-roof was a valid ID for gateways exchanging with modern protocols - but for gateways using legacy protocol, the ID was generated from the EUI, in a eui-<hex EUI> pattern (e.g. eui-0000024b08030c5f). To understand this better, you can try to register gateways on the current console.

Why do we need this?
In v3, IDs will no longer be tied to the EUI - the user can use any possible ID for any gateway. However, when registering a gateway, the user also has the possibility of specifying an EUI that can identify this gateway: see the API. This EUI cannot be changed or removed after registration, nor can an EUI be added after registration is complete. This change of modus operandi should be clear to the user.

Furthermore, if a gateway owner wants to add an EUI, this means that they might use legacy protocols. In that case, they should be offered additional gateway options specific to legacy protocols. The first one that comes to mind is the TX delay option, about which you can find more details here and here: https://github.com/TheThingsIndustries/ttn/issues/378.

What is already there? What do you see now?

The registration process of v2 console

What is missing? What do you want to see?

Gateway registration flow in the v3 console
- UX strategy
- Designs

How do you propose to implement this?

Regarding the EUIs: all gateways now have an ID and optionally an EUI. We should not imply that when entering an EUI, that people are using legacy protocols – these things are not related. Many manufacturers simply think in terms of EUIs, serial numbers, MAC addresses, etc. So the flow that we have in V2 (either ID or EUI, and EUI is legacy) changes: it will be ID and optionally an EUI, without any reference to protocols because it's irrelevant.

What can you do yourself and what do you need help with?
Needs to be worked out by @kschiffer @bafonins

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/579 by @egourlao

Clean up component registries when applications got deleted

Summary:

When an application is deleted from IS, the NS and AS registries should delete the application and associated devices.

Steps to Reproduce:

User deletes application
AS deletes all devices and deletes application

What do you see now?

Application gets deleted but devices stay there.

What do you want to see instead?

Devices be gone.

How do you propose to implement this?

An asynchronous garbage collection process in the NS and AS that deletes stale applications and its devices. This does not require an event stream, or even worse a connection, from IS to NS/AS, but rather have them be a client and check periodically if the applications are still there. This means that, as long as the garbage process has not run, unique device identifiers (i.e. JoinEUI, DevEUI) remain in the database, making it impossible to register a new device with the same identifiers.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/740 by @johanstokking

Database Backup/Restore

Summary:

We should be able to backup/restore the databases/stores of a network (or a single cluster).

What do you see now?

We have multiple databases (IS has SQL, NS+AS+JS have Redis), but the only way to export data (for backup or transfer to a different store, or cloud registry) would be through the APIs. However, those APIs don't return private fields, so any such data exports may be incomplete.

What do you want to see instead?

I would like to see a program or command that is able to export the entire network, including all stored data in a format that can later be imported into a new installation. This would be executed by the network operator for backup or in order to transfer the data to a (cloud) data store.

At the same time I'd like to see a program that can, based on such an export, restore the data to a clean installation.

This program needs to work on complete networks (including IS/ER), but also on independent clusters (with only NS/AS/JS).

How do you propose to implement this?

I think we can assume that the network operator runs these backups/restores, and has direct access to te SQL/Redis databases. Therefore, the command could probably be implemented on top of the pkg/identityserver/store interfaces, pkg/{networkserver,applicationserver,joinserver}.DeviceRegistry and pkg/applicationserver.LinkRegistry. The export format would either be protobuf or jsonpb.

Working with those interfaces allows us to build common import/export logic, while the implementation of those interfaces can talk to our usual databases/stores (SQL,Redis), but also to custom implementations (AWS IoT).

What can you do yourself and what do you need help with?

I could probably get started with this by building the "common" part on top of the interfaces, but as the Redis registries don't support ranging over all devices, we may have to add that. Perhaps @rvolosatovs can assist with that.

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/304 by @htdvisser

	// TODO: Extract this into a utility function shared with mac.HandleRejoinRequest. (https://github.com/TheThingsNetwork/lorawan-stack/issues/8)
	downAt := up.ReceivedAt.Add(-infrastructureDelay/2 + phy.JoinAcceptDelay1 - macState.DesiredParameters.Rx1Delay.Duration()/2 - nsScheduleWindow())
	if earliestAt := time.Now().Add(nsScheduleWindow()); downAt.Before(earliestAt) {
	downAt = earliestAt
	}
	logger.WithField("start_at", downAt).Debug("Add downlink task")
	if err := ns.downlinkTasks.Add(ctx, stored.EndDeviceIdentifiers, downAt, true); err != nil {
	logger.WithError(err).Error("Failed to add downlink task after join-request")
	}

thethingsnetwork / lorawan-stack Goto Github PK

lorawan-stack's People

Contributors

Stargazers

Watchers

Forkers

lorawan-stack's Issues

Recommend Projects

Recommend Topics

Recommend Org