Comments (2)
This is almost certainly better dealt with by partitioning and/or clustering, since it's currently all wired up already as receipt of raw data (there's nothing inherently special compared to any other type of data, if we were to accept this pattern, we should simply do this for every different kind, which we avoid in order to allow the data to evolve)
from data-gateway.
@thclark Allow me to partially disagree. This data (connection stats and battery info) has entirely different purpose from the rest (measurements of direct interest), and thus different use and hence different lifecycle. Just a quick examples:
- Almost no-one from outside of aerosense would be interested in this data.
- This data is never queried together /compared to other data
- Primary possible use is in system supervisory monitoring (i.e.) sending it to the dashboard (so #of requests is higher than for other "sensors") and maybe debugging/system development (?)
From another side, I see your point as the total amount of this type of data per day is very little:
10 bytes per sample * 22.(2) samples per second * 86400 seconds in day ~ 20 Mb (60Mb if we have 3 nodes)
And in terms of dashboard development I imagine it will rely more and more on data from post-processed tables rather than the raw data table.. so it might be worth just splitting it from the rest at post-process stage.
Lets put it as "won't fix" for now and close it later?
from data-gateway.
Related Issues (20)
- Create a separate table/tables with sensor coordinates information. HOT 9
- Add sessions table HOT 4
- Poetry installation prevents tests from running in CI HOT 1
- Add CLI or gateway configuration option to control maximum local storage limit.
- Update sensor conversion constants in configuration HOT 1
- Cloud function fails to add sensor data. HOT 3
- 5 word coolnames are not cool HOT 1
- Add a --reboot-nodes flag to the CLI (Allow daemonisation)
- problem in parsing the config file on fire-fly HOT 3
- Main service does not log to the Balena Dashboard Logs Window. HOT 1
- Reduce BigQuery costs HOT 6
- Implement WaitUntilSetComplete and timestamp synchronisation HOT 2
- log handling not idempotent across tests
- Cannot persist cloud data HOT 1
- Allow configurations of more than 8KiB HOT 1
- Parallelise cloud function deployment workflow
- Allow cloud functions to be deployed manually
- Remove `sensor_type_reference` field from sensor data materialised views
- Conflicting routine file warning message
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from data-gateway.