aerosense-ai / data-gateway Goto Github PK

[2021-12-06 18:04:30,224 | INFO | data_gateway.packet_reader] 0.000000 bytes waiting
[2021-12-06 18:04:30,241 | INFO | data_gateway.packet_reader] 769.000000 bytes waiting
[2021-12-06 18:04:30,242 | INFO | data_gateway.packet_reader] 522.000000 bytes waiting
[2021-12-06 18:04:30,243 | INFO | data_gateway.packet_reader] 275.000000 bytes waiting
[2021-12-06 18:04:30,244 | INFO | data_gateway.packet_reader] 28.000000 bytes waiting
[2021-12-06 18:04:30,265 | INFO | data_gateway.packet_reader] 0.000000 bytes waiting
[2021-12-06 18:04:34,236 | INFO | data_gateway.persistence] Window 0 written to disk.
[2021-12-06 18:04:34,236 | INFO | data_gateway.persistence] Saving Baros_P data to csv file.
[2021-12-06 18:04:34,494 | INFO | data_gateway.persistence] Saving Baros_T data to csv file.
[2021-12-06 18:04:34,664 | INFO | data_gateway.persistence] Saving Constat data to csv file.
[2021-12-06 18:04:34,721 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,722 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,722 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,723 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,723 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,724 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,725 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,725 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,726 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,726 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,727 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,727 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,728 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,728 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,728 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,729 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,729 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,730 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,730 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,731 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,731 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,732 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,732 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,733 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,733 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,733 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,734 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,734 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,735 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,735 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,736 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,736 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,737 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,737 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,737 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,738 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,738 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,739 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,739 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,740 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,740 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,741 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,741 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,742 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,742 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,742 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,743 | INFO | data_gateway.packet_reader] 4095.000000 bytes waiting
[2021-12-06 18:04:34,743 | INFO | data_gateway.packet_reader] 3952.000000 bytes waiting
[2021-12-06 18:04:34,744 | INFO | data_gateway.packet_reader] 3705.000000 bytes waiting
[2021-12-06 18:04:34,744 | INFO | data_gateway.packet_reader] 3458.000000 bytes waiting
[2021-12-06 18:04:34,745 | INFO | data_gateway.packet_reader] 3211.000000 bytes waiting
[2021-12-06 18:04:34,745 | INFO | data_gateway.packet_reader] 2964.000000 bytes waiting
[2021-12-06 18:04:34,746 | INFO | data_gateway.packet_reader] 2717.000000 bytes waiting
[2021-12-06 18:04:34,746 | INFO | data_gateway.packet_reader] 2470.000000 bytes waiting
[2021-12-06 18:04:34,746 | INFO | data_gateway.packet_reader] 2223.000000 bytes waiting
[2021-12-06 18:04:34,747 | INFO | data_gateway.packet_reader] 1976.000000 bytes waiting
[2021-12-06 18:04:34,747 | INFO | data_gateway.packet_reader] 1729.000000 bytes waiting
[2021-12-06 18:04:34,748 | INFO | data_gateway.packet_reader] 1482.000000 bytes waiting
[2021-12-06 18:04:34,748 | INFO | data_gateway.packet_reader] 1235.000000 bytes waiting
[2021-12-06 18:04:34,749 | INFO | data_gateway.packet_reader] 988.000000 bytes waiting
[2021-12-06 18:04:34,749 | INFO | data_gateway.packet_reader] 741.000000 bytes waiting
[2021-12-06 18:04:34,749 | INFO | data_gateway.packet_reader] 494.000000 bytes waiting
[2021-12-06 18:04:34,750 | INFO | data_gateway.packet_reader] 247.000000 bytes waiting
[2021-12-06 18:04:34,750 | INFO | data_gateway.packet_reader] 1023.000000 bytes waiting
[2021-12-06 18:04:34,751 | INFO | data_gateway.packet_reader] 776.000000 bytes waiting
[2021-12-06 18:04:34,751 | INFO | data_gateway.packet_reader] 529.000000 bytes waiting
[2021-12-06 18:04:34,752 | INFO | data_gateway.packet_reader] 282.000000 bytes waiting
[2021-12-06 18:04:34,752 | INFO | data_gateway.packet_reader] 35.000000 bytes waiting
[2021-12-06 18:04:34,766 | INFO | data_gateway.packet_reader] 7.000000 bytes waiting
[2021-12-06 18:04:34,766 | WARNING | data_gateway.packet_reader] Lost Baros_P packet(s): -57952858.82019043 ms gap
[2021-12-06 18:04:34,766 | WARNING | data_gateway.packet_reader] Lost Baros_T packet(s): -57952858.82019043 ms gap
[2021-12-06 18:04:34,811 | INFO | data_gateway.packet_reader] 741.000000 bytes waiting
[2021-12-06 18:04:34,811 | WARNING | data_gateway.packet_reader] Lost Baros_P packet(s): 57949038.85437011 ms gap
[2021-12-06 18:04:34,811 | WARNING | data_gateway.packet_reader] Lost Baros_T packet(s): 57949038.85437011 ms gap
[2021-12-06 18:04:34,812 | INFO | data_gateway.packet_reader] 494.000000 bytes waiting
[2021-12-06 18:04:34,812 | INFO | data_gateway.packet_reader] 247.000000 bytes waiting
[2021-12-06 18:04:34,813 | INFO | data_gateway.packet_reader] 0.000000 bytes waiting

[2021-12-06 15:38:51,200 | ERROR | data_gateway.persistence] Object of type Configuration is not JSON serializable
Traceback (most recent call last):
  File "/home/pi/aerosense/install/data-gateway/data_gateway/persistence.py", line 264, in _persist_window
    timeout=self.upload_timeout,
  File "/home/pi/flasks/gateway/lib/python3.7/site-packages/octue/cloud/storage/client.py", line 101, in upload_from_string
    self._overwrite_blob_custom_metadata(blob, metadata)
  File "/home/pi/flasks/gateway/lib/python3.7/site-packages/octue/cloud/storage/client.py", line 298, in _overwrite_blob_custom_metadata
    blob.metadata = self._encode_metadata(metadata)
  File "/home/pi/flasks/gateway/lib/python3.7/site-packages/octue/cloud/storage/client.py", line 320, in _encode_metadata
    return {key: json.dumps(value, cls=OctueJSONEncoder) for key, value in metadata.items()}
  File "/home/pi/flasks/gateway/lib/python3.7/site-packages/octue/cloud/storage/client.py", line 320, in <dictcomp>
    return {key: json.dumps(value, cls=OctueJSONEncoder) for key, value in metadata.items()}
  File "/usr/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/pi/flasks/gateway/lib/python3.7/site-packages/octue/utils/encoders.py", line 33, in default
    return TwinedEncoder.default(self, obj)
  File "/home/pi/flasks/gateway/lib/python3.7/site-packages/twined/utils/encoders.py", line 32, in default
    return json.JSONEncoder.default(self, obj)
  File "/usr/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Configuration is not JSON serializable
[2021-12-06 15:38:51,206 | WARNING | data_gateway.persistence] Upload of window may have failed - writing to disk at 'data_gateway/.backup/163946/window-0.json'

/usr/local/bin/python ~/.vscode-server/extensions/ms-python.python-2022.8.1/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear
cwd: .
[ERROR 2022-6-6 9:43:24.553]: Error discovering pytest tests:
 [n [Error]: --- Logging error ---
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1086, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/usr/local/lib/python3.9/multiprocessing/util.py", line 332, in _exit_function
    info('process shutting down')
  File "/usr/local/lib/python3.9/multiprocessing/util.py", line 54, in info
    _logger.log(INFO, msg, *args)
Message: 'process shutting down'
Arguments: ()

poetry install

poetry export -f requirements.txt --output requirements.txt --dev --without-hashes && pip install -r requirements.txt`

pip install <data_gateway_url>

#!/bin/bash
temp=`head -n 1 /sys/class/thermal/thermal_zone0/temp | xargs -I{} awk "BEGIN {printf \"%.2f\n\", {}/1000}"`
echo $((`cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq`/1000)) MHz, $temp degrees

while : ; do date; sh cpu_freq.sh; sleep 600; done

       1.  read bytes from the port (one by one) until packet is formed (tell it how many bytes to read)
       2. parse the packet
       3. accumulate the parsed data in memory
       4. if time more than 10 minutes (window full):
             -  upload / persist parsed data to disc and empty memory
       5. If the time taken to execute steps 2-4 is greater than the time taken for the buffer to fill, given the bandwidth of the input data, then our overflow error occurs, **which manifests in missed packets**. Question: Will it result in just missed packets? or corrupt ones? Almost certain that the last packet is corrupt.
             - so whether this condition occurs is a function of a) the speed of the CPU / efficiency of code in steps 1-4, b) the bandwidth of influx data and c) size of the buffer
       6. GOTO 1

{
  "commands": [
    ["startSync", 0],
    ["selMsgDest 3", 0.1],
    ["getBattery", 0.2],
    ["startBaros", 0.3],
    ["selMsgDest 4", 1.1],
    ["getBattery", 1.2],
    ["startBaros", 1.3],
    ["selMsgDest 5", 2.1],
    ["getBattery", 2.2],
    ["startBaros", 2.3],
    ["selMsgDest 3", 60.1],
    ["stopBaros", 60.2],
    ["selMsgDest 4", 61.1],
    ["stopBaros", 61.2],
    ["selMsgDest 5", 62.1],
    ["stopBaros", 62.2],
    ["stopSync", 65],
    ["enterHighSpeed", 65.1],
    ["selDevice 3", 70], // (wait until the connection parameter is updated, you can tell that with the constatpacket number)
    ["readSens", 75], // allow 1 minute download per minute collection
    ["selDevice 4", 135], // (wait until the connection parameter is updated, you can tell that with the constatpacket number)
    ["readSens", 140],
    ["selDevice 5", 200], // (wait until the connection parameter is updated, you can tell that with the constatpacket number)
    ["readSens", 205],
    ["exitHighSpeed", 265],
    ["selMsgDest 3", 270.0], // (now we reconnect all the nodes and put them into sleep)
    ["sleep", 270.1],
    ["selMsgDest 4", 271.0],
    ["sleep", 271.1],
    ["selMsgDest 5", 272.0],
    ["sleep", 272.1],
    ["selMsgDest 3", 330.0], // (wake up nodes and allow them to re-establish connection parameters)
    ["wakeUp", 330.1],
    ["selMsgDest 4", 331.0],
    ["wakeUp", 331.1],
    ["selMsgDest 5", 332.0],
    ["wakeUp", 332.1]
  ],
  "period": 360
}

 $ gateway start
/home/pi/.cache/pypoetry/virtualenvs/data-gateway-lz7aa1yn-py3.7/lib/python3.7/site-packages/google_crc32c/__init__.py:29: RuntimeWarning: As the c extension couldn't be imported, `google-crc32c` is using a pure python implementation that is significantly slower. If possible, please configure a c build environment and compile the extension
  warnings.warn(_SLOW_CRC32C_WARNING, RuntimeWarning)
[2022-06-09 14:29:20,591 | INFO | multiprocessing | MainProcess] No configuration file provided - using default configuration.
[2022-06-09 14:29:20,693 | WARNING | multiprocessing | MainProcess] No routine was provided and interactive mode is off - no commands will be sent to the sensors in this session.
[2022-06-09 14:29:20,705 | INFO | multiprocessing | MainProcess] allocating a new mmap of length 4096
[2022-06-09 14:29:20,718 | INFO | multiprocessing | Reader] child process calling self.run()
[2022-06-09 14:29:20,719 | INFO | multiprocessing | Reader] Packet reader process started.
[2022-06-09 14:29:20,725 | INFO | multiprocessing | Parser] child process calling self.run()
[2022-06-09 14:29:20,726 | INFO | multiprocessing | Parser] Packet parser process started.
[2022-06-09 14:29:20,751 | INFO | multiprocessing | Parser] Windows will be saved to 'data_gateway/661014/.backup' at intervals of 600.0 seconds.
[2022-06-09 14:29:20,751 | INFO | multiprocessing | Parser] Windows will be uploaded to 'data_gateway/661014' at intervals of 600.0 seconds.
^C[2022-06-09 14:29:53,966 | INFO | multiprocessing | Reader] Stopping gateway.
[2022-06-09 14:29:53,967 | INFO | multiprocessing | Reader] process shutting down
[2022-06-09 14:29:53,968 | INFO | multiprocessing | Reader] process exiting with exitcode 0
[2022-06-09 14:29:53,967 | INFO | multiprocessing | MainProcess] Sent 'stopBaros' command to sensors.
[2022-06-09 14:29:53,967 | ERROR | multiprocessing | Parser] expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/home/pi/aerosense/install/data-gateway/data_gateway/packet_reader.py", line 169, in parse_packets
    packet_type, packet = packet_queue.get(timeout=timeout).values()
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 920, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.7/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/aerosense/install/data-gateway/data_gateway/persistence.py", line 255, in _persist_window
    cloud_path=storage.path.generate_gs_path(self.bucket_name, self._generate_window_path()),
  File "/home/pi/.cache/pypoetry/virtualenvs/data-gateway-lz7aa1yn-py3.7/lib/python3.7/site-packages/octue/cloud/storage/path.py", line 63, in generate_gs_path
    return CLOUD_STORAGE_PROTOCOL + join(bucket_name, paths[0].lstrip("/"), *paths[1:])
  File "/home/pi/.cache/pypoetry/virtualenvs/data-gateway-lz7aa1yn-py3.7/lib/python3.7/site-packages/octue/cloud/storage/path.py", line 45, in join
    path = os.path.normpath(os.path.join(*paths)).replace("\\", "/")
  File "/usr/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
[2022-06-09 14:29:54,005 | WARNING | multiprocessing | Parser] Upload of window may have failed - writing to disk at 'data_gateway/661014/.backup/window-0.json'.
[2022-06-09 14:29:54,009 | INFO | multiprocessing | Parser] Window 0 written to disk.
[2022-06-09 14:29:54,011 | INFO | multiprocessing | Parser] Stopping gateway.
[2022-06-09 14:29:54,012 | INFO | multiprocessing | Parser] process shutting down
[2022-06-09 14:29:54,014 | INFO | multiprocessing | Parser] process exiting with exitcode 0
[2022-06-09 14:29:58,977 | INFO | multiprocessing | MainProcess] Sent 'stopDiffBaros' command to sensors.
[2022-06-09 14:30:03,983 | INFO | multiprocessing | MainProcess] Sent 'stopIMU' command to sensors.
[2022-06-09 14:30:08,989 | INFO | multiprocessing | MainProcess] Sent 'stopMics' command to sensors.

sudo apt update
sudo apt upgrade
cd ~
mkdir -p python-install && cd python-install
wget https://www.python.org/ftp/python/3.9.12/Python-3.9.12.tgz
tar -zxvf Python-3.9.12.tgz
cd Python-3.9.12
./configure --enable-optimizations
sudo make altinstall
cd /usr/bin
sudo rm python
sudo rm python3
sudo ln -s /usr/local/bin/python3.9 python
sudo ln -s /usr/local/bin/python3.9 python3
cd ~
python --version

libm.so.6: version `GLIBC_2.29' not found

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 171, in view_func
    function(data, context)
  File "/workspace/main.py", line 41, in upload_window
    window_handler.persist_window(unix_timestamped_window["sensor_data"], window_metadata)
  File "/workspace/window_handler.py", line 92, in persist_window
    self._store_microphone_data(
  File "/workspace/window_handler.py", line 134, in _store_microphone_data
    datafile = Datafile(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 144, in __init__
    self._use_cloud_metadata(**initialisation_parameters)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 495, in _use_cloud_metadata
    self.get_cloud_metadata()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 249, in get_cloud_metadata
    cloud_metadata = GoogleCloudStorageClient(self.project_name).get_metadata(cloud_path=self.cloud_path)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/cloud/storage/client.py", line 117, in get_metadata
    bucket = self.client.get_bucket(bucket_or_name=bucket_name)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/client.py", line 780, in get_bucket
    bucket.reload(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 1029, in reload
    super(Bucket, self).reload(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/_helpers.py", line 232, in reload
    api_response = client._get_resource(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/client.py", line 364, in _get_resource
    return self._connection.api_request(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/_http.py", line 80, in api_request
    return call()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http/__init__.py", line 480, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/data-gateway-processed-data?projection=noAcl&prettyPrint=false: [email protected] does not have storage.buckets.get access to the Google Cloud Storage bucket. 

{
  "textPayload": "Traceback (most recent call last):\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 2073, in wsgi_app\n    response = self.full_dispatch_request()\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1518, in full_dispatch_request\n    rv = self.handle_user_exception(e)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1516, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1502, in dispatch_request\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py\", line 171, in view_func\n    function(data, context)\n  File \"/workspace/main.py\", line 38, in clean_and_upload_window\n    window, window_metadata = window_handler.get_window()\n  File \"/workspace/window_handler.py\", line 73, in get_window\n    window_metadata = cloud_metadata[\"custom_metadata\"][\"data_gateway__configuration\"]\nKeyError: 'data_gateway__configuration'",
  "insertId": "000001-d74b9c37-a87c-4ad4-8879-37c0b2397de0",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "aerosense-twined",
      "function_name": "ingress-eu",
      "region": "europe-west6"
    }
  },
  "timestamp": "2021-12-11T12:54:03.594Z",
  "severity": "ERROR",
  "labels": {
    "execution_id": "22ajl43qdhwi"
  },
  "logName": "projects/aerosense-twined/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/aerosense-twined/traces/62cd5c882711eaae92ff91fb8fd7a643",
  "receiveTimestamp": "2021-12-11T12:54:04.565208817Z"
}

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 171, in view_func
    function(data, context)
  File "/workspace/main.py", line 38, in clean_and_upload_window
    window, window_metadata = window_handler.get_window()
  File "/workspace/window_handler.py", line 73, in get_window
    window_metadata = cloud_metadata["custom_metadata"]["data_gateway__configuration"]
KeyError: 'data_gateway__configuration'

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 171, in view_func
    function(data, context)
  File "/workspace/main.py", line 41, in upload_window
    window_handler.persist_window(unix_timestamped_window["sensor_data"], window_metadata)
  File "/workspace/window_handler.py", line 92, in persist_window
    self._store_microphone_data(
  File "/workspace/window_handler.py", line 141, in _store_microphone_data
    with datafile.open("w") as f:
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 688, in __enter__
    os.makedirs(os.path.split(self.datafile.local_path)[0], exist_ok=True)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 454, in local_path
    return self.download()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 440, in download
    raise e
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 433, in download
    GoogleCloudStorageClient(project_name=self.project_name).download_to_file(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/cloud/storage/client.py", line 173, in download_to_file
    blob.download_to_filename(local_path, timeout=timeout)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1281, in download_to_filename
    client.download_blob_to_file(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/client.py", line 1152, in download_blob_to_file
    _raise_from_invalid_response(exc)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 4466, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/download/storage/v1/b/data-gateway-processed-data/o/microphone%2F790791%2Fwindow-343.hdf5?alt=media: No such object: data-gateway-processed-data/microphone/790791/window-343.hdf5: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

 Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/client.py", line 1139, in download_blob_to_file
    blob_or_uri._do_download(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1001, in _do_download
    response = download.consume(transport, timeout=timeout)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/requests/download.py", line 214, in consume
    return _request_helpers.wait_and_retry(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/requests/_request_helpers.py", line 147, in wait_and_retry
    response = func()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/requests/download.py", line 207, in retriable_request
    self._process_response(result)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/_download.py", line 188, in _process_response
    _helpers.require_status_code(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/_helpers.py", line 105, in require_status_code
    raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>) 

	Constat packet: 2265
	Constat packet: 2266
	Constat packet: 2267
	Constat packet: 2268
	Lost set of Mics packets: 4345173.096801933 ms gap
	Lost set of Mics packets: -4317922.26965332 ms gap
	Lost set of Baros_P packets: 27359.23461917446 ms gap
	Lost set of Baros_T packets: 27359.23461917446 ms gap
	Lost Acc packet: 27708.526611328125 ms gap
	Lost Gyro packet: 27708.526611328125 ms gap
	Constat packet: 2297
	Lost set of Constat packets: 28075.860595725317 ms gap
	Constat packet: 2299
	Constat packet: 2300
	Lost Mag packet: 28500.274658203125 ms gap
	Constat packet: 2301
	Constat packet: 2302
	Constat packet: 2303
	Lost set of Analog Vbat packets: 23995.330810546875 ms gap
	Constat packet: 2304
	Constat packet: 2305
	Constat packet: 2306

  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 171, in view_func
    function(data, context)
  File "/workspace/main.py", line 41, in upload_window
    window_handler.persist_window(unix_timestamped_window["sensor_data"], window_metadata)
  File "/workspace/window_handler.py", line 99, in persist_window
    self.dataset.add_sensor_data(
  File "/workspace/big_query.py", line 80, in add_sensor_data
    errors = self.client.insert_rows(table=self.client.get_table(self.table_names["sensor_data"]), rows=rows)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 3411, in insert_rows
    return self.insert_rows_json(table, json_rows, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 3590, in insert_rows_json
    response = self._call_api(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 760, in _call_api
    return call()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http/__init__.py", line 480, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.GoogleAPICallError: 413 POST https://bigquery.googleapis.com/bigquery/v2/projects/aerosense-twined/datasets/greta/tables/sensor_data/insertAll?prettyPrint=false: <!DOCTYPE html>

[2021-12-06 14:13:58,234 | INFO | data_gateway.cli] Loaded configuration file from '/home/pi/data_gateway/configuration.json'.
[2021-12-06 14:13:58,235 | INFO | data_gateway.cli] Starting packet reader in non-interactive mode - files will be uploaded to cloud storage at intervals of 600.0 seconds.
[2021-12-06 14:13:58,236 | WARNING | data_gateway.packet_reader] Timestamp synchronization unavailable with current hardware; defaulting to using system clock.
[2021-12-06 14:13:58,977 | ERROR | data_gateway.packet_reader] Received packet with unknown type: 34
[2021-12-06 14:13:59,247 | ERROR | data_gateway.persistence] Object of type Configuration is not JSON serializable
Traceback (most recent call last):
  File "/home/pi/aerosense/install/data-gateway/data_gateway/packet_reader.py", line 132, in read_packets
    packet_type=packet_type, payload=payload, data=data, previous_timestamp=previous_timestamp
  File "/home/pi/aerosense/install/data-gateway/data_gateway/packet_reader.py", line 200, in _parse_payload
    raise exceptions.UnknownPacketTypeError("Received packet with unknown type: {}".format(packet_type))
data_gateway.exceptions.UnknownPacketTypeError: Received packet with unknown type: 34