dreem-organization / dosed Goto Github PK

I see that circleci.yml does a bunch of pip install while these should be in the requirements.txt

Hi, I’m very interested in the paper and the project you provided. I followed the code and method to detect sleep spindle on the database of MASS SS2 and tried to reproduce the reported results, making sure every detail is the same as instructed in the paper, but the result of f1 is only between 0.5-0.6 in test dataset, not 0.7-0.8 as reported.
I don’t know what’s wrong and why this happened. Hoping your reply, thanks very much!

When the dataset contains more than one event, running the training step first raises a VisibleDeprecationWarning and then a TypeError from Python:

• ...\Python\Python38\lib\site-packages\numpy\core_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray => return array(a, dtype, copy=False, order=order, subok=True)
• ...\Python\Python38\lib\site-packages\numpy\core\fromnumeric.py:87: TypeError: can only concatenate list (not "int") to list.

This occurs through a bug in the file dosed/dosed/functions/compute_metrics_dataset.py where only the metrics for the last event in the event list are averaged, whereas the metrics for the other events stay unchanged.

Original code (lines 62 to 70):

# If for any event and record the network predicted events, return -1
if found_some_events is False:
    return -1

for metric in metrics.keys():
    metrics_test[event_num][metric] = np.nanmean(
        np.array(metrics_test[event_num][metric]))

return metrics_test

The average should be calculated for each of the events instead:

# If for any event and record the network predicted events, return -1
if found_some_events is False:
    return -1

for event_num in range(network.number_of_classes - 1):
    for metric in metrics.keys():
        metrics_test[event_num][metric] = np.nanmean(np.array(metrics_test[event_num][metric]))

return metrics_test

for example in dosed1.py it should be:

from collections import OrderedDict

import torch
import torch.nn as nn

from ..functions import Detection
from ..base import BaseNet

also you will see that import order should be consistent. First imports from standard lib, then external deps then local imports

This is a suggestion for further improvement of the file dosed/functions/compute_metrics_dataset.py. When working with the framework on detecting two different types of events, it was necessary to adapt the function in order to achieve a good detection.

The current implementation averages the metrics for each type of event. If there are five records with ten events of each event type and the network detects all ten events in one of the records and none of the events in the other nine records, the metrics are still good. This is because a record with no predicted events results in an empty list, which is not added to metrics_test[event_num][metric] and therefore not included when averaging the result over all ten records.

for event_num in range(network.number_of_classes - 1):
    for metric in metrics.keys():
        metrics_test[event_num][metric] = np.nanmean(np.array(metrics_test[event_num][metric]))

return metrics_test

I suggest to adapt the function as follows. This will weight the metrics with the proportion of records where events of this type are found. With this adaption, the network learns to find the most possible number of events in each record and not only to maximize the results in one record. Further this suggestion weights the prediction of no events of one event type with -1 to achieve a more balanced prediction of the two event types.

for event_num in range(network.number_of_classes - 1):
    for metric in metrics.keys():
        num_records_with_found_events = len(metrics_test[event_num][metric])
        num_records = len(test_dataset.records)
        if (num_records_with_found_events > 0):
            metrics_test[event_num][metric] = np.nanmean(np.array(metrics_test[event_num][metric])) * (num_records_with_found_events / num_records)
        else:
            metrics_test[event_num][metric] = -1

Maybe this suggestion could improve the prediction process.

when trying to download data I get:

$ python minimum_example/download_data.py
Traceback (most recent call last):
  File "minimum_example/download_data.py", line 22, in <module>
    download_dir(client, resource, MINIMUM_EXAMPLE_SETTINGS["download_directory"])
  File "minimum_example/download_data.py", line 9, in download_dir
    for result in paginator.paginate(Bucket=bucket):
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/signers.py", line 157, in sign
    auth.add_auth(request)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/auth.py", line 425, in add_auth
    super(S3SigV4Auth, self).add_auth(request)
  File "/Users/alex/anaconda/envs/mne/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

use "%matplotlib inline" in notebooks so plots are visible in the rendered notebooks such as download_and_data_format_explanation.ipynb