iacobo / continual

Continual Learning of Electronic Health Records (EHR).

Hi, I'm Jacob, a Data Scientist at Oxford University / the UK Health Security Agency.

My background is in maths, but my work over the past few years has focused on infectious disease epidemiology, most recently the COVID-19 pandemic. I am currently a DPhil candidate at Oxford University.

continual's Issues

Correct code for ROC AUC and AUPRC

Cannot average metrics over minibatches as is done for other metrics, since they depend on threshold. Need to calculate over all. Check e.g. MeanScore for inspiration on metric definition.

Add early stopping to avoid over-large number of epochs for diff models

MLP / LSTM take shorter time to train than CNN / Transformer. Add early stopping to avoid overtraining, saturating.

Change strategy to base strategy inheriting from strat and earlystopping plugin.

Fix transformer

• include positional encoding
• correct n_heads fix to pad input instead (line 236)

RNN's fail using EWC on GPU

See here:

ContinualAI/avalanche#736

Ray Tune warnings

Ray Tune produces the following warnings:

INFO registry.py:66 -- Detected unknown callable for trainable. Converting to class.
WARNING experiment.py:295 -- No name detected on trainable. Using DEFAULT.

Non-fatal, but it's annoying to have these messages bloating the console output.

Add open access/data/code badges

Add Protected Access (data) and open access (code) badges from:
https://github.com/saa-osig/badges-for-open-practices

FIDDLE eICU has no train/test/val col

Change experience to class balanced replay

Have manually edited the replay definition for now. Will need to update avalanche and do change based on training.storage_policy.

May also need to change memory buffer to n_tasks * buffer (since GEM etc use this number for experience-wise buffer sizes).

Add Naive with no regularization?

Maybe add naive with no regularization? I.e. no dropout etc, to enable clearer ablation testing of naive fine tuning and inherent regularization mechanisms vs explicit CL strategy.

Need to add code for further experiments

plotting.plot_demographics()

# Secondary experiments:
########################
# Sensitivity to sequence length (4hr vs 12hr)
# Sensitivity to replay size Naive -> replay -> Cumulative
# Sensitivity to hyperparams of reg methods (Tune hyperparams over increasing number of tasks?)
# Sensitivity to number of variables (full vs Vitals only e.g.)
# Sensitivity to size of domains - e.g. white ethnicity much larger than all other groups, affect of order of sequence

No age 18-... column in ARF or Shock features tensors.

Need to amend code to ignore missing cols for demographic splits.

FIDDLE MIMIC outcome df has no 'stay' col for ARF, Shock

This is necessary to compute th subject ID's to recover the time element.

Providing the .json files in `config`?

Hi. Thanks for the interesting work and for sharing these codes. But since there are no json files in /config in the current repo, calling python3 main.py --train cannot work directly. I would like to suggest providing a link to download the json files of the configurations that you have tuned and included in the paper. That would be very helpful for us to reproduce the results without re-tuning :) Thanks in advance.

CNN fails with kernel_size 5 or 7

Getting the following error (on GPU) with CNN runs with kernel_size in [5,7]:

RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

https://stackoverflow.com/questions/66600362/runtimeerror-cuda-error-cublas-status-execution-failed-when-calling-cublassge?answertab=votes#tab-top

Freeze model hyperparams and tune only CL ones after training Naive?

Would it make more sense to do the above, to enable more fair comparison of methods?

Remove task labels for static variables for given experiment