tdrose / ionimage_embedding Goto Github PK

For the hyperparameter tuning, the following things should be considered/optimized:

• Each model type will be optimized independently (loss and architecture, data scenario)
• Hyperparameters for all models
  • Learning rate
  • Batch size?
  • dims
  • clip gradients
  • pertaining with AE model
  • ColocML preprocessing
  • final activation function
  • epochs
  • Weight decay
• Model/loss specific
  • Self contrast: initial, steps, ds_percentiles, knn
  • coloc contrast: initial, steps
  • Regression contrast: Maybe also a step function?
• Optimization metrics (different for random and transitivity):
  • Top accuracy will be the main metric (on latent training space)
• Baseline models:
  • Make a function to run UMAP, mean coloc and BioMedCLIP multiple interactions on a bunch test data

Next big issue Enforce dissimilarity across datasets for features being dissimilar within datasets:

• Compute dataset-specific similarity
• Take the lowest e.g. 20% (can also be a dynamic parameter like lower bound)
• Create complete Ion x Ion matrix
• Fill with pairs of lowest ions (int counts)
• Keep if ions they are lowest in a minimum 30% of datasets (requires additional counting of ion pairs (could be a with another matrix by just adding up all ions not only lowest 20%).
• Use these ions to in loss for A as forced zeros

Results

• Performance with increasing levels of transitivity (less overlap of datasets)
• Performance for different organs
• Run across all organs (evaluation performance per organ)
• Investigate variance
• Compare to LINEX?
• Highlight one case
• Same node similarity in loss function
• Weight less occurring node pairs inversely in the loss function.
• Investigate performance distribution on a node/edge/dataset level
  • easier/hard to predict metabolites/datasets (correlate to detection/co-detection levels)
  • Visualize the for individual metabolites (compare between UMAP/GNN/Mean) (particularly to see why MSE is so robust).
• Relative MSE with variance (idea from Jovan, think about it a bit more)
  • Compute square residuals for every prediction (and the number of times they occur in each datasest)
  • Additionally note the molecule identities to compute the variance

Writing

• Abstract
• Intro
• Results
• Methods
• Discussion
• Acknowledgements/Code availability/...

• 08.01. - 12.01.
  • Selection of evaluation datasets (more datasets per organ and different organs + look at annotation overlap stats)
  • Mean/median coloc UMAP (to compare against embedding of DL models, play with number of dimensions)
  • BioMedCLIP wrapper to use in the same evaluation framework
  • Check UMAP implementation (data leakage)
  • Check CRL evaluation code
  • Transitivity evaluation:
    • MSE for colocs that never occurred in the training data
    • Top-accuracy including only the pairs that did not occur in the training data
• 15.01. - 19.01.
  • Implement vision transformer model for CRL (It can deal with different image sizes (as long as they are dividable into patches, Further they have the number of classes as a parameter (returns linear layer output). Should be relatively easy to add similarly to ResNet architecture.) (Default, uses 1000 classes coming from a linear layer, just add another layer to resulting in desired dimension)
  • #9
  • #21
  • Check CVAE reconstruction & latent space for optimization
  • Make presentation for feedback session
• 22.01. - 26.01.
  • Graph Dataloader (top-N most colocalized per node, coloc quantile cut-off)
  • #22
  • Play with model
• 29.01. - 02.02.
  • Worked on GNN model performance/evaluation
• 05.02. - 09.02.
  • Model optimizations (fine-tuning, loss functions, architecture changes)
  • Hyperparameter tuning
  • Investigate Latent space
  • In which cases is transitivity performing badly?
  • More detailed info in #22
• 12.02. - 16.02.
  • Leave out ions training
  • Molecule embedding variance (quantify uncertainty)
  • Try euclidean distance
  • Prepare slides for Theo
  • Work on Theo feedback
• 19.02. - 23.02.
  • Evaluation on different scenarios
  • Finalize goal for poster (What do we want to show? Which features do we focus on?)
  • Make plots for poster
    • MSE (with BiomedCLIP & CLR
    • (Maybe ACC)
    • UMAP + cluster images
    • PCA
  • Draft poster
• 26.02. - 01.03.
  • Will not do much because of surgery
  • Send out Poster for feedback
• 04.03. - 08.03.
  • Incorporate Poster feedback
  • Print poster
• 12.03. - 15.03.
  • Conference
  • Structure feedback on poster

• Create evaluation datasets (selection of different organs + more datasets per organ)
• Fix RegContrast & colocContrast issue with rotating datasets
• Run optuna for different model scenarios
• Mean coloc UMAP representation (to compare latent embedding space of mean coloc embedding play with number of dimensions)
• Evaluate BioMedCLIP performance

Goal

Shared node embedding (ions are nodes) from coloc networks across multiple graphs (datasets).

Ideas

• (V)GAE model: discretizing edges (colocalized/not-colocalized)
• GNN learning latent space on a fully connected graph
  • Using coloc edge weights
  • Predict coloc from latent space directly (e.g. MSE loss function
• Shallow node embedding
  • Sample node sequence proportionally to coloc (probably requires softmax probabilities)

Notes

• GAE approaches will predict different node embeddings per network. Requires aggregation such as centroiding, but can be advantageous since it gives us a variance/variability for the embedding of each node

Links

• Node2Vec
• GAE example
• torch-geometric GAE
• torch-geometric InnerProductDecoder

As another model for included in the evaluation.

Potentially using embeddings as conditions, to integrate additional datasets later that were not part of training. I made a simple reference implementation of a scPoli like model: https://github.com/tdrose/lightning-cvae

As an additional model for the package. Use previous work from Dominik for the dataloaders.

Try to incorporate intensities instead of just thresholds.

For the evaluation of the project, we discussed a few scenarios from simple to more complex. Dataloaders (training, validation, and testing) for each scenario should be implemented.

• One cohort, randomly leave out molecules
• One cohort, leave out molecules, such that they never occur in one dataset
• One cohort, leave out complete datasets
• Diverse datasets, leave out random molecules
• Diverse datasets, leave out molecules, such that they never occur in one dataset
• Diverse datasets, leave out complete datasets

• Run without CAE
• Add FC layers + fc output
• Test different fc activation layers.
• set up optuna run
• ColocML preprocessing
• Tests for coloc framework and evaluations

Move the current torch implementation to lightning.

The is room for improvement in the CRL model, by adapting the model architecture and trying out different loss functions

What should definitely be explored:

• Double normalization in Loss function (from legacy model implementation
• Explore loss functions
• Use previous layers as embedding layers

Notes:

• InfoNCE (https://arxiv.org/pdf/1807.03748v2.pdf)
  • Works only if one positive pair per batch. Not really our case. I could change sampling to this but that will probably make this more inefficiently
• Overview of contrastive loss functions (https://lilianweng.github.io/posts/2021-05-31-contrastive/)
• When changing the final activation layer (I need to exponate the similarity afterwards: https://towardsdatascience.com/contrastive-loss-explaned-159f2d4a87ec)
  • Play with this and properly understand model loss behavior.
  • Cross entropy works with log(p) --> p is probability. The similarity matrix does not consist of probabilities.
  • But I could just sample from the batch to build the loss

1. We need a baseline model for the evaluation (e.g. just basic cosine-based colocalization).
2. Evaluation protocoll
   • Coloc and the model output operate in very different spaces, we need a comparable metric for their results
     • Check output distributions, and ranges of outputs, ...
   • What can we use as ground truth?
     • Probably the true colocalization between molecules
     • If molecules have never been observed together, we might use prior knowledge metabolic networks
     • Simulated data?