xulabs / aitom Goto Github PK

create a tutorial page and add links to introductory materials of cryo-ET

We will gradually interface AITom with the Scipion framework as a mid-term effort.

https://github.com/I2PC/scipion/wiki

https://github.com/scipion-em

https://journals.iucr.org/d/issues/2019/10/00/ic5108/index.html

In line 91 of
https://github.com/xulabs/aitom/blob/master/aitom/pick/dog/particle_picking_dog__util.py

No function named "gaussian.dog_smooth__large_map" in https://github.com/xulabs/aitom/blob/master/aitom/filter/gaussian.py

@xiangruz ask MCDS student to do the following

add the code under following module
aitom/aitom/classify/deep/supervised/cnn/semi/autoencoder

use existing functions in AITom as much as possible

write a tutorial and add to
https://github.com/xulabs/aitom/blob/master/tutorials/012_semi_supervised_autoencoder.py

prepare page and figures to be added to
https://github.com/xulabs/projects/semi_supervised_autoencoder/readme.md

write 10 - 20 slides and add to google drive

@zhuzhenxi please add instructions of pip deployment to the following document:

https://github.com/xulabs/aitom/blob/master/doc/deploy.md

@ijinjay please take a look and see if it is possible to integrate the following plugin into aitom

https://github.com/scipion-em/scipion-em-aitom

entry function
aitom.average.ml.faml.faml.test_EM_real_data

The method is based on the following paper:

Zhao Y, Zeng X, Guo Q, Xu M. An integration of fast alignment and maximum-likelihood methods for electron subtomogram averaging and classification. ISMB 2018. Bioinformatics. 2018 Jul 1; 34(13): i227–i236. doi:10.1093/bioinformatics/bty267. arXiv:1804.01203

Let's first fix the bugs to make the program run.

@ijinjay for technical questions, can consult with @xiangruz .

code from
https://github.com/xulabs/projects/tree/master/particle_subdivision

add to
aitom/aitom/classify/deep/supervised/cnn/

use data from following location for testing
https://github.com/xulabs/aitom/blob/master/tutorials/000_general.md

https://github.com/xulabs/aitom/blob/master/tutorials/006_autoencoder.py

In the file saliency_detection.py, at line 384, the distance formulae were expected to be from page 4 of the source paper by Zhou B. et al. However, the weight coefficient is omitted from the code. As varying the weight coefficient results in comparatively different outputs, I wanted to know whether the omission was intended or not. We have integrated the weight coefficient as such,

D = math.sqrt(dc2 + w * ds2 * math.pow(m/interval , 2) )

Hello,

I run the tutorial 005_subtomogram_alignment.py, and an error happens as following:

It seems that the aitom->tomominer->core has no file called 'core'. Can you help me? very thanks

Thanks for presenting this excellent work!

There is an error when reproducing, and I have tried too many times with googling and searching on stackoverflow, however it hasn't been addressed.

The error looks like this: terminal outputs that point2model doesn't exist, what's the reason?

(aitom) root@autodl-container-1c794199d2-3cea689b:~/aitom/doc/tutorials# python 008_particle_picking.py sigma1=2 file has been read partition num (6, 6, 2, 3, 2) (0, 0, 0) (0, 0, 1) (0, 1, 0) (0, 1, 1) (0, 2, 0) (0, 2, 1) (0, 3, 0) (0, 3, 1) (0, 4, 0) (0, 4, 1) (0, 5, 0) (0, 5, 1) (1, 0, 0) (1, 0, 1) (1, 1, 0) (1, 1, 1) (1, 2, 0) (1, 2, 1) (1, 3, 0) (1, 3, 1) (1, 4, 0) (1, 4, 1) (1, 5, 0) (1, 5, 1) (2, 0, 0) (2, 0, 1) (2, 1, 0) (2, 1, 1) (2, 2, 0) (2, 2, 1) (2, 3, 0) (2, 3, 1) (2, 4, 0) (2, 4, 1) (2, 5, 0) (2, 5, 1) (3, 0, 0) (3, 0, 1) (3, 1, 0) (3, 1, 1) (3, 2, 0) (3, 2, 1) (3, 3, 0) (3, 3, 1) (3, 4, 0) (3, 4, 1) (3, 5, 0) (3, 5, 1) (4, 0, 0) (4, 0, 1) (4, 1, 0) (4, 1, 1) (4, 2, 0) (4, 2, 1) (4, 3, 0) (4, 3, 1) (4, 4, 0) (4, 4, 1) (4, 5, 0) (4, 5, 1) (5, 0, 0) (5, 0, 1) (5, 1, 0) (5, 1, 1) (5, 2, 0) (5, 2, 1) (5, 3, 0) (5, 3, 1) (5, 4, 0) (5, 4, 1) (5, 5, 0) (5, 5, 1) 3691 particles detected, containing redundant peaks peak number reduced to 1573 T=m+t*(M-m)/20 T=-1.859648 m=1.082458 t=3.000000 M=-18.531582 DoG done, 1000 particles picked [{'uuid': '1447f963-8b95-4b07-b707-387f095d4b4e', 'val': -18.53158187866211, 'x': [52, 78, 18]}, {'uuid': '2d0e899d-4fc1-4700-b912-46668aa6b152', 'val': -7.802406311035156, 'x': [111, 231, 59]}, {'uuid': '61069558-935a-4d8e-a153-84e9ed8564d8', 'val': -7.356437683105469, 'x': [40, 180, 40]}, {'uuid': 'c1074f77-2c69-4a2f-9b1a-7b4eae26f988', 'val': -7.14599609375, 'x': [59, 139, 44]}, {'uuid': '79bf4582-46ab-441f-a292-1b590c609242', 'val': -6.927955627441406, 'x': [200, 231, 79]}] Save subvolumes .pickle file to: demo_single_particle_subvolumes.pickle generating /tmp/3dmod-points--5535779a-fb0b-11ee-a41d-2329819c1e8c /tmp/3dmod-points--5535779a-fb0b-11ee-a41d-2329819c1e8c: 1: point2model: not found Traceback (most recent call last): File "/root/aitom/doc/tutorials/008_particle_picking.py", line 138, in <module> main() File "/root/aitom/doc/tutorials/008_particle_picking.py", line 134, in main display_map_with_lines(l=l, map_file=path) File "/root/.local/lib/python3.9/site-packages/aitom-0.0.1-py3.9-linux-x86_64.egg/aitom/pick/plot/particle_location_display_imod.py", line 65, in display_map_with_lines assert os.path.isfile(model_file) AssertionError

put the code under:
aitom/aitom/classify/deep/supervised/cnn/autoencoding

@xiangruz please guide @wendingp

basic steps:

1. particle picking
   https://github.com/xulabs/aitom/blob/master/tutorials/008_particle_picking.py

2. autoencoder
   https://github.com/xulabs/aitom/blob/master/tutorials/006_autoencoder.py

3. template search
   need to prepare a tutorial for this part
   related issue #14

As an exercise, see if you can process the following tomogram

https://github.com/xulabs/aitom/blob/master/tutorials/000_general.md

Then try the public data of the following paper

Routine single-particle CryoEM sample and grid characterization by tomography

The tomograms are separated in multiple empiar engries. Following is one of examples:
https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10143/

We can use numpy to crop a volume, but if we save the volume into a mrc file, the header information is incorrect. To solve this problem there are two possible ways:

1. try to find a command line tool to do such cropping. This command line tool may exist in bsoft package
2. write our own code to correct the header and write into the mrc file using the mrcfile package.

Following is how to crop a tomogram using bshow: To cut a subvolume using bshow: 1) choose micrograph / pick particles 2) set box size 3) in boxes menu, choose extract particles 4) may check individual output files. 5) use micrograph / write parameter file. You can first cut according to the low passed filtered map, save the cutted subvolume and star file, then modified star file and load it with the unfiltered tomogram. Then cut the unfiltered tomogram and save the subvolume.

Integrate the code from our simulated annealing paper

In the first tutorial:

https://github.com/xulabs/aitom/blob/master/tutorials/1.Load_volume_and_display.ipynb

Cannot use aitom.image.vol.util.dsp_img to display image.

Pls add aitom.tomominer.image.util to fix it.

write 10 - 20 slides and add to google drive, need to complete before Sep 15

The following tasks are not urgent, can be performed after the AAAI deadline

add the code under following module
aitom/aitom/classify/deep/supervised/cnn/openset

use existing functions in AITom as much as possible

write a tutorial and add to
https://github.com/xulabs/aitom_doc/blob/master/tutorials/011_openset_learning.py

prepare page and figures to be added to
https://github.com/xulabs/projects/openset_learning/readme.md

(low priority task)

first submit it to arXiv, so that users can cite it when needed
in future, will submit it to a journal for publication

Would be nice to have a faster Gaussian blur implementation than scipy's implementation in the case of larger, unbinned tomogram input

Would be cool to offer .hdf support in aitom.io!

See if you can understand following tutorial and successfully run it

https://github.com/xulabs/aitom/blob/master/tutorials/010_deep_learning_subtomogram_classification.py

It would be great if AITom could be used in Jupyter Notebook and Google Colab.

1：
https://github.com/xulabs/aitom/blob/master/aitom/pick/dog/particle_picking_dog__util.py
ImportError: No module named 'aitom.tomominer.image.vol.partition' in line 43.
Maybe it should be changed to 'aitom.image.vol.partition' ?
2:
https://github.com/xulabs/aitom/blob/master/aitom/filter/gaussian.py
NameError: name 'fftn' is not defined in line 34. It seems that some modules or functions have not been imported. Currently, tutorial_008 uses dog_smooth instead of dog_smooth__large_map.

Detect which same function is defined in multiple places. So that we can remove them and only keep a single copy.

import tomominer_mbc

in faml.py & aitom/geometry/volume/hypervolume/util.py raised error
"No module named 'tomominer_mbc'"

module aitom.filter.gaussian has no attribute dog_smooth__large_map

Many IndentationError errors need to be fixed.

‼️ We should make a toy example to test the code as this week's objective. For example:

1. read mrc file using io module
2. filter the image with Guassian smoothing kernel.
3. align the data from different mrc files
4. other tasks.

😃

aitom library now relies on the external program installed in Linux to show result pictures. This will make the library not working on clusters and docker environments. We need to use qt or web to show the pictures.

related functions are inside aitom.aitom.pick.dog.particle_picking_dog__util

can start with making it working on a simulated tomogram

Then make it working on a publically avaliable real tomogram, such as
https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10045/

Related paper:
Pei L, Xu M, Frazier Z, Alber F. Simulating Cryo-Electron Tomograms of Crowded Mixtures of Macromolecular Complexes and Assessment of Particle Picking. BMC Bioinformatics. 2016; 17: 405.

To keep the codebase lean, we only include source code and text files. On the other hand, often there is a need to have other files to run the code. We need to find a way to manage these non-text files and patch them after downloading the AITom code.

change name and folder

https://github.com/xulabs/aitom/blob/master/doc/publications.md

Hi,

I was testing the Gum-Net and for that used the provided demo data set.
After around 30 epochs and around 15 hours of training I stopped it because there is no improvement in the
loss function, please see below the logs of the training procedure.

Before finetuning:
Rotation error: 1.7350925500744534 +/- 0.6650064011311111 Translation error: 8.442523177067761 +/- 3.44293514383784 ----------
Training Iteration 0
4/4 [==============================] - 1784s 404s/step - loss: 0.8216
Training Iteration 1
4/4 [==============================] - 1781s 405s/step - loss: 0.8218
Training Iteration 2
4/4 [==============================] - 1774s 404s/step - loss: 0.8251
Training Iteration 3
4/4 [==============================] - 1788s 406s/step - loss: 0.8274
Training Iteration 4
4/4 [==============================] - 1783s 405s/step - loss: 0.8334
Training Iteration 5
4/4 [==============================] - 1782s 405s/step - loss: 0.8201
Training Iteration 6
4/4 [==============================] - 1777s 405s/step - loss: 0.8250
Training Iteration 7
4/4 [==============================] - 1797s 407s/step - loss: 0.8310
Training Iteration 8
4/4 [==============================] - 1787s 407s/step - loss: 0.8336
Training Iteration 9
4/4 [==============================] - 1784s 406s/step - loss: 0.8207
Training Iteration 10
4/4 [==============================] - 1787s 406s/step - loss: 0.8258
Training Iteration 11
4/4 [==============================] - 1779s 405s/step - loss: 0.8235
Training Iteration 12
4/4 [==============================] - 1784s 406s/step - loss: 0.8296
Training Iteration 13
4/4 [==============================] - 1773s 402s/step - loss: 0.8271
Training Iteration 14
4/4 [==============================] - 1773s 403s/step - loss: 0.8199
Training Iteration 15
4/4 [==============================] - 1785s 406s/step - loss: 0.8315
Training Iteration 16
4/4 [==============================] - 1789s 407s/step - loss: 0.8264
Training Iteration 17
4/4 [==============================] - 1777s 405s/step - loss: 0.8336
Training Iteration 18
4/4 [==============================] - 1774s 403s/step - loss: 0.8299
Training Iteration 19
4/4 [==============================] - 1790s 407s/step - loss: 0.8303
Training Iteration 20
4/4 [==============================] - 1784s 406s/step - loss: 0.8244
Training Iteration 21
4/4 [==============================] - 1786s 407s/step - loss: 0.8242
Training Iteration 22
4/4 [==============================] - 1789s 406s/step - loss: 0.8245
Training Iteration 23
4/4 [==============================] - 1782s 406s/step - loss: 0.8253
Training Iteration 24
4/4 [==============================] - 1789s 405s/step - loss: 0.8258
Training Iteration 25
4/4 [==============================] - 1784s 406s/step - loss: 0.8238
Training Iteration 26
4/4 [==============================] - 1782s 405s/step - loss: 0.8200
Training Iteration 27
4/4 [==============================] - 1779s 405s/step - loss: 0.8282
Training Iteration 28
4/4 [==============================] - 1780s 405s/step - loss: 0.8251
Training Iteration 29
2/4 [==============>...............] - ETA: 19:00 - loss: 0.8142

Do you have any suggestions or explanation why the training with your demo dataset is not working? I did not change the source code.

Kind regards!

Hi, I am looking through the repository especially for the simulation of tomogram data.
However, as mentioned in the issue #167, there are some missing modules not publicly available such as aitom_core.
Is there any plan to share those modules in the near future, hopefully?

Best Regards!

prepare a single set of demo data for as many tasks as possible, for example
particle picking, autoencoder, subtomogram alignment, faml, etc
make the demo data publically avaliable

Can be a binned version of the following tomogram

http://ftp.ebi.ac.uk/pub/databases/empiar/archive/10045/data/ribosomes/Tomograms/05/IS002_291013_005.mrc

for the binning, need to use bint instead of downscale_local_mean
because bint will automatically adjust the MRC header information

According to
Feature Decomposition Based Saliency Detection in Electron Cryo-Tomograms.
https://arxiv.org/abs/1801.10562

the supervoxel computation could be implemented on GPU using numba.

see issue #13

can test on existing data
https://github.com/xulabs/aitom/blob/master/tutorials/000_general.md

also on simulated subtomograms
https://github.com/xulabs/aitom/blob/master/tutorials/003_simulate_subtomogram.py

to be added to aitom.average.align.simple_iterative.average

Greetings,

I am currently facing a problem whenever I try to work with ProtoNet. As per the documentation, if I run python run_train.py, I get the following error.

Traceback (most recent call last):
  File "run_train.py", line 4, in <module>
    from train import main
  File "/shared/home/v_ajmain_yasar_ahmed_sahil/aitom/aitom/classify/deep/supervised/cnn/few_shot/protonet/train.py", line 20, in <module>
    from protonets.utils import data as data_utils
  File "/shared/home/v_ajmain_yasar_ahmed_sahil/aitom/aitom/classify/deep/supervised/cnn/few_shot/protonet/protonets/utils/data.py", line 1, in <module>
    import protonets.data
ModuleNotFoundError: No module named 'protonets.data'

Apparently, there is no protonets.data file under the directory protonets/utils where the file data.py resides. Am I missing something here?

Thank you.

Hey,

thanks for developing AITOM. I can't get the program to run on colab as well as on docker because it runs every time in Module not Found errors. When building the docker container, the processes exits every time with error code because of Keras being 2.1 or 2.6 but the other version is needed or vice versa. This happens for the package six also. Even after editing the requirements.txt and/or Dockerfile it still runs in dependency issues. Are the build and dependecy files maybe outdated?

Many thanks in advance!

add autoencoder code under

aitom.classify.deep.unsupervised.autoencoder

according to
https://github.com/xulabs/projects/tree/master/autoencoder

and make it working.

write a tutorial on how to use it, and put the tutorial in
aitom/tutorials/006_autoencoder.py

Add functions needed to produce figures that are similiar to following

https://user-images.githubusercontent.com/31047726/51266413-3613ec00-1989-11e9-810f-f8cb4924f435.png

write a tutoral so that user can produce such figure themself by reading the tutorial.

Hi, when calling aitom/aitom/geometry/pack/sphere/few/map_tomo/map2tomogram.py on 171.
import tomominer.simulation.reconstruction__simple_convolution as TSRSC cause no module tomominer, and this function can be used before 2021/1/11.
When I try using:
sys.path.append('/ldap_shared/shared/src/aitom_core/')
import aitom_core.simulation.reconstruction.reconstruction__simple_convolution as TSRSC
to load this function, Marshal loads failed error occurs.
I'm using py2.7 and I found reconstruction__simple_convolution function is compiled recently using py3.5. I'm not sure how to call this function correctly now.

Here we will put any images to be linked to the documents or code of AITom

tutorial 002 & 003:

No module named 'aitom.tomominer.structure'
No module named 'aitom.tomominer.simulation'

Pls fix it.

this could be done using astor
place the code under aitom.prog.qa module