Preprocessing methylation pipeline, written in python. Easy to use and highly parallelized.
Home Page: https://christensen-lab-dartmouth.github.io/PyMethylProcess/
License: MIT License
Hello @jlevy44 Thanks for the awesome tool. This is not an issue but a query. Is the preprocessed data, dependent or independent from the labels provided in the samplesheet file?
Thanks in advance.
Adding dask support would be a huge boost to this workflow in terms of parallelization, and could speed up many of the preprocessing commands.
First target could be speeding up BMIQ normalization, which needs debugging anyways.
Hello,
I am using pymethyl-process to process IDAT files in the hope of extracting the beta values from multiple samples. Currently I am running it as such:
pymethyl-preprocess preprocess_pipeline --n_cores 8 --idat_dir /mnt/d/IDATS/205130650082/ --pipeline minfi --noob_norm --output_pk
l /mnt/d/TRUD/pymethyl_run/
Where the idat directory is a folder containing all IDAT files. One thing to note, II am currently running this all through the Ubuntu subarchitecture through WSL.
However, running this produces an error which suggests that the path is wrong:
/home/varundwaraka/anaconda3/lib/python3.8/site-packages/rpy2/robjects/pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace.
from pandas.core.index import Index as PandasIndex
/home/varundwaraka/anaconda3/lib/python3.8/site-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: [read.metharray.sheet] Found the following CSV files:
warnings.warn(x, RRuntimeWarning)
character(0)
/home/varundwaraka/anaconda3/lib/python3.8/site-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: Error in list.files(path = base, pattern = "_Grn.idat$", recursive = recursive, :
invalid 'path' argument
warnings.warn(x, RRuntimeWarning)
/home/varundwaraka/anaconda3/lib/python3.8/site-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: In addition:
warnings.warn(x, RRuntimeWarning)
/home/varundwaraka/anaconda3/lib/python3.8/site-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: There were 25 warnings (use warnings() to see them)
warnings.warn(x, RRuntimeWarning)
/home/varundwaraka/anaconda3/lib/python3.8/site-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning:
warnings.warn(x, RRuntimeWarning)
Traceback (most recent call last):
File "/home/varundwaraka/anaconda3/bin/pymethyl-preprocess", line 8, in <module>
sys.exit(preprocess())
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/pymethylprocess/preprocess.py", line 293, in preprocess_pipeline
preprocesser.preprocess_enmix_pipeline(n_cores=n_cores, pipeline=pipeline, noob=noob_norm, use_cache=use_cache, qc_only=qc_only, bmiq=bmiq)
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/pymethylprocess/PreProcessDataTypes.py", line 606, in preprocess_enmix_pipeline
self.load_idats()
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/pymethylprocess/PreProcessDataTypes.py", line 514, in load_idats
self.RGset = self.minfi.read_metharray_exp(targets=targets, extended=True)
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/rpy2/robjects/functions.py", line 178, in __call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "/home/varundwaraka/anaconda3/lib/python3.8/site-packages/rpy2/robjects/functions.py", line 106, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in list.files(path = base, pattern = "_Grn.idat$", recursive = recursive, :
invalid 'path' argument
However, checking the path to the folder suggests that it is correct:
(base) varundwaraka@LAPTOP-6T391O1J:/mnt/d/IDATS/205130650082$ pwd
/mnt/d/IDATS/205130650082
Is there any explanation as to why this error is occuring? Is it a path issue? Or is this a different issue within pandas? Any insight will be helpful!
Thanks,
Varun
Note that generation of QC objects runs to completion, need to use use_cache option to finish with normalization. Laptop has low memory so consider reducing number of cores and increase docker memory. Should be able to work because got over half way through QC process. May need to make normalization more memory efficient. Almost there!!!
$plot
Check QC report and select number of PCs. Will add option in future to adjust thresholds.
/usr/local/lib/python3.6/dist-packages/rpy2/robjects/pandas2ri.py:191: FutureWarning: from_items is deprecated. Please use DataFrame.from_dict(dict(items), ...) instead. DataFrame.from_dict(OrderedDict(items)) may be used to preserve the key order.
res = PandasDataFrame.from_items(items)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: Error in FUN(X[[i]], ...) :
The operating system has decided that some forks of mclapply are using too much memory.
Try reducing the max.bytes parameter or the R option 'mc.cores'.
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: In addition:
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: There were 29 warnings (use warnings() to see them)
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning:
warnings.warn(x, RRuntimeWarning)
Traceback (most recent call last):
File "/usr/local/bin/pymethyl-preprocess", line 10, in <module>
sys.exit(preprocess())
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pymethylprocess/preprocess.py", line 280, in preprocess_pipeline
preprocesser.preprocessMeffil(n_cores=n_cores,n_pcs=n_pcs,qc_report_fname=os.path.join(output_dir,'qc.report.html'), normalization_report_fname=os.path.join(output_dir,'norm.report.html'), pc_plot_fname=os.path.join(output_dir,'pc.plot.pdf'), useCache=use_cache, qc_only=qc_only, qc_parameters=qc_parameters)
File "/usr/local/lib/python3.6/dist-packages/pymethylprocess/PreProcessDataTypes.py", line 453, in preprocessMeffil
pval_beadnum=load_detection_p_values_beadnum(qc_list,n_cores)
File "/usr/local/lib/python3.6/dist-packages/pymethylprocess/meffil_functions.py", line 64, in load_detection_p_values_beadnum
}""")(qc_list, n_cores)
File "/usr/local/lib/python3.6/dist-packages/rpy2/robjects/functions.py", line 178, in __call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/rpy2/robjects/functions.py", line 106, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in FUN(X[[i]], ...) :
The operating system has decided that some forks of mclapply are using too much memory.
Try reducing the max.bytes parameter or the R option 'mc.cores'.
Hello, I was unable to find it anywhere. I wanted to know the difference between ENmix and meffil library. Is one better than the other in some way?
Implement in rpy2:
I followed the example to download the dataset GSE87571, but I can only find the idat files in the directory, but not the CSV file.
I ran:
pymethyl-preprocess create_sample_sheet -is ./geo_idats/GSE81961_clinical_info.csv -s geo -i geo_idats/ -os geo_idats/samplesheet.csv -d "disease state:ch1" -c include_col.txt
And got:
Please move all other sample sheets out of this directory.
Please remove ./geo_idats/GSE81961_clinical_info.csv from geo_idats/, if it exists in that directory.
Any idea what I need to change?
rpy2 cannot seem to find original R install path (R 3.6 installed but looking for 3.5 path specific to older mac os):
pymethyl-preprocess -h
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/bin/pymethyl-preprocess", line 11, in <module>
load_entry_point('pymethylprocess==0.1.4', 'console_scripts', 'pymethyl-preprocess')()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
return ep.load()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymethylprocess/preprocess.py", line 8, in <module>
from pymethylprocess.PreProcessDataTypes import *
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymethylprocess/PreProcessDataTypes.py", line 8, in <module>
import rpy2.robjects as robjects
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpy2/robjects/__init__.py", line 16, in <module>
import rpy2.rinterface as rinterface
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpy2/rinterface/__init__.py", line 51, in <module>
from rpy2.rinterface._rinterface import (baseenv,
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpy2/rinterface/_rinterface.cpython-37m-darwin.so, 2): Library not loaded: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libR.dylib
Referenced from: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpy2/rinterface/_rinterface.cpython-37m-darwin.so
Reason: image not found
Issue documented here:
https://bitbucket.org/rpy2/rpy2/issues/503/cannot-use-rpy2-in-macos-mojave
Related: https://bitbucket.org/rpy2/rpy2/issues/403/cannot-pip-install-rpy2-with-latest-r-340
Possible solution: https://bitbucket.org/rpy2/rpy2/issues/487/library-not-loaded-usr-local-opt-r-lib-r
Finds R 3.6 with python3 -m rpy2.situation
Hi, I am encountering the following reported error while preprocessing my own methylated idat data using the following command line. Can you please help me to check what is the reason for this error.
Both '-u' and '-qc' work correctly on the example dataset, while only '-qc' works on my dataset and '-u' produces the following error.
errored commend line:
pymethyl-preprocess preprocess_pipeline -i ADNI_iDAT_files/ -p minfi -noob -u
The idat file is from the methylation raw data from ADNI, And i have checked the correctness of the samplesheet.csv file format.
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: Loading required package: IlluminaHumanMethylationEPICmanifest
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: Error in ctrl_g[cc$Address, ] : incorrect number of dimensions
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: In addition:
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning: There were 25 warnings (use warnings() to see them)
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/__init__.py:146: RRuntimeWarning:
warnings.warn(x, RRuntimeWarning)
Traceback (most recent call last):
File "/usr/local/bin/pymethyl-preprocess", line 10, in <module>
sys.exit(preprocess())
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pymethylprocess/preprocess.py", line 282, in preprocess_pipeline
preprocesser.preprocess_enmix_pipeline(n_cores=n_cores, pipeline=pipeline, noob=noob_norm, use_cache=use_cache, qc_only=qc_only)
File "/usr/local/lib/python3.6/dist-packages/pymethylprocess/PreProcessDataTypes.py", line 606, in preprocess_enmix_pipeline
self.preprocessRAW()
File "/usr/local/lib/python3.6/dist-packages/pymethylprocess/PreProcessDataTypes.py", line 359, in preprocessRAW
self.qcinfo = self.enmix.QCinfo(self.RGset, detPthre=1e-7)
File "/usr/local/lib/python3.6/dist-packages/rpy2/robjects/functions.py", line 178, in __call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/rpy2/robjects/functions.py", line 106, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in ctrl_g[cc$Address, ] : incorrect number of dimensions
Reported issue when loading pymethyl-utils, need to upgrade setuptools to latest version in install.
Hello, when following the example for download of GSE87571, I get the following error when running create_sample_sheet command. Could you please help? (I am running the pyMethylProcess from a singularity container created form the docker image provided)
Traceback (most recent call last): File "/usr/local/bin/pymethyl-preprocess", line 10, in <module> sys.exit(preprocess()) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/pymethylprocess/preprocess.py", line 76, in create_sample_sheet pheno_sheet.format_geo(disease_class_column, include_columns) File "/usr/local/lib/python3.6/dist-packages/pymethylprocess/PreProcessDataTypes.py", line 138, in format_geo idat_basenames = np.unique(np.vectorize(lambda x: '_'.join(x.split('/')[-1].split('_')[:3]))(idats)) File "/usr/local/lib/python3.6/dist-packages/numpy/lib/function_base.py", line 1972, in __call__ return self._vectorize_call(func=func, args=vargs) File "/usr/local/lib/python3.6/dist-packages/numpy/lib/function_base.py", line 2042, in _vectorize_call ufunc, otypes = self._get_ufunc_and_otypes(func=func, args=args) File "/usr/local/lib/python3.6/dist-packages/numpy/lib/function_base.py", line 1998, in _get_ufunc_and_otypes raise ValueError('cannot call vectorizeon size 0 inputs ' ValueError: cannot callvectorizeon size 0 inputs unlessotypes is set
The next major release of pymethylprocess will have an experimental module with support for other input formats of methylated/unmethylated signal rather than only IDATs.
Following will be added support for BS-Seq and RRBS.
I'm trying to download the clinical information on a Docker Machine using pymethyl-preprocess download_clinical and I'm getting the following error:
PS: It would be great if someone can upload the clinical data for TCGA. So that I can continue working in the meantime. 😃
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.