raamana / neuropredict Goto Github PK
View Code? Open in Web Editor NEWEasy and comprehensive assessment of predictive power, with support for neuroimaging features
Home Page: https://raamana.github.io/neuropredict/
License: Apache License 2.0
Easy and comprehensive assessment of predictive power, with support for neuroimaging features
Home Page: https://raamana.github.io/neuropredict/
License: Apache License 2.0
Given a data structure such pyradigm capable of holding attributes for each sample (such as site, scanner and other meta-data), device an estimating pipeline (CV loop + other infra) that can produce reports on the impact of that attribute.
The [frequently] misclassified subjects are a helpful diagnostic list in classification analyses, and the regression equivalent would be a list of subjects with high frequency of large residuals.
besides the misclassification frequency plots (which can be a helpful diagnostic), it may be useful to offer an option to enable the user to compare and study the differences characteristics between frequency misclassified vs. frequently correctly classified..
Characteristics could be the input features themselves, or the other associated meta data and attributes (e.g. provided as part of pyradigm
)
examples include: those frequently misclassified were mostly the older patients , and those frequently correctly classified has "property X" etc.
such as Facebook Ax etc
CVResults
might be individually even more useful than others in the API. So document it thoroughly align with the rest with a tutorial or two for use by dev and reuse by the replicator or meta-analytic researcher.
Hi Pradeep,
When trying to launch multi-class classifier (whatever the classifier chosen) on a 3-class (CN, MCI, AD) problem, I got the following problem:
Traceback (most recent call last):
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/bin/neuropredict", line 8, in <module>
sys.exit(main())
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/__main__.py", line 11, in main
run_workflow.cli()
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/run_workflow.py", line 969, in cli
grid_search_level, classifier, feat_select_method = parse_args()
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/run_workflow.py", line 525, in parse_args
class_set, subgroups, positive_class = validate_class_set(classes, user_args.sub_groups, user_args.positive_class)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/run_workflow.py", line 659, in validate_class_set
''.format(comb))
ValueError: Subgroup all does not contain 2 unique classes! Each subgroup must contain atleast two classes for classification experiments.
I don't understand since I have well defined 3-class in the metadata.csv file and used the following command line:
neuropredict -m meta_data.csv \
-d metaROI.csv metaROI_split.csv HCP_parcellation.csv \
-t 0.8 -n 250 -k 'all' --sub_groups 'all' --gs_level 'exhaustive' \
--feat_select_method 'variancethreshold' --classifier ${classifier} --make_vis ./visu \
--num_procs ${NCPU} --print_options ./options
Could you help me?
Best,
Matthieu
Currently auto-testing usage and behaviour.
Hi,
When I try to run neuropredict on the command line using
neuropredict -m ~/NeuroPredict/meta_data.csv -o ~/NeuroPredict/results/ --user_feature_paths ~/NeuroPredict/face/ ~/NeuroPredict/cnlg/ ~/NeuroPredict/glmn/ ~/NeuroPredict/cnsr/ ~/NeuroPredict/bimndir/ ~/NeuroPredict/bimnemo/ ~/NeuroPredict/msep/ -t 0.8 -n 20 -g 'light'
I get the error that ImportError: cannot import name 'MultiDatasetClassify' from 'pyradigm.multiple'
It appears that the multiple.py file that I've downloaded does not match the current version that is in your pyradigm repository. I installed it just now though, using pip3 install neuropredict. Do you know why I am getting this problem?
Thanks
few ideas:
CSV input
sbuject_id,class,feature1,feature2,age,sex
acbde_01,healthy,1.2,2.3,3.4,74,female
fghijk_01,healthy,1.2,2.3,3.4,76,male
...
lmnop_01,alzheimer,1.2,2.3,3.4,70,female
...
qrstu_01,depression,1.2,2.3,3.4,71,male
Directory of directories input
Hi @raamana ,
I got an error when trying to launch both multi-class or binary classifications using xgboost:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/rhst.py", line 624, in holdout_trial_compare_datasets
feat_select_method=feat_select_method)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/rhst.py", line 102, in eval_optimized_model_on_testset
param_grid, train_perc)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/rhst.py", line 194, in optimize_pipeline_via_grid_search_CV
gs.fit(train_data_mat, train_labels)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/sklearn/model_selection/_search.py", line 688, in fit
self._run_search(evaluate_candidates)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/sklearn/model_selection/_search.py", line 1149, in _run_search
evaluate_candidates(ParameterGrid(self.param_grid))
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/sklearn/model_selection/_search.py", line 667, in evaluate_candidates
cv.split(X, y, groups)))
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/joblib/parallel.py", line 1003, in __call__
if self.dispatch_one_batch(iterator):
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/joblib/parallel.py", line 834, in dispatch_one_batch
self._dispatch(tasks)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/joblib/parallel.py", line 753, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 201, in apply_async
result = ImmediateResult(func)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 582, in __init__
self.results = batch()
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/joblib/parallel.py", line 256, in __call__
for func, args, kwargs in self.items]
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/joblib/parallel.py", line 256, in <listcomp>
for func, args, kwargs in self.items]
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 516, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/sklearn/pipeline.py", line 356, in fit
self._final_estimator.fit(Xt, y, **fit_params)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/xgboost/sklearn.py", line 732, in fit
callbacks=callbacks)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/xgboost/training.py", line 216, in train
xgb_model=xgb_model, callbacks=callbacks)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/xgboost/training.py", line 74, in _train_internal
bst.update(dtrain, i, obj)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/xgboost/core.py", line 1109, in update
dtrain.handle))
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/xgboost/core.py", line 176, in _check_call
raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: Invalid Parameter format for num_feature expect int (non-negative) but value='sqrt'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/bin/neuropredict", line 8, in <module>
sys.exit(main())
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/__main__.py", line 11, in main
run_workflow.cli()
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/run_workflow.py", line 976, in cli
grid_search_level, classifier, feat_select_method)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/run_workflow.py", line 951, in prepare_and_run
options_path=options_path)
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/rhst.py", line 382, in run
cv_results = pool.map(partial_func_holdout, range(num_repetitions))
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
xgboost.core.XGBoostError: Invalid Parameter format for num_feature expect int (non-negative) but value='sqrt'
Could you help me with this?
Hi Pradeep,
For small and unbalanced dataset, do you recommend to use -t 0.8
or -t 0.9
?
Isn't possible to deactivate in the implemented pipeline the feature selection? If not, what is the advantage of always using feature selection when dealing with a small features' dataset?
Best,
Matthieu
Much needed to make it easy to debug issues on HPC systems and other place
I would like to add a functionality to easily run statistical tests (against null, against other classifiers) on an independent test set. Since the test set is independent, this should be easy to do (no need to deal with dependencies between folds).
IMHO the main task will be to make some usable API
Ensure all parameters are estimated only from training data in sklearn
So far the dimensionality reduction techniques offered are only feature selection
methods, and feature transformation
methods.
it's about time to integrate non-linear (and linear) dimensionality reduction techniques, to offer maximum choice (and potential benefit) to users:
https://scikit-learn.org/stable/modules/manifold.html#manifold
Hi Pradeep,
I can see the file structure for adding a single feature is:
subject1/features.txt
subject2/features.txt
subject3/features.txt
subject4/features.txt
Where the feature is a vector in each case. To clarify, if I wanted to add a second feature, would this be a second column in the features.txt file of each participant?
Thanks,
John
nemenyi posthoc analysis
Missing functionality: non-parametric friedman test on the results from RHsT
xgboost.core.XGBoostError: b"Invalid Parameter format for num_feature expect int (non-negative) but value='sqrt'"
It seems like XGBoost does not accept num_feature parameter anymore - their docs say "it does not need to be set", but the error sounds more like format/range issue
not just compare different feature sets with a fixed builtin model, but users can also input a model of their own choice. it does not limit exploration of new models or pipelines - they can use implementation of best practices while evaluating the such new models on features of their choice
Need to fix it.
Ability to input a new dataset, from a different site or dataset or country, and use the best model to report performance on this dataset
Or an option to specify attribute-based criterion to hold a certain subset out completely to report performance
A good output in the regression version would be residual plots
Good example is from yellowbrick: http://www.scikit-yb.org/en/latest/api/regressor/residuals.html
To do:
We can add/remove as we realize them.
Please, port neuropredict to R ?
Thanks.
for record, and debugging purposes.
Kia Ora,
I have been using the classification workflow, for a binary classification with a positive class set.
However, the output only contains the figures folder and results_neuropredict.pkl
The CSV files that are specified in the neuropredict docs isnt being outputted.
I am trying to get the ROC curve and the full feature importance information - any help would be appreciated.
Thanks
Mukish
> C:\Users\ymye313>neuropredict_classify -y "C:/Users/Documents/Texture - 1st set data/conversion_HC_texture.pkl" -m "C:/Users/Documents/Texture - 1st set data/group.csv" -o "C:/Users/Documents/Texture - 1st set data/predict" -p MCIc -t 0.75 -n 20
>
> neuropredict version 0.6.2 for Classification
> Time stamp : 2023-10-29 14:19:21
>
> Using meta data from:
> C:\Users\Documents\Texture - 1st set data\conversion_HC_texture.pkl
>
> Positive class specified for AUC calculation: MCIc
>
> Requested features for analysis:
> get_pyradigm from C:\Users\Documents\Texture - 1st set data\conversion_HC_texture.pkl
>
> Data import is done.
>
>
> Requested processing for the following subgroups:
> MCInc,MCIc
>
> --------------------------------------------------------------------------------
> Processing subgroup : MCInc,MCIc (1/1)
> --------------------------------------------------------------------------------
> MCIc_MCInc:
> 85 samples, 1 modalities, dims: [14]
> Identifiers: 0
> Attributes: age
> Classes n=2, sizes MCInc: 41, MCIc: 44
>
> Ignoring imputation strategy chosen, as no missing data were found!
>
>
> CURRENT EXPERIMENT:
> --------------------------------------------------
> Training percentage : 0.75
> Number of CV repetitions : 20
> Number of processors : 4
> Dim reduction method : variancethreshold
> Dim reduction size : tenth
> Predictive model chosen : randomforestclassifier
> Grid search level : light
>
> Estimated chance accuracy : 0.500
>
> Saving results to:
> C:\Users\Documents\Texture - 1st set data\predict\MCIc_MCInc
>
>
Sometimes some advanced users would like to choose their model or hyper-parameters for many good reasons including model not existing in sklearn etc or needing deep control over them etc, then having the API or CLI allowing users to do it would be great!
normal run would print only the most important options such as which classifier, cross-validation details etc, whereas verbose option would print which subjects and more info into feature sets etc.
Hello,
I have problems when using multiple 1-D features with the following command:
neuropredict -m meta_data.csv \
-d mask_WMpet_av45_early.ero1.25mm.csv mask_WMpet_av45_early.ero1.5mm.csv \
-o outdir -t 0.8 -n 250 -k 'all' \
--gs_level 'exhaustive' --classifier "LinearSVC"
The CV trial were well launched for the first subgroup CN,MCI for both 1-D features but it seemed that at the end of CV trials the process got stuck at this stage:
>
> Python 3.7.3
> > SGE recognized, job set up with 40 slots.
> > Positive class inferred for AUC calculation: CN
> > Running neuropredict 0.5+34.g220af55.dirty
> >
> > Requested features for analysis:
> > get_data_matrix from /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/mask_WMpet_av45_early.ero1.25mm.csv
> > get_data_matrix from /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/mask_WMpet_av45_early.ero1.5mm.csv
> > Ignoring imputation strategy chosen, as no missing data were found!
> >
> > Data import is done.
> >
> >
> > Requested processing for the following subgroups:
> > DIS,CN
> >
> > --------------------------------------------------------------------------------
> > Processing subgroup : DIS,CN (1/1)
> > --------------------------------------------------------------------------------
> > SGE recognized, job set up with 40 slots.
> > Training percentage : 0.8
> > Number of CV repetitions : 250
> > Classifier chosen : linearsvc
> > Feature selection chosen : variancethreshold
> > Level of grid search : exhaustive
> > Number of processors : 40
> > Saving the results to
> > /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/linearsvc/binary_WMmasks/CN_DIS
> >
> > -------------------------
> > All datasets contain:
> >
> > 101 samples, 2 classes, 1 features
> > Class CN : 71 samples
> > Class DIS : 30 samples
> > -------------------------
> >
> > Estimated chance accuracy : 0.500
> >
> > Different classes in the training set are stratified to match the smallest class!
> > Parallelizing the repetitions of CV with 40 processes ...
> > CV trial 52 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525 weighted AUC: 0.7482
> > CV trial 78 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 weighted AUC: 0.8227
> > CV trial 50 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5993 weighted AUC: 0.6596
> > CV trial 72 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6082 weighted AUC: 0.6596
> > CV trial 46 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 weighted AUC: 0.7021
> > CV trial 58 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.8475
> > CV trial 38 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 weighted AUC: 0.7518
> > CV trial 16 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 weighted AUC: 0.7801
> > CV trial 12 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.8794
> > CV trial 0 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738 weighted AUC: 0.6950
> > CV trial 68 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7057 weighted AUC: 0.8369
> > CV trial 14 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8794
> > CV trial 22 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 weighted AUC: 0.8333
> > CV trial 6 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 weighted AUC: 0.7234
> > CV trial 48 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 weighted AUC: 0.8298
> > CV trial 10 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 weighted AUC: 0.7624
> > CV trial 64 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.8191
> > CV trial 20 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 weighted AUC: 0.8688
> > CV trial 4 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.8688
> > CV trial 56 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 weighted AUC: 0.8262
> > CV trial 70 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.8440
> > CV trial 74 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7553
> > CV trial 36 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.8050
> > CV trial 2 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.9043
> > CV trial 54 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.8830
> > CV trial 24 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 weighted AUC: 0.9043
> > CV trial 76 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 weighted AUC: 0.9220
> > CV trial 42 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 weighted AUC: 0.8759
> > CV trial 26 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.9078
> > CV trial 62 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 weighted AUC: 0.8103
> > CV trial 66 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418 weighted AUC: 0.7624
> > CV trial 60 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8511 weighted AUC: 0.8972
> > CV trial 72 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6294 weighted AUC: 0.6418
> > CV trial 40 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.9149
> > CV trial 52 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720 weighted AUC: 0.7340
> > CV trial 16 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.7411
> > CV trial 46 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720 weighted AUC: 0.6489
> > CV trial 38 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6507 weighted AUC: 0.7376
> > CV trial 78 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 weighted AUC: 0.8014
> > CV trial 50 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6099 weighted AUC: 0.6454
> > CV trial 58 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 weighted AUC: 0.8298
> > CV trial 6 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.6879
> > CV trial 14 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8688
> > CV trial 36 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844 weighted AUC: 0.8156
> > CV trial 18 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.8014
> > CV trial 0 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904 weighted AUC: 0.6454
> > CV trial 20 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.8582
> > CV trial 68 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677 weighted AUC: 0.7979
> > CV trial 4 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8511 weighted AUC: 0.8440
> > CV trial 74 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.7358
> > CV trial 56 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.8050
> > CV trial 34 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5816 weighted AUC: 0.7518
> > CV trial 12 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.8723
> > CV trial 66 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.7376
> > CV trial 70 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844 weighted AUC: 0.8227
> > CV trial 64 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 weighted AUC: 0.8121
> > CV trial 24 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8794
> > CV trial 76 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234 weighted AUC: 0.9007
> > CV trial 79 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5904 weighted AUC: 0.7021
> > CV trial 53 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 weighted AUC: 0.7766
> > CV trial 2 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8936
> > CV trial 73 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.9255
> > CV trial 54 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.8759
> > CV trial 62 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.8014
> > CV trial 17 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 weighted AUC: 0.7624
> > CV trial 22 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677 weighted AUC: 0.8191
> > CV trial 39 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 weighted AUC: 0.7695
> > CV trial 60 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 weighted AUC: 0.9007
> > CV trial 51 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8404 weighted AUC: 0.8227
> > CV trial 26 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.9007
> > CV trial 42 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8546
> > CV trial 48 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.8121
> > CV trial 59 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8014
> > CV trial 69 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7695
> > CV trial 47 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 weighted AUC: 0.7695
> > CV trial 21 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 weighted AUC: 0.9220
> > CV trial 28 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8316 weighted AUC: 0.9007
> > CV trial 37 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.8475
> > CV trial 57 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.8582
> > CV trial 1 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8440
> > CV trial 40 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.9149
> > CV trial 30 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.8582
> > CV trial 18 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 weighted AUC: 0.7589
> > CV trial 65 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7837
> > CV trial 75 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8617
> > CV trial 25 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 weighted AUC: 0.7943
> > CV trial 13 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8121
> > CV trial 79 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525 weighted AUC: 0.6915
> > CV trial 61 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 weighted AUC: 0.7766
> > CV trial 71 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8156
> > CV trial 32 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8316 weighted AUC: 0.9291
> > CV trial 3 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8440
> > CV trial 53 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.7411
> > CV trial 55 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8298
> > CV trial 15 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 weighted AUC: 0.8582
> > CV trial 23 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7872
> > CV trial 34 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5284 weighted AUC: 0.7376
> > CV trial 59 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.7713
> > CV trial 43 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.7766
> > CV trial 77 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 weighted AUC: 0.7305
> > CV trial 49 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 weighted AUC: 0.8528
> > CV trial 27 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8191
> > CV trial 73 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 weighted AUC: 0.8936
> > CV trial 39 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5798 weighted AUC: 0.7305
> > CV trial 47 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.7234
> > CV trial 69 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.7199
> > CV trial 21 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021 weighted AUC: 0.9149
> > CV trial 44 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 weighted AUC: 0.8369
> > CV trial 65 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.7943
> > CV trial 41 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 weighted AUC: 0.7092
> > CV trial 80 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6011 weighted AUC: 0.7305
> > CV trial 51 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738 weighted AUC: 0.8121
> > CV trial 57 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7128 weighted AUC: 0.8369
> > CV trial 75 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8156
> > CV trial 1 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8191
> > CV trial 63 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 weighted AUC: 0.8582
> > CV trial 82 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8475
> > CV trial 3 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8050
> > CV trial 23 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.7270
> > CV trial 61 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.7553
> > CV trial 55 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 weighted AUC: 0.8014
> > CV trial 43 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 weighted AUC: 0.7270
> > CV trial 77 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.7234
> > CV trial 25 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613 weighted AUC: 0.7872
> > CV trial 84 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.9362
> > CV trial 30 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 weighted AUC: 0.8333
> > CV trial 19 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 weighted AUC: 0.8298
> > CV trial 37 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8298
> > CV trial 88 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8121
> > CV trial 96 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6507 weighted AUC: 0.7270
> > CV trial 86 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8759
> > CV trial 13 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.7837
> > CV trial 41 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206 weighted AUC: 0.6809
> > CV trial 5 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617 weighted AUC: 0.9043
> > CV trial 102 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 weighted AUC: 0.8333
> > CV trial 80 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 weighted AUC: 0.7163
> > CV trial 92 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.9220
> > CV trial 15 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.8227
> > CV trial 104 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6436 weighted AUC: 0.7660
> > CV trial 35 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8333
> > CV trial 106 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5585 weighted AUC: 0.7163
> > CV trial 108 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418 weighted AUC: 0.7163
> > CV trial 110 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.6844
> > CV trial 94 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617 weighted AUC: 0.9255
> > CV trial 98 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 weighted AUC: 0.8652
> > CV trial 82 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8546
> > CV trial 100 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7890 weighted AUC: 0.8652
> > CV trial 49 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8440
> > CV trial 63 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234 weighted AUC: 0.8475
> > CV trial 112 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8475
> > CV trial 90 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8511 weighted AUC: 0.8511
> > CV trial 118 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525 weighted AUC: 0.7411
> > CV trial 27 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.7943
> > CV trial 44 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.8014
> > CV trial 124 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.8085
> > CV trial 84 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553 weighted AUC: 0.9255
> > CV trial 110 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645 weighted AUC: 0.6454
> > CV trial 31 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.9149
> > CV trial 96 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401 weighted AUC: 0.7021
> > CV trial 108 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.6738
> > CV trial 126 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8582
> > CV trial 81 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 weighted AUC: 0.7589
> > CV trial 116 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.9184
> > CV trial 104 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.7163
> > CV trial 106 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5585 weighted AUC: 0.7163
> > CV trial 67 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7996 weighted AUC: 0.9007
> > CV trial 92 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.9043
> > CV trial 88 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.7695
> > CV trial 120 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 weighted AUC: 0.7695
> > CV trial 122 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 weighted AUC: 0.8511
> > CV trial 130 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 weighted AUC: 0.7376
> > CV trial 19 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 weighted AUC: 0.8085
> > CV trial 102 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.8262
> > CV trial 98 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8688
> > CV trial 112 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234 weighted AUC: 0.8298
> > CV trial 114 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 weighted AUC: 0.8652
> > CV trial 86 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8475
> > CV trial 35 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8121
> > CV trial 100 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7057 weighted AUC: 0.8475
> > CV trial 71 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.7996
> > CV trial 118 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525 weighted AUC: 0.6986
> > CV trial 94 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 weighted AUC: 0.8901
> > CV trial 83 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738 weighted AUC: 0.8475
> > CV trial 128 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8369
> > CV trial 85 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 weighted AUC: 0.7695
> > CV trial 17 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6631 weighted AUC: 0.7447
> > CV trial 90 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404 weighted AUC: 0.8511
> > CV trial 45 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617 weighted AUC: 0.8191
> > CV trial 111 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6188 weighted AUC: 0.8333
> > CV trial 124 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.7713
> > CV trial 97 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 weighted AUC: 0.8582
> > CV trial 109 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8652
> > CV trial 105 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 weighted AUC: 0.8369
> > CV trial 120 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.7340
> > CV trial 113 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 weighted AUC: 0.6702
> > CV trial 87 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.9362
> > CV trial 81 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 weighted AUC: 0.7305
> > CV trial 136 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5904 weighted AUC: 0.8050
> > CV trial 132 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738 weighted AUC: 0.7589
> > CV trial 116 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8901
> > CV trial 99 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 weighted AUC: 0.8972
> > CV trial 119 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021 weighted AUC: 0.8759
> > CV trial 103 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.9113
> > CV trial 95 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.6879
> > CV trial 134 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 weighted AUC: 0.8014
> > CV trial 130 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.6986
> > CV trial 93 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 weighted AUC: 0.8121
> > CV trial 125 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 weighted AUC: 0.7482
> > CV trial 101 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784 weighted AUC: 0.8652
> > CV trial 91 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021 weighted AUC: 0.9255
> > CV trial 85 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.7482
> > CV trial 128 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8262
> > CV trial 107 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 weighted AUC: 0.7801
> > CV trial 111 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6188 weighted AUC: 0.7979
> > CV trial 5 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 weighted AUC: 0.8865
> > CV trial 31 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.9220
> > CV trial 138 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784 weighted AUC: 0.8901
> > CV trial 114 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 weighted AUC: 0.8333
> > CV trial 121 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8546
> > CV trial 113 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.6312
> > CV trial 97 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.8546
> > CV trial 28 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8528 weighted AUC: 0.8617
> > CV trial 89 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418 weighted AUC: 0.6702
> > CV trial 109 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8688
> > CV trial 122 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.8422
> > CV trial 87 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.9149
> > CV trial 105 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.8191
> > CV trial 83 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.8475
> > CV trial 95 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826 weighted AUC: 0.6879
> > CV trial 140 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8546
> > CV trial 7 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.8830
> > CV trial 125 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.6844
> > CV trial 142 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.8723
> > CV trial 136 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 weighted AUC: 0.7908
> > CV trial 103 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.8759
> > CV trial 119 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021 weighted AUC: 0.8262
> > CV trial 99 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 weighted AUC: 0.8759
> > CV trial 132 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6436 weighted AUC: 0.7234
> > CV trial 129 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7021
> > CV trial 91 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915 weighted AUC: 0.9184
> > CV trial 126 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8493
> > CV trial 150 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.7376
> > CV trial 93 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 weighted AUC: 0.8014
> > CV trial 144 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.7872
> > CV trial 148 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.7323
> > CV trial 146 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.7624
> > CV trial 107 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6294 weighted AUC: 0.7411
> > CV trial 152 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 weighted AUC: 0.8723
> > CV trial 101 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950 weighted AUC: 0.8617
> > CV trial 115 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7234 weighted AUC: 0.9007
> > CV trial 154 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8759
> > CV trial 134 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401 weighted AUC: 0.8014
> > CV trial 29 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8582
> > CV trial 158 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7128 weighted AUC: 0.8121
> > CV trial 166 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 weighted AUC: 0.7872
> > CV trial 137 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7234
> > CV trial 168 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 weighted AUC: 0.7057
> > CV trial 160 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.8440
> > CV trial 123 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8121
> > CV trial 117 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.8652
> > CV trial 164 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 weighted AUC: 0.8369
> > CV trial 45 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950 weighted AUC: 0.7979
> > CV trial 170 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6011 weighted AUC: 0.7766
> > CV trial 150 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.7057
> > CV trial 7 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.8723
> > CV trial 121 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8050
> > CV trial 174 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784 weighted AUC: 0.8298
> > CV trial 176 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 weighted AUC: 0.8369
> > CV trial 162 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 weighted AUC: 0.7660
> > CV trial 156 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.7908
> > CV trial 127 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 weighted AUC: 0.8014
> > CV trial 8 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8723 weighted AUC: 0.9326
> > CV trial 172 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6117 weighted AUC: 0.7589
> > CV trial 144 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553 weighted AUC: 0.7589
> > CV trial 148 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5691 weighted AUC: 0.7021
> > CV trial 129 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4965 weighted AUC: 0.6809
> > CV trial 180 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.8121
> > CV trial 140 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8191
> > CV trial 146 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.7624
> > CV trial 142 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 weighted AUC: 0.8652
> > CV trial 154 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8475
> > CV trial 115 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915 weighted AUC: 0.8865
> > CV trial 29 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 weighted AUC: 0.8227
> > CV trial 166 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021 weighted AUC: 0.7553
> > CV trial 89 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 weighted AUC: 0.6348
> > CV trial 133 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.8298
> > CV trial 168 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.6702
> > CV trial 184 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5780 weighted AUC: 0.7589
> > CV trial 137 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 weighted AUC: 0.6950
> > CV trial 158 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.7837
> > CV trial 152 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234 weighted AUC: 0.8262
> > CV trial 151 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5479 weighted AUC: 0.7305
> > CV trial 131 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.9255
> > CV trial 160 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 weighted AUC: 0.8227
> > CV trial 123 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 weighted AUC: 0.7766
> > CV trial 32 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.0709
> > CV trial 135 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.7713
> > CV trial 170 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 weighted AUC: 0.7270
> > CV trial 188 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.6454
> > CV trial 145 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7234 weighted AUC: 0.7553
> > CV trial 117 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677 weighted AUC: 0.8369
> > CV trial 156 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.7837
> > CV trial 127 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.7890
> > CV trial 186 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 weighted AUC: 0.7979
> > CV trial 178 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.7624
> > CV trial 141 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 weighted AUC: 0.8511
> > CV trial 147 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 weighted AUC: 0.8759
> > CV trial 180 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 weighted AUC: 0.7979
> > CV trial 174 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 weighted AUC: 0.8085
> > CV trial 149 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.8582
> > CV trial 162 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6507 weighted AUC: 0.7163
> > CV trial 172 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904 weighted AUC: 0.7270
> > CV trial 184 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5674 weighted AUC: 0.7340
> > CV trial 151 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826 weighted AUC: 0.7163
> > CV trial 176 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677 weighted AUC: 0.7872
> > CV trial 194 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525 weighted AUC: 0.7092
> > CV trial 188 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645 weighted AUC: 0.5993
> > CV trial 169 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 weighted AUC: 0.8688
> > CV trial 155 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 weighted AUC: 0.7872
> > CV trial 171 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 weighted AUC: 0.7766
> > CV trial 159 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6223 weighted AUC: 0.7482
> > CV trial 167 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8511
> > CV trial 153 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8652
> > CV trial 133 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465 weighted AUC: 0.7766
> > CV trial 143 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.8617
> > CV trial 198 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8333
> > CV trial 161 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.8475
> > CV trial 202 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.6950
> > CV trial 182 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.8333
> > CV trial 164 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844 weighted AUC: 0.8191
> > CV trial 186 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401 weighted AUC: 0.7411
> > CV trial 8 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8830 weighted AUC: 0.0674
> > CV trial 157 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 weighted AUC: 0.9574
> > CV trial 141 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021 weighted AUC: 0.8617
> > CV trial 135 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6631 weighted AUC: 0.7695
> > CV trial 145 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 weighted AUC: 0.7199
> > CV trial 190 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 weighted AUC: 0.7553
> > CV trial 181 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.7979
> > CV trial 196 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7766
> > CV trial 33 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8103 weighted AUC: 0.8936
> > CV trial 192 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8422 weighted AUC: 0.8546
> > CV trial 138 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 weighted AUC: 0.8901
> > CV trial 200 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.9468
> > CV trial 149 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8546
> > CV trial 173 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8475
> > CV trial 177 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8014
> > CV trial 178 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465 weighted AUC: 0.7340
> > CV trial 204 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8404
> > CV trial 206 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.7872
> > CV trial 163 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.8085
> > CV trial 185 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8298
> > CV trial 171 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.7801
> > CV trial 202 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206 weighted AUC: 0.6950
> > CV trial 169 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.8440
> > CV trial 167 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8369
> > CV trial 153 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.8369
> > CV trial 131 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 weighted AUC: 0.9326
> > CV trial 159 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950 weighted AUC: 0.7340
> > CV trial 194 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 weighted AUC: 0.6879
> > CV trial 187 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 weighted AUC: 0.7553
> > CV trial 155 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738 weighted AUC: 0.7660
> > CV trial 212 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.8546
> > CV trial 198 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826 weighted AUC: 0.8191
> > CV trial 189 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8794
> > CV trial 165 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8652
> > CV trial 157 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 weighted AUC: 0.9397
> > CV trial 208 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 weighted AUC: 0.8546
> > CV trial 139 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 weighted AUC: 0.7855
> > CV trial 190 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.7305
> > CV trial 214 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5780 weighted AUC: 0.6879
> > CV trial 181 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7128 weighted AUC: 0.7660
> > CV trial 210 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.8121
> > CV trial 143 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553 weighted AUC: 0.8404
> > CV trial 9 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 weighted AUC: 0.8511
> > CV trial 196 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738 weighted AUC: 0.7518
> > CV trial 182 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.8227
> > CV trial 147 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.8582
> > CV trial 175 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 weighted AUC: 0.9113
> > CV trial 216 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 weighted AUC: 0.7943
> > CV trial 173 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8440
> > CV trial 177 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.7979
> > CV trial 206 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 weighted AUC: 0.7518
> > CV trial 179 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 weighted AUC: 0.8121
> > CV trial 204 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 weighted AUC: 0.8440
> > CV trial 218 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 weighted AUC: 0.8759
> > CV trial 185 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.8014
> > CV trial 226 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6312 weighted AUC: 0.8227
> > CV trial 224 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8475
> > CV trial 220 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.7801
> > CV trial 195 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7872
> > CV trial 200 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8511 weighted AUC: 0.9291
> > CV trial 163 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465 weighted AUC: 0.7908
> > CV trial 161 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8156
> > CV trial 230 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6401 weighted AUC: 0.8227
> > CV trial 199 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 weighted AUC: 0.8050
> > CV trial 212 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613 weighted AUC: 0.8262
> > CV trial 187 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.7447
> > CV trial 232 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6844 weighted AUC: 0.7837
> > CV trial 207 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.4752 weighted AUC: 0.6064
> > CV trial 33 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.0975
> > CV trial 228 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.8440
> > CV trial 203 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 weighted AUC: 0.7766
> > CV trial 139 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525 weighted AUC: 0.7624
> > CV trial 183 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 weighted AUC: 0.8262
> > CV trial 208 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.8404
> > CV trial 165 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8191
> > CV trial 234 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 weighted AUC: 0.9468
> > CV trial 214 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206 weighted AUC: 0.6631
> > CV trial 216 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.7589
> > CV trial 191 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 weighted AUC: 0.8511
> > CV trial 240 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.8440
> > CV trial 236 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.8262
> > CV trial 210 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.7943
> > CV trial 242 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 weighted AUC: 0.8582
> > CV trial 224 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553 weighted AUC: 0.7943
> > CV trial 189 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 weighted AUC: 0.8387
> > CV trial 220 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.7163
> > CV trial 213 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6950 weighted AUC: 0.8156
> > CV trial 238 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6844 weighted AUC: 0.8085
> > CV trial 218 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8582
> > CV trial 230 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401 weighted AUC: 0.7730
> > CV trial 226 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6312 weighted AUC: 0.8014
> > CV trial 179 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.8014
> > CV trial 201 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6507 weighted AUC: 0.8298
> > CV trial 207 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645 weighted AUC: 0.5816
> > CV trial 246 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 weighted AUC: 0.8121
> > CV trial 248 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6117 weighted AUC: 0.7163
> > CV trial 195 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.7518
> > CV trial 199 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.7837
> > CV trial 183 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525 weighted AUC: 0.8298
> > CV trial 244 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 weighted AUC: 0.8298
> > CV trial 203 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.7270
> > CV trial 209 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 weighted AUC: 0.8511
> > CV trial 205 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5691 weighted AUC: 0.7801
> > CV trial 217 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.9149
> > CV trial 240 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.8191
> > CV trial 221 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 weighted AUC: 0.9291
> > CV trial 225 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 weighted AUC: 0.8262
> > CV trial 232 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6436 weighted AUC: 0.7660
> > CV trial 211 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 weighted AUC: 0.8262
> > CV trial 234 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 weighted AUC: 0.9326
> > CV trial 236 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.7943
> > CV trial 246 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720 weighted AUC: 0.7730
> > CV trial 213 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404 weighted AUC: 0.8050
> > CV trial 209 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 weighted AUC: 0.8316
> > CV trial 219 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 weighted AUC: 0.7305
> > CV trial 248 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5071 weighted AUC: 0.7092
> > CV trial 201 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613 weighted AUC: 0.7979
> > CV trial 67 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7163 weighted AUC: 0.8617
> > CV trial 228 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.8121
> > CV trial 9 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.8262
> > CV trial 217 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 weighted AUC: 0.8830
> > CV trial 215 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 weighted AUC: 0.9255
> > CV trial 244 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 weighted AUC: 0.8121
> > CV trial 175 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 weighted AUC: 0.8688
> > CV trial 231 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 weighted AUC: 0.7979
> > CV trial 227 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 weighted AUC: 0.8369
> > CV trial 241 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 weighted AUC: 0.8652
> > CV trial 211 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 weighted AUC: 0.8014
> > CV trial 205 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 weighted AUC: 0.7660
> > CV trial 235 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 weighted AUC: 0.8156
> > CV trial 191 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 weighted AUC: 0.8475
> > CV trial 249 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 weighted AUC: 0.6773
> > CV trial 233 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6206 weighted AUC: 0.7234
> > CV trial 225 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 weighted AUC: 0.7908
> > CV trial 247 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021 weighted AUC: 0.8865
> > CV trial 237 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 weighted AUC: 0.8652
> > CV trial 242 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 weighted AUC: 0.8333
> > CV trial 229 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 weighted AUC: 0.7872
> > CV trial 219 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904 weighted AUC: 0.7199
> > CV trial 221 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.8901
> > CV trial 238 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 weighted AUC: 0.7943
> > CV trial 245 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 weighted AUC: 0.8528
> > CV trial 222 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.8794
> > CV trial 215 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.9078
> > CV trial 249 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.6312
> > CV trial 231 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 weighted AUC: 0.7447
> > CV trial 235 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720 weighted AUC: 0.7943
> > CV trial 241 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 weighted AUC: 0.8582
> > CV trial 247 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915 weighted AUC: 0.8511
> > CV trial 229 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 weighted AUC: 0.7624
> > CV trial 233 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206 weighted AUC: 0.6915
> > CV trial 237 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 weighted AUC: 0.8546
> > CV trial 227 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 weighted AUC: 0.8262
> > CV trial 239 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 weighted AUC: 0.8901
> > CV trial 245 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404 weighted AUC: 0.8121
> > CV trial 243 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7996 weighted AUC: 0.8688
> > CV trial 222 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 weighted AUC: 0.8723
> > CV trial 239 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 weighted AUC: 0.8901
> > CV trial 192 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8316 weighted AUC: 0.8298
> > CV trial 197 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.9539
> > CV trial 197 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 weighted AUC: 0.9433
> > CV trial 223 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5709 weighted AUC: 0.7908
> > CV trial 243 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7890 weighted AUC: 0.8617
> > CV trial 193 feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 weighted AUC: 0.9255
> > CV trial 223 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4982 weighted AUC: 0.7943
> > CV trial 193 feature 1 mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 weighted AUC: 0.9397
>
If I launch the previous command line with only one 1-D feature as follows, everything is going well:
neuropredict -m meta_data.csv \
-d mask_WMpet_av45_early.ero1.25mm.csv \
-o outdir -t 0.8 -n 250 -k 'all' \
--gs_level 'exhaustive' --classifier "LinearSVC"
Do you have any idea?
Best,
Matthieu
Traceback (most recent call last):
File "/Users/John/anaconda3/bin/neuropredict", line 11, in
load_entry_point('neuropredict==0.2.4.5', 'console_scripts', 'neuropredict')()
File "/Users/John/anaconda3/lib/python3.5/site-packages/neuropredict/main.py", line 6, in main
neuropredict.run()
Often, performance is estimated in aggregate on the entire test-set (in a split), regardless of their covariate- or attribute characteristics.. hence looking into performance estimates segregated by certain characteristics of covariates would reveal additional insights - such as difference in AUC in men vs women, or across different age bins, or between sites etc.
Atleast parameters most often selected
For some reason, the latest version does not accept int or float for -k
Readthedocs.org is unreliable.
Need to set up GitHub pages and use the existing sphinx docs to autogenerate GitHub pages.
Hi @raamana ,
I would like to add LinearSVC classifier based on liblinear implementation. Does the current implementation of neuropredict need that predictions are based on probability values? Because, LinearSVC doesn't allow prediction of probabilities.
Have a separate markdown file
Separate file at the top level, one line with a label per feature
Traceback (most recent call last):
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/reports.py", line 73, in export_results
avg_cfmat, misclf_rate = visualize.compute_pairwise_misclf(confusion_matrix)
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/visualize.py", line 337, in compute_pairwise_misclf
avg_cfmat[dd, :, :] = mean_over_cv_trials(cfmat_array[:, :, :, dd], num_classes)
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/visualize.py", line 314, in mean_over_cv_trials
class_size_elementwise = np.transpose(np.matlib.repmat(np.sum(avg_cfmat, axis=1), num_classes, 1))
AttributeError: module 'numpy' has no attribute 'matlib'
Traceback (most recent call last):
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/reports.py", line 73, in export_results
avg_cfmat, misclf_rate = visualize.compute_pairwise_misclf(confusion_matrix)
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/visualize.py", line 337, in compute_pairwise_misclf
avg_cfmat[dd, :, :] = mean_over_cv_trials(cfmat_array[:, :, :, dd], num_classes)
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/visualize.py", line 314, in mean_over_cv_trials
class_size_elementwise = np.transpose(np.matlib.repmat(np.sum(avg_cfmat, axis=1), num_classes, 1))
AttributeError: module 'numpy' has no attribute 'matlib'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/global/home/hpc4089/.local/bin/neuropredict", line 11, in <module>
sys.exit(main())
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/__main__.py", line 11, in main
run_workflow.cli()
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/run_workflow.py", line 967, in cli
grid_search_level, classifier, feat_select_method)
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/run_workflow.py", line 942, in prepare_and_run
options_path=options_path)
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/rhst.py", line 405, in run
export_results(dict_to_save, out_results_dir, options_path)
File "/global/home/hpc4089/.local/lib/python3.5/site-packages/neuropredict/reports.py", line 108, in export_results
raise IOError('Unable to export the results to CSV files.')
OSError: Unable to export the results to CSV files.
/global/home/hpc4089/.local/lib/python3.5/site-packages/numpy/core/_methods.py:140: RuntimeWarning: Degrees of freedom <= 0 for slice
keepdims=keepdims)
/global/home/hpc4089/.local/lib/python3.5/site-packages/numpy/core/_methods.py:132: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
With clear warnings of trade off
Hi Pradeep,
I finally got neuropredict installed with the dependencies using python 2.7.13! I'm having some issues with running the command, see the error message below,
Cheers,
John
(py2713) Johns-MacBook-Pro-2:KNN John$ neuropredict -m /Users/John/Desktop/KNN/METADATAFILE.csv -o /Users/John/Desktop/KNN/features/results -u /Users/John/Desktop/KNN/features
Requested features for analysis:
get_dir_of_dirs from /Users/John/Desktop/KNN/features
Traceback (most recent call last):
File "/Users/John/anaconda3/envs/py2713/lib/python2.7/site-packages/neuropredict/neuropredict.py", line 333, in getfeatures
data, feat_names = getmethod(featdir, subjid)
File "/Users/John/anaconda3/envs/py2713/lib/python2.7/site-packages/neuropredict/neuropredict.py", line 274, in get_dir_of_dirs
raise IOError('Unable to load features from \n{}'.format(featfile))
Groups of features not significantly different from each other!
Focus on intuitive interpretability, even if it means generating more than one figure.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.