francescopisu / modelsight Goto Github PK
View Code? Open in Web Editor NEWBetter insights into Machine Learning models performance
License: GNU General Public License v3.0
Better insights into Machine Learning models performance
License: GNU General Public License v3.0
The AUC barplot embedded in the ROC plot is sized and positioned through the bars_pos
input parameter that accepts a tuple of four floats (xpos, ypos, width, height). See average_roc_curves
API reference.
However, the API could be clearer if four distinct input parameters were available for sizing and positioning.
Replace bars_pos
with four different parameters:
xpos
: position along the x-axis (bottom left corner is (0,0))ypos
: position along the y-axis (bottom left corner is (0,0))width
: width of embedded barplotheight
: height of embedded barplotThen, compose a tuple with these four elements and pass it to ax.inset_axes like so:
bars_pos = (xpos, ypos, width, height)
ins = ax.inset_axes(bars_pos)
N.A.
Trying to create a numpy array from a list of arrays with varying number of elements results in a ValueError.
See stack trace:
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (20,) + inhomogeneous part.
The following attributes of CVModellingOutput
should be lists instead:
N.A.
As of now (modelsight version 0.3.6), importing modelsight raises an error due to src folder not being recognized as a module.
>>> import modelsight
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/francescopisu/Workspace/Research/Projects/Pericarditis/prognosis/pyvenv/lib/python3.10/site-packages/modelsight/__init__.py", line 4, in <module>
from src.modelsight import calibration
ModuleNotFoundError: No module named 'src'
Get rid of the unnecessary src folder.
N.A.
It would be great if the average ROC plot showed statistical comparisons between AUCs of different models. The statistical test can be assumed to be the nonparametric DeLong test.
The average ROC plot should show average/median AUCs of the different models, (1-alpha)% confidence intervals and vertical whiskers for comparison between pairs of models.
N.A.
See image below.
The delong calculation breaks due to np.float
being a deprecated alias for the builtin float
.
See stack trace:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[60], line 28
15 kwargs = dict()
17 f, ax, barplot, bars, all_data = average_roc_curves(cv_results,
18 colors=palette,
19 model_keys=model_names,
(...)
25 ax=ax,
26 **kwargs)
---> 28 roc_comparisons_results = roc_comparisons(cv_results, "EBM")
30 kwargs = dict(space_between_whiskers = 0.07)
31 order = [
32 ("EBM", "RF"),
33 ("EBM", "SVC"),
34 ("EBM", "LR"),
35 ("EBM", "KNN")
36 ]
File ~/Workspace/Research/libraries/modelsight/src/modelsight/curves/compare.py:238, in roc_comparisons(cv_preds, target_algo)
236 for algo_name in cv_preds.keys():
237 if algo_name != target_algo:
--> 238 cmp = roc_single_comparison(cv_preds, target_algo, algo_name)
239 comparisons = dict(cmp, **comparisons)
241 return comparisons
File ~/Workspace/Research/libraries/modelsight/src/modelsight/curves/compare.py:229, in roc_single_comparison(cv_preds, fst_algo, snd_algo)
227 print("A"*100)
228 print(fst_algo_probas.shape, snd_algo_probas.shape)
--> 229 P = delong_roc_test(ground_truths, fst_algo_probas, snd_algo_probas)
230 cmp_key = f"{fst_algo}_{snd_algo}"
231 return {cmp_key: (fst_algo, snd_algo, P)}
File ~/Workspace/Research/libraries/modelsight/src/modelsight/curves/_delong.py:123, in delong_roc_test(ground_truth, predictions_one, predictions_two)
121 order, label_1_count = compute_ground_truth_statistics(ground_truth)
122 predictions_sorted_transposed = np.vstack((predictions_one, predictions_two))[:, order]
--> 123 aucs, delongcov = fastDeLong(predictions_sorted_transposed, label_1_count)
124 return 10**calc_pvalue(aucs, delongcov).item()
File ~/Workspace/Research/libraries/modelsight/src/modelsight/curves/_delong.py:61, in fastDeLong(predictions_sorted_transposed, label_1_count)
58 negative_examples = predictions_sorted_transposed[:, m:]
59 k = predictions_sorted_transposed.shape[0]
---> 61 tx = np.empty([k, m], dtype=np.float)
62 ty = np.empty([k, n], dtype=np.float)
63 tz = np.empty([k, m + n], dtype=np.float)
File ~/Workspace/Research/libraries/modelsight/envs/modelsight_env/lib/python3.10/site-packages/numpy/__init__.py:319, in __getattr__(attr)
314 warnings.warn(
315 f"In the future `np.{attr}` will be defined as the "
316 "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
318 if attr in __former_attrs__:
--> 319 raise AttributeError(__former_attrs__[attr])
321 if attr == 'testing':
322 import numpy.testing as testing
AttributeError: module 'numpy' has no attribute 'float'.
`np.float` was a deprecated alias for the builtin `float`. To avoid this error in existing code, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
We should replace any instance of np.float
with np.float64
.
N.A.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.