Coder Social home page Coder Social logo

modelsight's People

Contributors

francescopisu avatar

Stargazers

 avatar  avatar

Watchers

 avatar

modelsight's Issues

Simplify AUC barplot sizing and positioning within ROC plot

Is your proposal related to a problem?

The AUC barplot embedded in the ROC plot is sized and positioned through the bars_pos input parameter that accepts a tuple of four floats (xpos, ypos, width, height). See average_roc_curves API reference.
However, the API could be clearer if four distinct input parameters were available for sizing and positioning.

Describe the solution you'd like

Replace bars_pos with four different parameters:

  • xpos: position along the x-axis (bottom left corner is (0,0))
  • ypos: position along the y-axis (bottom left corner is (0,0))
  • width: width of embedded barplot
  • height: height of embedded barplot

Then, compose a tuple with these four elements and pass it to ax.inset_axes like so:

bars_pos = (xpos, ypos, width, height)
ins = ax.inset_axes(bars_pos)

Describe alternatives you've considered

N.A.

Additional context

Lists of arrays with varying numbers of elements should not be converted into numpy arrays

Is your proposal related to a problem?

Trying to create a numpy array from a list of arrays with varying number of elements results in a ValueError.

See stack trace:

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (20,) + inhomogeneous part.

Describe the solution you'd like

The following attributes of CVModellingOutput should be lists instead:

  • gts_train
  • gts_val
  • probas_train
  • probas_val
  • errors
  • correct
  • features

Describe alternatives you've considered

N.A.

Additional context

ModuleNotFoundError: No module named 'src' when importing modelsight

Is your proposal related to a problem?

As of now (modelsight version 0.3.6), importing modelsight raises an error due to src folder not being recognized as a module.

>>> import modelsight
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/francescopisu/Workspace/Research/Projects/Pericarditis/prognosis/pyvenv/lib/python3.10/site-packages/modelsight/__init__.py", line 4, in <module>
    from src.modelsight import calibration
ModuleNotFoundError: No module named 'src'

Describe the solution you'd like

Get rid of the unnecessary src folder.

Describe alternatives you've considered

N.A.

Additional context

Allow statistical comparisons between AUCs of different models

Is your proposal related to a problem?

It would be great if the average ROC plot showed statistical comparisons between AUCs of different models. The statistical test can be assumed to be the nonparametric DeLong test.

Describe the solution you'd like

The average ROC plot should show average/median AUCs of the different models, (1-alpha)% confidence intervals and vertical whiskers for comparison between pairs of models.

Describe alternatives you've considered

N.A.

Additional context

See image below.

Screenshot 2023-09-07 alle 20 35 13

AUC comparison fails when computing DeLong test

Is your proposal related to a problem?

The delong calculation breaks due to np.float being a deprecated alias for the builtin float.

See stack trace:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[60], line 28
     15 kwargs = dict()
     17 f, ax, barplot, bars, all_data = average_roc_curves(cv_results,
     18                                                     colors=palette,
     19                                                     model_keys=model_names,
   (...)
     25                                                     ax=ax,
     26                                                     **kwargs)
---> 28 roc_comparisons_results = roc_comparisons(cv_results, "EBM")
     30 kwargs = dict(space_between_whiskers = 0.07)
     31 order = [
     32     ("EBM", "RF"),
     33     ("EBM", "SVC"),
     34     ("EBM", "LR"),
     35     ("EBM", "KNN")
     36 ]

File ~/Workspace/Research/libraries/modelsight/src/modelsight/curves/compare.py:238, in roc_comparisons(cv_preds, target_algo)
    236 for algo_name in cv_preds.keys():
    237     if algo_name != target_algo:
--> 238         cmp = roc_single_comparison(cv_preds, target_algo, algo_name)
    239         comparisons = dict(cmp, **comparisons)
    241 return comparisons

File ~/Workspace/Research/libraries/modelsight/src/modelsight/curves/compare.py:229, in roc_single_comparison(cv_preds, fst_algo, snd_algo)
    227 print("A"*100)
    228 print(fst_algo_probas.shape, snd_algo_probas.shape)
--> 229 P = delong_roc_test(ground_truths, fst_algo_probas, snd_algo_probas)
    230 cmp_key = f"{fst_algo}_{snd_algo}"
    231 return {cmp_key: (fst_algo, snd_algo, P)}

File ~/Workspace/Research/libraries/modelsight/src/modelsight/curves/_delong.py:123, in delong_roc_test(ground_truth, predictions_one, predictions_two)
    121 order, label_1_count = compute_ground_truth_statistics(ground_truth)
    122 predictions_sorted_transposed = np.vstack((predictions_one, predictions_two))[:, order]
--> 123 aucs, delongcov = fastDeLong(predictions_sorted_transposed, label_1_count)
    124 return 10**calc_pvalue(aucs, delongcov).item()

File ~/Workspace/Research/libraries/modelsight/src/modelsight/curves/_delong.py:61, in fastDeLong(predictions_sorted_transposed, label_1_count)
     58 negative_examples = predictions_sorted_transposed[:, m:]
     59 k = predictions_sorted_transposed.shape[0]
---> 61 tx = np.empty([k, m], dtype=np.float)
     62 ty = np.empty([k, n], dtype=np.float)
     63 tz = np.empty([k, m + n], dtype=np.float)

File ~/Workspace/Research/libraries/modelsight/envs/modelsight_env/lib/python3.10/site-packages/numpy/__init__.py:319, in __getattr__(attr)
    314     warnings.warn(
    315         f"In the future `np.{attr}` will be defined as the "
    316         "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
    318 if attr in __former_attrs__:
--> 319     raise AttributeError(__former_attrs__[attr])
    321 if attr == 'testing':
    322     import numpy.testing as testing

AttributeError: module 'numpy' has no attribute 'float'.
`np.float` was a deprecated alias for the builtin `float`. To avoid this error in existing code, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

Describe the solution you'd like

We should replace any instance of np.float with np.float64.

Describe alternatives you've considered

N.A.

Additional context

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.