ianozsvald / ipython_memory_usage Goto Github PK

IPython tool to report memory usage deltas for every command you type

License: MIT License

Python 28.10% Batchfile 0.07% Shell 0.04% Jupyter Notebook 71.78%

ipython_memory_usage's Issues

Ambiguous output when calling start_watching_memory consecutively

When calling start_watching_memory consecutively unclear output is printed:

In [1]: import ipython_memory_usage.ipython_memory_usage as imu                                                
In [2]: import numpy as np
                                                                                  
In [3]: imu.start_watching_memory()                                                                            
In [3] used 17.2734 MiB RAM in 174.15s, peaked 0.00 MiB above current, total RAM usage 68.12 MiB

In [4]: imu.start_watching_memory()                                                                            
In [4] used 0.0977 MiB RAM in 0.10s, peaked 0.00 MiB above current, total RAM usage 68.21 MiB
In [4] used -0.0117 MiB RAM in 0.21s, peaked 0.01 MiB above current, total RAM usage 68.20 MiB

It's unclear what the output in the last call refers to.

Confirm if perf experimental code works

can we confirm if perf still works? Ian can't install it on his 4.9 kernel

As noted in the README.md we have %run -i ipython_memory_usage_perf.py but Ian can't run perf on his kernel - can someone please check if this code still works?

notes for conda for the future

Add memory_profiler to dependency

We need a conda dependency on memory_profiler and IPython to mirror the setup.py

Please package the LICENSE file

This would help with making self-contained conda-forge packages.

report if python (not ipython) used for the import

Re-raise if the import is done outside of the ipython environment to give the user a clue. With python this lacks the appropriate attributes:

$ python
Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ipython_memory_usage
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ian/workspace/personal_projects/hackathon_ipython_memory_usage/ipython_memory_usage/src/ipython_memory_usage/__init__.py", line 5, in <module>
    import ipython_memory_usage.ipython_memory_usage as imu
  File "/home/ian/workspace/personal_projects/hackathon_ipython_memory_usage/ipython_memory_usage/src/ipython_memory_usage/ipython_memory_usage.py", line 19, in <module>
    input_cells = get_ipython().user_ns['In']
AttributeError: 'NoneType' object has no attribute 'user_ns'

Using IPython is good:

$ ipython
Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.9.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ipython_memory_usage                                                                                                                 
In [2]:

Collect pandas (and other) weird memory usage cases

pandas-dev/pandas#11050
pandas-dev/pandas#17092
numpy/numpy#6732
https://github.com/liushuchun/pynote/tree/master/slots_magic (use case)
https://github.com/jni/streaming-talk (use case)
https://github.com/ianozsvald/ipython_memory_usage/blob/master/src/ipython_memory_usage/examples/example_usage_np_pd.ipynb (my example for pandas del vs drop via Jamie Brunning)

Auto start imu when `%load_ext ipython_memory_usage` is called

Rather than print that the user should run %start_imu, instead kick it off and tell them how to stop it

Way to retrieve the memory delta, peak and usage

It would be nice to retrieve the memory delta, peak and usage values after executing a command in IPython. After looking at the code, it seems that these are measured inside a thread, making accessing them a bit difficult.

Tasks for AHL hackathon

Does the README still work for Python 3.7?
Look at #13 to refactor the code
Build a conda recipe - test account first and then upload to conda-forge
Can we change the default behaviour from "import ipython_memory_usage.ipython_memory_usage as imu; imu.start_watching_memory()" to "import ipython_memory_usage; ipython_memory_usage.start_watching_memory()"?
Is it possible to turn "start_watching_memory()" into an IPython magic? "%start_watching_memory"?
Check this works with Python 3.8 and the latest numpy, is the numpy behaviour still the same as we have in the README or does it need an update?

Error while import in XPython kernel (debugger tool in JupyterLab)

When trying to debug notebook with XPython kernel (used for debugging in JupyterLab) import was fail:

import ipython_memory_usage
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
In  [8]:
Line 3:     import ipython_memory_usage

File /home/user/anaconda3/lib/python3.8/site-packages/ipython_memory_usage/__init__.py, in <module>:
Line 5:     import ipython_memory_usage.ipython_memory_usage as imu

File /home/user/anaconda3/lib/python3.8/site-packages/ipython_memory_usage/ipython_memory_usage.py, in <module>:
Line 19:    input_cells = get_ipython().user_ns['In']

KeyError: 'In'

update installer

Try switching to support wheels https://pythonwheels.com/ and move the subfolder up 1 level

Upload to PyPI

Thanks for publishing this useful package! Any interest in uploading it to PyPI so it's easier for users to discover and install?

Consider adding visual sparkline of cpu and memory usage

Rather than showing the maximal cpu usage, instead consider making a graphical representation if <17 cores (just 'cos that'd fit ok on screen).

Consider also making a time-series sparkline memory plot, so we can see if anything weird happens with that maximal usage during the cell's execution.

Example has errors

fix numpy example bug (below), fix the doc
check a = np.sqrt(a) example, the behaviour seems to have changed, fix the doc
confirm d = a * b + c behaviour has changed too, fix the doc

This looks like a change in numpy behaviour:

In [4]: a = np.ones(1e7)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-a702330132ac> in <module>()
----> 1 a = np.ones(1e7)

~/anaconda3/envs/ipython_memory_usage/lib/python3.6/site-packages/numpy/core/numeric.py in ones(shape, dtype, order)
    186 
    187     """
--> 188     a = empty(shape, dtype, order)
    189     multiarray.copyto(a, 1, casting='unsafe')
    190     return a

TypeError: 'float' object cannot be interpreted as an integer
In [4] used 0.6836 MiB RAM in 0.17s, peaked 0.00 MiB above current, total RAM usage 55.93 MiB

In [5]: a = np.ones(int(1e7))
In [5] used 76.1758 MiB RAM in 0.12s, peaked 0.00 MiB above current, total RAM usage 132.11 MiB

check docs for autotime

Ian to check https://github.com/cpcloud/ipython-autotime

give a clue to user on first install

Add a docstring to the package and add %ipython_memory_usage_start to give the user a clue.

Update init.py with a """docstring that tells the user how to get started""" so that when they do:

import ipython_memory_usage 
ipython_memory_usage?

they get a couple of lines telling them to start with %ipython_memory_usage_start or %ipython_memory_usage_stop to show memory changes on each typed command

Refactor the main file to drop Python 2 support

https://github.com/ianozsvald/ipython_memory_usage/blob/master/src/ipython_memory_usage/ipython_memory_usage.py was written in Python 2 several years ago. You'll see from __future__ imports - these could do with being removed!

The functions lack useful 1-liner docstrings, these could be added.

There's a __version__ number in the file, that ought to be updated, maybe to v 1.1?

Double check that the README is still valid after you've refactored

ianozsvald / ipython_memory_usage Goto Github PK

ipython_memory_usage's People

Contributors

Stargazers

Watchers

Forkers

ipython_memory_usage's Issues

Recommend Projects

Recommend Topics

Recommend Org