Possible race condition with psutil about memory_profiler

[brainstorm]

That clearly biases the measurement since the information is lost (together with @guillermo-carrasco we saw some strange numbers after this tweak).

I guess the next step would be reproducing the error (in tests?), which might prove tricky. @fabianp, we are having this issue while we run this particular test:

https://github.com/SciLifeLab/facs/blob/master/tests/test_fastqscreen.py#L112

If you see something clearly wrong in the way we call it, please, let us know.

[brainstorm]

@guillermo-carrasco I guess you meant OSError as the exception, since:

  File "/home/roman/.pyenv/versions/2.7.6/lib/python2.7/site-packages/memory_profiler.py", line 53, in _get_memory
    except NoSuchProcess:
NameError: global name 'NoSuchProcess' is not defined

[guillermo-carrasco]

Not really, I meant NoSuchProcess, you have to import it from psutil

[kblin]

This just happened to me on a profiling run. Seeing how the uncaught exception causes the profiling to stop completely, I'd prefer to just miss out on the short-running job's memory usage.

[fabianp]

I pushed a fix in a9e5a02 . Feel free to reopen/send a pull request if it doesn't solve the issue.

[kblin]

Didn't try this yet, but it looks like that won't work. I'm clearly getting a psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=1363), not an OSError. But I can try a similar patch.

[fabianp]

It would indeed be great if you can send a pull request for that

…

On Tue, Jan 10, 2017 at 3:10 PM, Kai Blin ***@***.***> wrote: Didn't try this yet, but it looks like that won't work. I'm clearly getting a psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=1363), not an OSError. But I can try a similar patch. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/fabianp/memory_profiler/issues/71#issuecomment-271584350>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQ8hwdisHB0f2IK6Sajep0HT9qZuhW4ks5rQ5E6gaJpZM4Bfo5e> .

I ran into a similar problem on macOS, which I assume is also caused by this issue. I'm trying to track a process which creates tens or hundreds of children which live for a very short time, and occasionally I get the following error:

Traceback (most recent call last):
  File "/Users/sasa/work/proj/env/bin/mprof", line 554, in <module>
    actions[get_action()]()
  File "/Users/sasa/work/proj/env/bin/mprof", line 244, in run_action
    multiprocess=options.multiprocess, stream=f)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/memory_profiler.py", line 358, in memory_usage
    for idx, chldmem in enumerate(_get_child_memory(proc.pid)):
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/memory_profiler.py", line 115, in _get_child_memory
    yield getattr(child, meminfo_attr)()[0] / _TWO_20
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_common.py", line 336, in wrapper
    return fun(self)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/__init__.py", line 1119, in memory_info
    return self._proc.memory_info()
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_psosx.py", line 295, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_psosx.py", line 421, in memory_info
    rawtuple = self._get_pidtaskinfo()
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_common.py", line 336, in wrapper
    return fun(self)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_psosx.py", line 352, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_psosx.py", line 326, in catch_zombie
    raise AccessDenied(proc.pid, proc._name)
psutil.AccessDenied: psutil.AccessDenied (pid=43916)

I'll send a PR to ignore psutil.AccessDenied in addition to psutil.NoSuchProcess shortly.