mgedmin / objgraph

Visually explore Python object graphs

Python Object Graphs

objgraph is a module that lets you visually explore Python object graphs.

You'll need graphviz if you want to draw the pretty graphs.

I recommend xdot for interactive use. pip install xdot should suffice; objgraph will automatically look for it in your PATH.

Installation and Documentation

pip install objgraph or download it from PyPI.

Documentation lives at https://mg.pov.lt/objgraph.

History

I've developed a set of functions that eventually became objgraph when I was hunting for memory leaks in a Python program. The whole story -- with illustrated examples -- is in this series of blog posts:

• Hunting memory leaks in Python
• Python object graphs
• Object graphs with graphviz

Support and Development

The source code can be found in this Git repository: https://github.com/mgedmin/objgraph.

To check it out, use git clone https://github.com/mgedmin/objgraph.

Report bugs at https://github.com/mgedmin/objgraph/issues.

objgraph's Issues

Graph output raises a syntax error in Dot

I'm not entirely sure if this is an error in Objgraph or (X)Dot but I'll take my chances.

I'm trying to run down a leak caused by a C Extension that I'm refactoring. I was following the Reference Counting Bugs section of the quickstart guide when I had the "brilliant" idea to just graph everything for . . . well, stupid reasons.

(Please put your torch and pitchfork down.)

Dot didn't like the syntax output. I encountered the following message:

Graph written to /tmp/objgraph-t8r7rn7w.dot (3250 nodes)
Warning: /tmp/objgraph-t8r7rn7w.dot: syntax error in line 5857 near ','
Image generated as leaks_bulkwalk_py3.png

The image was not generated, however I managed to save the output Dot file with the code I used. Line 5857 contains the following: o21209296[label="str [1]\n'map(func, *iterables) --> map object\\n\\"]; I'm not familiar with Dot but, after reading a few sections of the file, I guess it either mis-parsed it as a function or perhaps commas are illegal in strings?

How do show_growth deltas behave when memory runs out?

I'm profiling to find a leak in an API endpoint that runs a complex process. I have objgraph.show_growth report at two or three points inside the endpoint's main method.

Hitting the endpoint over and over, I quickly hit a MemoryError, and for a while large positive deltas are reported, but finally the deltas stop being reported at all.

I guess conceivably this could mean one of two things:

1. There is no positive delta to report: memory is saturated with objects involved and no new ones are created.
2. Objects are being created, but show_growth can't function properly in the given memory, so it doesn't report them.

I don't really understand this library well enough to see which one is more likley, but can you comment as to which one you think it is?

Thanks.

Why are new dictionaries not detected by show_growth()?

I am trying basically:

>>> import objgraph as o
>>> o.show_growth()
...
>>> d = {1: 2}
>>> o.show_growth()
>>>

I have learned from the documentation that references to primitive types are not tracked, but I wonder why the reference to a dictionary (which is not an of primitive type?) is not counted? For example, if I define a list with only primitive element, it is tracked:

>>> l = [2]
>>> o.show_growth()
list      356        +1

but the same with a dictionary does not work?

objgraph.show_growth() doesn't notice numpy arrays (and other non-GC-tracked types)

See this code:

import objgraph
import numpy as np
objgraph.show_growth()
j = 20
y = []
for i in range(5):
    for l in range(j):
        y.append(np.array([np.random.randint(500),np.random.randint(500)]))
    print 'i:',i
    objgraph.show_growth()
    print '___'
    #objgraph.show_most_common_types(limit=100)
    j += 1

the result is:

function                       2973     +2973
wrapper_descriptor             1584     +1584
builtin_function_or_method      873      +873
dict                            867      +867
method_descriptor               823      +823
weakref                         622      +622
tuple                           518      +518
getset_descriptor               514      +514
list                            422      +422
member_descriptor               223      +223
i: 0
wrapper_descriptor     1593        +9
getset_descriptor       518        +4
member_descriptor       226        +3
list                    424        +2
weakref                 624        +2
dict                    869        +2
listiterator              1        +1
___
i: 1
wrapper_descriptor     1596        +3
weakref                 625        +1
dict                    870        +1
method_descriptor       824        +1
___
i: 2
___
i: 3
___
i: 4
___

for the 2,3 and 4 epoch, it shows nothing growing. But it should show that the number of numpy.array grows

Allow saving graph dot to in memory string instead of file

Would it be possible to support outputting the generated graph dot text to file objects also?

This would allow rendering in IPython notebooks without needing to write and track temporary files by e.g. writing to a StringIO object.

(Internally pygraphviz handles all this with STDIN/STDOUT)

Currently you need to do this (which admittedly isn't too bad...):

import pygraphviz
import IPython.display
import objgraph

x = [[1,2,3], 4,5,6]
objgraph.show_refs(x, filename='temp.dot')
IPython.display.Image(pygraphviz.AGraph(filename='temp.dot').draw(format='png', prog='dot'))

Rendering to a PNG file below is shorter, but I think the dot example above demonstrates the general purpose usefulness of being able to capture that version of the graph.

import pygraphviz
import IPython.display
import objgraph

x = [[1,2,3], 4,5,6]
objgraph.show_refs(x, filename='temp.png')
IPython.display.Image('temp.png')

Multiple test failures with PyPy3.10 7.3.15

When running the test suite with PyPy3.10, I'm getting a bunch of test failures:

tox output

$ tox -e pypy3
.pkg: install_requires> python -I -m pip install 'setuptools>=40.8.0' wheel
.pkg: _optional_hooks> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: install_requires_for_build_wheel> python -I -m pip install wheel
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
pypy3: install_package_deps> python -I -m pip install graphviz
pypy3: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/objgraph/.tox/.tmp/package/1/objgraph-3.5.1.dev0.tar.gz
pypy3: commands[0]> python tests.py
..F........................s..........FF.s.......FF^[[5;7~FF./tmp/objgraph/objgraph.py:966: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='utf-8'>
  gc.collect()
.F/tmp/objgraph/objgraph.py:438: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='utf-8'>
  gc.collect()
F.
======================================================================
FAIL: test_new_garbage (__main__.ByTypeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/objgraph/tests.py", line 412, in test_new_garbage
    self.assertLessEqual(len(gc.get_referrers(res[0])), 2)
AssertionError: 3 not less than or equal to 2

======================================================================
FAIL: test_short_repr_mocked_instance_method_bound (__main__.StringRepresentationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/objgraph/tests.py", line 474, in test_short_repr_mocked_instance_method_bound
    self.assertRegex(objgraph._short_repr(obj.my_method), '<Mock')
AssertionError: Regex didn't match: '<Mock' not found in '<bound method ? of <__main__.StringRepre'

======================================================================
FAIL: test_short_repr_mocked_instance_method_bound_with_mocked_name (__main__.StringRepresentationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/objgraph/tests.py", line 497, in test_short_repr_mocked_instance_method_bound_with_mocked_name
    self.assertRegex(objgraph._short_repr(obj.my_method), '<Mock')
AssertionError: Regex didn't match: '<Mock' not found in '<bound method ? of <__main__.StringRepre'

======================================================================
FAIL: /tmp/objgraph/docs/extra-info.txt
Doctest: extra-info.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/doctest.py", line 2223, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for extra-info.txt
  File "/tmp/objgraph/docs/extra-info.txt", line 0

----------------------------------------------------------------------
File "/tmp/objgraph/docs/extra-info.txt", line 29, in extra-info.txt
Failed example:
    objgraph.at(id(a))
Expected nothing
Got:
    'a string'


======================================================================
FAIL: /tmp/objgraph/docs/generator-sample.txt
Doctest: generator-sample.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/doctest.py", line 2223, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for generator-sample.txt
  File "/tmp/objgraph/docs/generator-sample.txt", line 0

----------------------------------------------------------------------
File "/tmp/objgraph/docs/generator-sample.txt", line 43, in generator-sample.txt
Failed example:
    objgraph.show_chain(
        objgraph.find_backref_chain(objgraph.by_type('Canary')[0],
                                    objgraph.is_proper_module),
        filename='canary-chain.png')
Expected:
    Graph written to ....dot (11 nodes)
    Image generated as canary-chain.png
Got:
    Graph written to /tmp/test-objgraph-nwf3s8tc/objgraph-rtr5hst2.dot (9 nodes)
    Image generated as canary-chain.png


======================================================================
FAIL: /tmp/objgraph/docs/highlighting.txt
Doctest: highlighting.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/doctest.py", line 2223, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for highlighting.txt
  File "/tmp/objgraph/docs/highlighting.txt", line 0

----------------------------------------------------------------------
File "/tmp/objgraph/docs/highlighting.txt", line 16, in highlighting.txt
Failed example:
    objgraph.show_backrefs(a, max_depth=15,
        extra_ignore=[id(locals())],
        highlight=lambda x: isinstance(x, Node),
        filename='highlight.png')
Expected:
    Graph written to ....dot (12 nodes)
    Image generated as highlight.png
Got:
    Graph written to /tmp/test-objgraph-0qqpwuhn/objgraph-wjse_9g6.dot (100 nodes)
    Image generated as highlight.png


======================================================================
FAIL: /tmp/objgraph/docs/index.txt
Doctest: index.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/doctest.py", line 2223, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for index.txt
  File "/tmp/objgraph/docs/index.txt", line 0

----------------------------------------------------------------------
File "/tmp/objgraph/docs/index.txt", line 110, in index.txt
Failed example:
    objgraph.show_chain(
        objgraph.find_backref_chain(
            random.choice(objgraph.by_type('MyBigFatObject')),
            objgraph.is_proper_module),
        filename='chain.png')
Expected:
    Graph written to ...dot (13 nodes)
    Image generated as chain.png
Got:
    Graph written to /tmp/test-objgraph-3z1llwbv/objgraph-jk5ezpur.dot (10 nodes)
    Image generated as chain.png
----------------------------------------------------------------------
File "/tmp/objgraph/docs/index.txt", line 158, in index.txt
Failed example:
    objgraph.show_refs(roots[:3], refcounts=True, filename='roots.png')
    # doctest: +NODES_VARY
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/pypy3.10/doctest.py", line 1352, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest index.txt[16]>", line 1, in <module>
        objgraph.show_refs(roots[:3], refcounts=True, filename='roots.png')
      File "/tmp/objgraph/objgraph.py", line 805, in show_refs
        return _show_graph(objs, max_depth=max_depth, extra_ignore=extra_ignore,
      File "/tmp/objgraph/objgraph.py", line 976, in _show_graph
        _obj_label(target, extra_info,
      File "/tmp/objgraph/objgraph.py", line 1118, in _obj_label
        label[0] += ' [%d]' % (sys.getrefcount(obj) - 4)
    AttributeError: module 'sys' has no attribute 'getrefcount'


======================================================================
FAIL: /tmp/objgraph/docs/references.txt
Doctest: references.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/doctest.py", line 2223, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for references.txt
  File "/tmp/objgraph/docs/references.txt", line 0

----------------------------------------------------------------------
File "/tmp/objgraph/docs/references.txt", line 7, in references.txt
Failed example:
    objgraph.show_refs([list(range(7))], too_many=5, filename='too-many.png')
Expected:
    Graph written to ....dot (6 nodes)
    Image generated as too-many.png
Got:
    Graph written to /tmp/test-objgraph-3mjnayr1/objgraph-9o1bm5qw.dot (1 nodes)
    Image generated as too-many.png
----------------------------------------------------------------------
File "/tmp/objgraph/docs/references.txt", line 17, in references.txt
Failed example:
    objgraph.show_backrefs([moo], too_many=5, max_depth=1, filename='42.png')
Expected:
    Graph written to ....dot (6 nodes)
    Image generated as 42.png
Got:
    Graph written to /tmp/test-objgraph-3mjnayr1/objgraph-sp9hh556.dot (2 nodes)
    Image generated as 42.png
----------------------------------------------------------------------
File "/tmp/objgraph/docs/references.txt", line 34, in references.txt
Failed example:
    objgraph.show_backrefs([one_reference], refcounts=True,
        filename='refcounts.png') # doctest: +NODES_VARY
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/pypy3.10/doctest.py", line 1352, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest references.txt[7]>", line 1, in <module>
        objgraph.show_backrefs([one_reference], refcounts=True,
      File "/tmp/objgraph/objgraph.py", line 725, in show_backrefs
        return _show_graph(objs, max_depth=max_depth, extra_ignore=extra_ignore,
      File "/tmp/objgraph/objgraph.py", line 976, in _show_graph
        _obj_label(target, extra_info,
      File "/tmp/objgraph/objgraph.py", line 1118, in _obj_label
        label[0] += ' [%d]' % (sys.getrefcount(obj) - 4)
    AttributeError: module 'sys' has no attribute 'getrefcount'


======================================================================
FAIL: doctest_get_new_ids_prints (__main__)
Doctest: __main__.doctest_get_new_ids_prints
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/doctest.py", line 2223, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for __main__.doctest_get_new_ids_prints
  File "/tmp/objgraph/tests.py", line 376, in doctest_get_new_ids_prints

----------------------------------------------------------------------
File "/tmp/objgraph/tests.py", line 383, in __main__.doctest_get_new_ids_prints
Failed example:
    _ = objgraph.get_new_ids(limit=1)
    # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
Expected:
    ========================================================
    Type      Old_ids  Current_ids      New_ids Count_Deltas
    ========================================================
    list          ...          ...          ...           +2
    ========================================================
Got:
    =======================================================
    Type      Old_ids  Current_ids      New_ids Count_Deltas
    =======================================================
    int        15089        16956        +1867        +1867
    =======================================================


----------------------------------------------------------------------
Ran 58 tests in 88.947s

FAILED (failures=9, skipped=2)
pypy3: exit 1 (89.41 seconds) /tmp/objgraph> python tests.py pid=37859
.pkg: _exit> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  pypy3: FAIL code 1 (100.12=setup[10.71]+cmd[89.41] seconds)
  evaluation failed :( (100.22 seconds)

At least some of them seem to be caused by relying on aggressive CPython GC behavior.

RFE: is it possible to start making github releases?🤔

Is it possible next time on release new version make the github release to have entry on https://github.com/mgedmin/objgraph/releases? 🤔

I'm asking because only on make gh release is spread notification about new release to those who have set watch->releases.
My automation process those notification trying make preliminary automated upgrade of building packages which allow save some time on maintaining packaging procedures.

More about gh releases is possible to find on
https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository
https://github.com/marketplace/actions/github-release

output is graphviz.files.Source object

In the document it says

>>> x = []
>>> y = [x, [x], dict(x=x)]
>>> import objgraph
>>> objgraph.show_refs([y], filename='sample-graph.png')
Graph written to ....dot (... nodes)
Image generated as sample-graph.png

But I get this instead, there is no image file

Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 22:20:52) [MSC v.1916 32 bit (Intel)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.2 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.10.2
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 22:20:52) [MSC v.1916 32 bit (Intel)] on win32
x = []
y = [x, [x], dict(x=x)]
import objgraph
objgraph.show_refs([y], filename='sample-graph.png')
Out[5]: <graphviz.files.Source at 0x58aa170>

graphviz is installed

pip3 install graphviz
Requirement already satisfied: graphviz in d:\program files\python\python37-32\lib\site-packages (0.13.2)

Issue with using objgraph in jupyter console

If I use objgraph in an IPython session things work as expected:

In [1]: import objgraph

In [2]: a = 1

In [3]: objgraph.show_backrefs(objgraph.by_type('int'))
Graph written to /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/objgraph-5iuhcjwd.dot (0 nodes)
Graph viewer (xdot) not found, generating a png instead
Image generated as /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/objgraph-5iuhcjwd.png

However if I use jupyter qtconsole there is no output from show_backrefs, and if I try and specify an output filename, nothing happens:

In [1]: import objgraph

In [2]: a = 1

In [3]: objgraph.show_backrefs(objgraph.by_type('int'))

what could be going on?

Convenience wrappers for show_backref_chain(obj, inspect.ismodule)

Using objgraph.show_chain(objgraph.find_backref_chain(...)) is a lot of typing. Provide convenience wrappers.

3.5.0: documentation build fails

Looks like something is wrong with conf.py

+ /usr/bin/sphinx-build -n -T -b man docs build/sphinx/man
Running Sphinx v6.2.1

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 353, in eval_config_file
    code = compile(f.read(), filename.encode(fs_encoding), 'exec')
  File "/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/docs/conf.py", line 27
    exec open(relative('../objgraph.py')).read() in d
         ^
SyntaxError: invalid syntax

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/usr/lib/python3.8/site-packages/sphinx/application.py", line 207, in __init__
    self.config = Config.read(self.confdir, confoverrides or {}, self.tags)
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 177, in read
    namespace = eval_config_file(filename, tags)
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 357, in eval_config_file
    raise ConfigError(msg % err) from err
sphinx.errors.ConfigError: There is a syntax error in your configuration file: invalid syntax (conf.py, line 27)

API redesign

This is a long-term wishlist idea.

The API of objgraph grew organically during manual debugging session. As a result there are plenty of ad-hoc function arguments that need to be passed from function to function. Also, common tasks like "find me a chain of references from a module to this particular object, then display it" need to be spelled in cumbersome ways.

It'd be nice to come up with a better API.

Add unittest coverage

Right now, many of the tests are essentially unittests but are written using the doctests. If these were made unittests it would be easier to add additional coverage.

Graphs say "instance" instead of "DocTest"

I was about to release version 2.0.0 and ran make clean docs and git diff docs/*.png, then saw some unexpected changes. For example, here's canary-chain.png in current git master:

and here's what I get with the code in current git master:

(different font rendering is a result of a different graphviz version, I think)

Blocking given objects from further expansion.

Can we please have a direly missing option to block given objects from further expansion?
For example, I want to have very deep expansion of my object but I don't want to expose any low level details of foreign objects, e.g. pandas dataframe. Unless I'm very much mistaken, I couldn't see an option for this.
Thank you.

allow disabling automatic full collection when computing growth

I recently experienced a manifestation of https://bugs.python.org/issue39061 which I had worked around by running a full collection (gc.collect()) at regular intervals. I started using objgraph to try to find the source of the "leak" (the reference cycle) which moved and accumulated objects in the generation 2 collection which also made the issue disappear, just as my regular garbage collection was. I thus assumed that objgraph was calling gc.collect() and indeed it was.

I commented out the call to gc.collect() in growth which allowed me to identify the reference cycle which I could fix using weakrefs.

I would like to propose a new option to the growth/show_growth functions to disable the automatic full collection. Maybe running a generation 1 collection or none at all even if it means that there would be more false positive showing up in the growth report.

This is a proposition and I will gladly do the change if it sounds like it would be useful.

By the way, objgraph has helped me more than once at this point. Thank you very much.

Graphs not being saved to pngs

Using the example from the docs:

>>> x = []
>>> y = [x, [x], dict(x=x)]
>>> import objgraph
>>> objgraph.show_refs([y], filename="x.png")
Graph written to C:\Users\Userame\AppData\Local\Temp\objgraph-99xk348b.dot (4 nodes)
Image renderer (dot) not found, not doing anything else

Despite having graphviz and xdot installed.

How to perform before and after diff between leaking objects

Hi @mgedmin,

Referring to https://mg.pov.lt/objgraph/objgraph.html#objgraph.get_leaking_objects, I've discovered that get_leaking_objects API has already returned leaking objects through import of external library not sourcing from my code.

My thought is to ignore these externally leaked object but focus on any new leaked object my code has introduced. Hence I'm thinking to call get_leaking_objects twice: once before my code runs and once after my code runs.

As you've mentioned in the documentation, get_leaking_objects could report thousands of leaked object. I would like to check with you how do I perform a diff between before and after get_leaking_objects and then only dump leaking objects in the diff?

Consolidate graph creation

Right now there are multiple places that have very similar functionality. For example show_graph and find_chain both search for a chain from the supplied object to a predicate (in find_chain the predicate is user specified, in show_graph it is a module). This results in some weird cases where to show a set of chains, find_chain is called to collect all nodes that are involved in the chain, then show_graph calculates all chains again and just uses the already built chains as a filter.

Instead I imagine this could be consolidated into something more like:

find_graph(...) and find_graph(..., shortest_path=True) where the logic is more or less the same but shortest_path just has an early end condition. Then find_chain is obsoleted by find_graph(..., shortest_path=True) and show_chain is obsoleted by show_graph which only has to contain the logic for creating the DOT format.

Just want to say Aciū (thanks)!

Can't get to work on Ipython

I'm sorry if this seems like a dumb question, but I'm running Jupyter, Python 2.7, Windows, and Ipython notebook. I have installed graphviz and xdot.

from StringIO import StringIO
import objgraph

x = [[1,2,3], 4,5,6]
objgraph.show_refs([x], output=StringIO())

I don't see anything when I run this.

objgraph should not merge distinct classes with same name

Forwarded from LaunchPad #1183768:

Classes in different may have same class name, like

module.a.A
module.b.A

But typestats merges these objects into one entry key "A", making the result of show_growth confusing.

I think it's better to use the full qualified name of a class as key is better,
{
"module.a.A": 12,
"module.b.A": 3
}

Tests fail under Python 3.11

Three tests fail under Python 3.11.0rc2+:

======================================================================
FAIL: /home/mg/src/objgraph/docs/generator-sample.txt
Doctest: generator-sample.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mg/opt/python311/lib/python3.11/doctest.py", line 2221, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for generator-sample.txt
  File "/home/mg/src/objgraph/docs/generator-sample.txt", line 0

----------------------------------------------------------------------
File "/home/mg/src/objgraph/docs/generator-sample.txt", line 43, in generator-sample.txt
Failed example:
    objgraph.show_chain(
        objgraph.find_backref_chain(objgraph.by_type('Canary')[0],
                                    objgraph.is_proper_module),
        filename='canary-chain.png')
Expected:
    Graph written to ....dot (11 nodes)
    Image generated as canary-chain.png
Got:
    Graph written to /tmp/test-objgraph-_jyc3mp_/objgraph-ah0l_c0o.dot (9 nodes)
    Image generated as canary-chain.png


======================================================================
FAIL: /home/mg/src/objgraph/docs/highlighting.txt
Doctest: highlighting.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mg/opt/python311/lib/python3.11/doctest.py", line 2221, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for highlighting.txt
  File "/home/mg/src/objgraph/docs/highlighting.txt", line 0

----------------------------------------------------------------------
File "/home/mg/src/objgraph/docs/highlighting.txt", line 16, in highlighting.txt
Failed example:
    objgraph.show_backrefs(a, max_depth=15,
        extra_ignore=[id(locals())],
        highlight=lambda x: isinstance(x, Node),
        filename='highlight.png')
Expected:
    Graph written to ....dot (12 nodes)
    Image generated as highlight.png
Got:
    Graph written to /tmp/test-objgraph-9ljbdx5m/objgraph-xu76dngb.dot (8 nodes)
    Image generated as highlight.png


======================================================================
FAIL: /home/mg/src/objgraph/docs/index.txt
Doctest: index.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mg/opt/python311/lib/python3.11/doctest.py", line 2221, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for index.txt
  File "/home/mg/src/objgraph/docs/index.txt", line 0

----------------------------------------------------------------------
File "/home/mg/src/objgraph/docs/index.txt", line 110, in index.txt
Failed example:
    objgraph.show_chain(
        objgraph.find_backref_chain(
            random.choice(objgraph.by_type('MyBigFatObject')),
            objgraph.is_proper_module),
        filename='chain.png')
Expected:
    Graph written to ...dot (13 nodes)
    Image generated as chain.png
Got:
    Graph written to /tmp/test-objgraph-c9315zw4/objgraph-7oqkrzee.dot (12 nodes)
    Image generated as chain.png


----------------------------------------------------------------------
Ran 58 tests in 2.392s

FAILED (failures=3, skipped=2)

Document the meaning of colours in the graph

When calling objgraph.show_refs, some nodes can have their text coloured red. It's unclear what the meaning of this is. The documentation should specify this.

show_growth prints incorrect results if peak_stats not passed

the problem

show_growth() relies on the strange behavior of taking a mutable as the default value of an arg. That leads to a bug where incorrect results are returned. At least they are incorrect to my understanding. Could be my understanding is wrong though :)

See repro below.

objgraph version: 3.4.1

repro:

import objgraph


class Something:
    def __init__(self, a, b, c):
        self.a = a
        self.b = b
        self.c = c


def main():
    works()
    broken()


def works():
    # if pass peak_stats it works better:
    print('\n------------this works-------------')
    peak_stats = {}
    objgraph.show_growth(shortnames=False, peak_stats=peak_stats)
    peak_stats = objgraph.typestats(shortnames=False)

    x = [Something(i, i ** 2, str(i)) for i in range(1000)]
    print(f'\nshould have 1000 somethings now. first one is {x[0]}')
    objgraph.show_growth(shortnames=False, peak_stats=peak_stats)
    peak_stats = objgraph.typestats(shortnames=False)

    del x
    print(f'\nsomethings should be gone now:')
    objgraph.show_growth(shortnames=False, peak_stats=peak_stats)
    peak_stats = objgraph.typestats(shortnames=False)

    x = [Something(i, i ** 2, str(i)) for i in range(1000)]
    print(f'\nshould have 1000 somethings again, and we do. first one is {x[0]}')
    objgraph.show_growth(shortnames=False, peak_stats=peak_stats)
    peak_stats = objgraph.typestats(shortnames=False)


def broken():
    print('\n------------this is broken-------------')
    objgraph.show_growth(shortnames=False)
    x = [Something(i, i ** 2, str(i)) for i in range(1000)]
    print(f'\nshould have 1000 somethings now. first one is {x[0]}')
    objgraph.show_growth(shortnames=False)

    del x
    print(f'\nsomethings should be gone now:')
    objgraph.show_growth(shortnames=False)

    x = [Something(i, i ** 2, str(i)) for i in range(1000)]
    print(f'\nshould have 1000 somethings again, but we do not. first one is {x[0]}')
    #
    # the bug is provoked on this call.
    #
    # it should print a line with "__main__.Something     1000     +1000"
    #
    # instead it prints nothing. I think this is due to relying on the strange behavior
    # of specifying a mutable as a default value for an arg, in this case the peak_stats arg of show_growth()
    #
    objgraph.show_growth(shortnames=False)
    print(x[0])


if __name__ == "__main__":
    main()

which produces something like this:

------------this works-------------
builtins.function                       2261     +2261
builtins.tuple                          1885     +1885
builtins.dict                           1425     +1425
builtins.wrapper_descriptor              998      +998
builtins.weakref                         894      +894
builtins.set                             805      +805
builtins.method_descriptor               732      +732
builtins.builtin_function_or_method      702      +702
builtins.list                            551      +551
builtins.getset_descriptor               408      +408

should have 1000 somethings now. first one is <__main__.Something object at 0x7fdcef2726a0>
__main__.Something     1000     +1000
builtins.frame            7        +2
builtins.list           552        +1
builtins.cell            42        +1

somethings should be gone now:
builtins.frame        7        +2
builtins.cell        42        +1

should have 1000 somethings again, and we do. first one is <__main__.Something object at 0x7fdcee6986a0>
__main__.Something     1000     +1000
builtins.frame            7        +2
builtins.list           552        +1
builtins.cell            42        +1

------------this is broken-------------
builtins.function                       2261     +2261
builtins.tuple                          1864     +1864
builtins.dict                           1425     +1425
builtins.wrapper_descriptor              998      +998
builtins.weakref                         894      +894
builtins.set                             805      +805
builtins.method_descriptor               732      +732
builtins.builtin_function_or_method      702      +702
builtins.list                            551      +551
builtins.getset_descriptor               408      +408

should have 1000 somethings now. first one is <__main__.Something object at 0x7fdcedfabbe0>
__main__.Something     1000     +1000
builtins.list           552        +1

somethings should be gone now:

should have 1000 somethings again, but we do not. first one is <__main__.Something object at 0x7fdcedf3b8d0>
<__main__.Something object at 0x7fdcedf3b8d0>

3.5.0: sphinx warnings `reference target not found`

On building my packages I'm using sphinx-build command with -n switch which shows warmings about missing references. These are not critical issues.
Here is the output with warnings:

+ /usr/bin/sphinx-build -n -T -b man docs build/sphinx/man
Running Sphinx v6.2.1
making output directory... done
WARNING: The config value `release' has type `list', defaults to `str'.
building [mo]: targets for 0 po files that are out of date
writing output...
building [man]: all manpages
updating environment: [new config] 11 added, 0 changed, 0 removed
reading sources... [100%] uncollectable
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
writing... python-objgraph.3 { objgraph references extra-info highlighting uncollectable generator-sample chain quoting CHANGES HACKING } /home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/docs/index.txt:100: WARNING: py:class reference target not found: MyBigFatObject
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/docs/index.txt:122: WARNING: py:mod reference target not found: linecache
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/docs/index.txt:122: WARNING: py:mod reference target not found: doctest
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/docs/index.txt:122: WARNING: py:func reference target not found: computate_something
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/objgraph.py:docstring of objgraph.show_chain:14: WARNING: py:func reference target not found: gc.get_referrers
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/objgraph.py:docstring of objgraph.show_chain:14: WARNING: py:func reference target not found: gc.get_referents
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: find_chain
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: show_graph
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: obj_node_id
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: obj_label
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: quote
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: long_typename
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: safe_repr
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: short_repr
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: gradient
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: edge_label
/home/tkloczko/rpmbuild/BUILD/objgraph-3.5.0/CHANGES.rst:130: WARNING: py:func reference target not found: _program_in_path
done
build succeeded, 18 warnings.

You can peak on fixes that kind of issues in other projects
RDFLib/rdflib-sqlalchemy#95
RDFLib/rdflib#2036
click-contrib/sphinx-click@abc31069
frostming/unearth#14
jaraco/cssutils#21
latchset/jwcrypto#289
latchset/jwcrypto#289
pypa/distlib@98b9b89f
pywbem/pywbem#2895
sissaschool/elementpath@bf869d9e
sissaschool/xmlschema@42ea98f2
sqlalchemy/sqlalchemy@5e88e6e8

objgraph.by_type('file', untracked=True)

So, not all objects are tracked by the Python garbage collector, and therefore not all show up in gc.get_objects(). For example: file objecs don't, so objgraph.by_type('file') will always return an empty list.

Implement a workaround that does

    return [r for o in objects for r in gc.get_referents(o) if type(r).__name__ == typename]

if the user asks for it.

Objgraph is not Generating Images - GraphViz is

This issue is probably with my installation in a VirtualEnv on OS X 10.6.8.
ObjGraph 1.8.1 doesn't seem to be generating image files, although it states that it has.
GraphViz is working as expected when run from the command line:

dot -Tpng graph1.dot > output.png

Move objgraph.py to objgraph/__init__.py

In light of #39 I would like to start a refactoring: The first step is to move objgraph.py to objgraph/__init__.py. This will establish a code directory that is separate from all the deployment resources found in the main directory, and it will be where we add further (graph) modules, and maybe even break up the main code file.

Highlight memory leak cycles

It would be nice to highlight the cycles that are responsible for memory leaks. It would also be nice to highlight objects that are solely preventing large amounts of memory to be gc'ed. I am thinking of a dominator analysis

incorrect labes for dict keys

import objgraph

objgraph.show_refs({"a": None, "b": None}, filename="bad.png")

output:

`test_edge_label_frame_locals` fails on Python 3.13

When running the test suite against Python 3.13.0b2, I'm getting the following failure:

$ tox -e py313
.pkg: _optional_hooks> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py313: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/objgraph/.tox/.tmp/package/3/objgraph-3.6.2.dev0.tar.gz
py313: commands[0]> python tests.py
..........................F.s..............s....ss.....s....
======================================================================
FAIL: test_edge_label_frame_locals (__main__.StringRepresentationTest.test_edge_label_frame_locals)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/objgraph/tests.py", line 470, in test_edge_label_frame_locals
    self.assertEqual(' [label="f_locals",weight=10]',
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     objgraph._edge_label(frame, frame.f_locals))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: ' [label="f_locals",weight=10]' != ''
-  [label="f_locals",weight=10]


----------------------------------------------------------------------
Ran 60 tests in 1.750s

FAILED (failures=1, skipped=5)
py313: exit 1 (1.88 seconds) /tmp/objgraph> python tests.py pid=139558
  py313: FAIL code 1 (7.76=setup[5.89]+cmd[1.88] seconds)
  evaluation failed :( (7.88 seconds)

Image renderer (dot) not found, not doing anything else

Mac Info

MacBook Pro (14-inch):
Chip:  Apple M1 Max
Memory:  64 GB
Starup disk:  Macintosh HD
MacOS: Ventura 13.3.1 (a)

Python Info

$ python -V
Python 3.11.3

Install dependency packages

$ pip install objgraph xdot graphviz

Using objgraph

$ python
Python 3.11.3 (main, Apr  6 2023, 20:15:36) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> x = []
>>> y = [x, [x], dict(x=x)]
>>> import objgraph
>>> objgraph.show_refs([y], filename='sample-graph.png')
Graph written to /var/folders/kh/vtm21r5s3bz53z809m5z21d80000gn/T/objgraph-na8jmm62.dot (4 nodes)
Image renderer (dot) not found, not doing anything else

>>> import objgraph
>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> a.append(b)
>>> b.append(a)
>>> objgraph.show_refs([a])
Graph written to /var/folders/kh/xxxxxxgn/T/objgraph-na8xxxxxxxx.dot (4 nodes)
Image renderer (dot) not found, not doing anything else

Question

Did I miss anything? Why can't the graphics be displayed even though all the required packages have been installed?

Is it possible to inspect objects from a list

Hi, show_growth can show created lists since last call. Is it possible for objgraph to show the specific objects inside a list?

Objgraph segfaults when printing data about leaking C objects

traceback from gdb: (show_most_common_types was called)

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff7b5949f in PyObject_Malloc () from /usr/lib/libpython3.7m.so.1.0
(gdb) bt
#0  0x00007ffff7b5949f in PyObject_Malloc () from /usr/lib/libpython3.7m.so.1.0
#1  0x00007ffff7b80ba5 in _PyLong_New () from /usr/lib/libpython3.7m.so.1.0
#2  0x00007ffff7b8132b in PyLong_FromLong () from /usr/lib/libpython3.7m.so.1.0
#3  0x00007ffff7b6d88d in PyNumber_Add () from /usr/lib/libpython3.7m.so.1.0
#4  0x00007ffff7bf365e in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#5  0x00007ffff7ba1d18 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#6  0x00007ffff7ba2da3 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#7  0x00007ffff7bb5c30 in ?? () from /usr/lib/libpython3.7m.so.1.0
#8  0x00007ffff7bf3b96 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#9  0x00007ffff7ba1d18 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#10 0x00007ffff7ba2da3 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#11 0x00007ffff7bb5c30 in ?? () from /usr/lib/libpython3.7m.so.1.0
#12 0x00007ffff7bf3b96 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#13 0x00007ffff7ba1d18 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#14 0x00007ffff7ba2da3 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#15 0x00007ffff7bb5c30 in ?? () from /usr/lib/libpython3.7m.so.1.0
#16 0x00007ffff7bf3b96 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#17 0x00007ffff7ba1d18 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#18 0x00007ffff7ba2aca in PyEval_EvalCodeEx () from /usr/lib/libpython3.7m.so.1.0
#19 0x00007ffff7ba2aec in PyEval_EvalCode () from /usr/lib/libpython3.7m.so.1.0
#20 0x00007ffff7c738a5 in ?? () from /usr/lib/libpython3.7m.so.1.0
#21 0x00007ffff7c73c2b in PyRun_FileExFlags () from /usr/lib/libpython3.7m.so.1.0
#22 0x00007ffff7c79ec7 in PyRun_SimpleFileExFlags () from /usr/lib/libpython3.7m.so.1.0
#23 0x00007ffff7c7c022 in ?? () from /usr/lib/libpython3.7m.so.1.0
#24 0x00007ffff7c7c1cc in _Py_UnixMain () from /usr/lib/libpython3.7m.so.1.0
#25 0x00007ffff7de5ee3 in __libc_start_main () from /usr/lib/libc.so.6
#26 0x000055555555505e in _start ()

For context, it seems about 1.7 GB worth of memory is leaking, so it's likely objgraph is unable to keep up with such heaps of data and tries to allocate more than is available.

function `show_growth` return ``deltas``

I am using function show_growth to find memory leak in my project. Howerver, at the moment, the result outputs to stdout or file.I just wonder if we can return the result, with which we can automaticly detech memory leak when regression testing.
To achieve the target, I have two suggestion. First, simple but ungraceful, just return deltas in function show_growth. Second, adding a function get_growth, which return the increasing deltas, and the function show_growth just output the increasing deltas