shopify / heap-profiler

Ruby heap profiler

HeapProfiler

A memory profiler for Ruby

Requirements

Ruby(MRI) Version 2.5 and above.

Installation

Add this line to your application's Gemfile:

gem 'heap-profiler'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install heap-profiler

Usage

Profiling Mode

HeapProfiler can be used to measure memory allocations and retentions of a Ruby code snippet.

To record a profile:

require 'heap-profiler'
HeapProfiler.report('path/to/report/directory') do
  # You code here
end

To then analyse the profile, run the heap-profiler command against the directory you specified. Note that on large applications this can take a while, but if you are profiling a production application, you can download the profile directory and do the analysis on another machine.

Options

Usage: heap-profiler <directory_or_heap_dump> OPTIONS

OPTIONS

    -r, --retained-only              Only compute report for memory retentions.
    -m, --max=NUM                    Max number of entries to output. (Defaults to 50)
        --batch-size SIZE            Sets the simdjson parser batch size. It must be larger than the largest JSON document in the heap dump, and defaults to 10MB.

$ heap-profiler path/to/report/directory
      Total allocated: 3.72 kB (36 objects)
      Total retained: 808.00 B (12 objects)

      allocated memory by gem
      -----------------------------------
         3.72 kB  other

      allocated memory by file
      -----------------------------------
         3.72 kB  bin/generate-report

      allocated memory by location
      -----------------------------------
         3.17 kB  bin/generate-report:34
        157.00 B  bin/generate-report:28
         80.00 B  bin/generate-report:21
         72.00 B  bin/generate-report:26
         40.00 B  bin/generate-report:31
         40.00 B  bin/generate-report:30
         40.00 B  bin/generate-report:25
         40.00 B  bin/generate-report:24
         40.00 B  bin/generate-report:23
         40.00 B  bin/generate-report:22

      allocated memory by class
      -----------------------------------
         1.18 kB  Class
        848.00 B  <iseq> (IMEMO)
        597.00 B  String
        384.00 B  <ment> (IMEMO)
        200.00 B  Array
        192.00 B  Hash
         80.00 B  <ifunc> (IMEMO)
         80.00 B  <cref> (IMEMO)
         72.00 B  Date
         40.00 B  Symbol
         40.00 B  SomeCustomStuff

      allocated objects by gem
      -----------------------------------
              36  other

      allocated objects by file
      -----------------------------------
              36  bin/generate-report

      allocated objects by location
      -----------------------------------
              27  bin/generate-report:34
               1  bin/generate-report:31
               1  bin/generate-report:30
               1  bin/generate-report:28
               1  bin/generate-report:26
               1  bin/generate-report:25
               1  bin/generate-report:24
               1  bin/generate-report:23
               1  bin/generate-report:22
               1  bin/generate-report:21

      allocated objects by class
      -----------------------------------
              12  String
               8  <ment> (IMEMO)
               4  Array
               2  Class
               2  <iseq> (IMEMO)
               2  <ifunc> (IMEMO)
               2  <cref> (IMEMO)
               1  Symbol
               1  SomeCustomStuff
               1  Hash
               1  Date

      retained memory by gem
      -----------------------------------
        808.00 B  other

      retained memory by file
      -----------------------------------
        808.00 B  bin/generate-report

      retained memory by location
      -----------------------------------
        168.00 B  bin/generate-report:30
        168.00 B  bin/generate-report:28
        160.00 B  bin/generate-report:34
         80.00 B  bin/generate-report:21
         72.00 B  bin/generate-report:26
         40.00 B  bin/generate-report:25
         40.00 B  bin/generate-report:24
         40.00 B  bin/generate-report:23
         40.00 B  bin/generate-report:22

      retained memory by class
      -----------------------------------
        336.00 B  Hash
        240.00 B  String
         80.00 B  Array
         72.00 B  Date
         40.00 B  Symbol
         40.00 B  SomeCustomStuff

      retained objects by gem
      -----------------------------------
              12  other

      retained objects by file
      -----------------------------------
              12  bin/generate-report

      retained objects by location
      -----------------------------------
               4  bin/generate-report:34
               1  bin/generate-report:30
               1  bin/generate-report:28
               1  bin/generate-report:26
               1  bin/generate-report:25
               1  bin/generate-report:24
               1  bin/generate-report:23
               1  bin/generate-report:22
               1  bin/generate-report:21

      retained objects by class
      -----------------------------------
               6  String
               2  Hash
               1  Symbol
               1  SomeCustomStuff
               1  Date
               1  Array

      Allocated String Report
      -----------------------------------
         80.00 B       2  "foo="
                       2  bin/generate-report:34

         80.00 B       2  "foo"
                       2  bin/generate-report:34

         80.00 B       2  "bar="
                       2  bin/generate-report:34

         80.00 B       2  "I am retained"
                       1  bin/generate-report:23
                       1  bin/generate-report:22

         40.00 B       1  "I am retained too"
                       1  bin/generate-report:24

         40.00 B       1  "I am allocated too"
                       1  bin/generate-report:31

         40.00 B       1  "I am allocated"
                       1  bin/generate-report:30

        157.00 B       1  "I am a very very long string I am a very very long string I am a very very long string I am a very very long string "
                       1  bin/generate-report:28


      Retained String Report
      -----------------------------------
         80.00 B       2  "I am retained"
                       1  bin/generate-report:23
                       1  bin/generate-report:22

         40.00 B       1  "foo="
                       1  bin/generate-report:34

         40.00 B       1  "foo"
                       1  bin/generate-report:34

         40.00 B       1  "bar="
                       1  bin/generate-report:34

         40.00 B       1  "I am retained too"
                       1  bin/generate-report:24

Heap Analysis

Alternatively if you with to analyse the entire heap of your Ruby process.

If you can, you should enable allocation tracing as early as possible during your application boot process, e.g. in config/boot.rb for Rails apps.

require 'objspace'
ObjectSpace.trace_object_allocations_start

Then to dump the heap:

require 'objspace'
ObjectSpace.dump_all(output: File.open('path/to/file.heap', 'w+'))

Then run heap-profiler against it:

heap-profiler path/to/file.heap

How is it different from memory_profiler?

heap-profiler is heavilly inspired of memory_profiler, it aims at being as similar as possible. However it uses a different Ruby API to gather data.

memory_profiler uses ObjectSpace.each_object which contrary to what its name suggest doesn't expose all existing object. There are many objects that the Ruby VM consider "internal" (see MRI's internal_object_p(VALUE)) and won't yield to each_object.

On the other hand heap-profiler uses ObjectSpace.dump_all, which does serialize every objects, including internal ones, into JSON files. This leads to more exhaustive reports.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/Shopify/heap-profiler.

Thanks

This gem was heavilly inspired from http://github.com/SamSaffron/memory_profiler, it even borrowed some code from it, so thanks to @SamSaffron.

It also makes heavy use of https://github.com/simdjson/simdjson for fast heap dump parsing. So big thanks to Daniel Lemire and John Keiser.

License

The gem is available as open source under the terms of the MIT License.

heap-profiler's Issues

Better handle corrupted dumps

ObjectSpace.dump sometimes has bugs (e.g. ruby/ruby#6161) and can cause some lines of the dump to be invalid JSON.

Ideally we should just skip these lines and continue. I need to check if it's possible with simdjson.

Alternatively we can document this in the readme and offer a cleanup script to do this.

cc @benoittgt

exc: The JSON element does not have the requested type. (HeapProfiler::Error)

Hi 👋 , when parsing a heap dump created with ObjectSpace.dump_all I'm getting the following error:

.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/parser.rb:51:in `_build_index': exc: The JSON element does not have the requested type. (HeapProfiler::Error)
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/parser.rb:51:in `build_index'
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/parser.rb:63:in `build_index'
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/index.rb:14:in `build!'
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/index.rb:10:in `initialize'
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/results.rb:79:in `new'
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/results.rb:79:in `pretty_print'
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/cli.rb:47:in `print_report'
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/lib/heap_profiler/cli.rb:23:in `run'
	from .asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/heap-profiler-0.7.0/exe/heap-profiler:5:in `<top (required)>'
	from .asdf/installs/ruby/3.2.1/bin/heap-profiler:25:in `load'
	from .asdf/installs/ruby/3.2.1/bin/heap-profiler:25:in `<main>'

The heap dump is about 600MB in size and it worked fine on another project with a smaller heap dump