mozilla / fix-stacks

This program post-processes the stack frames produced by `MozFormatCodeAddress()`.

fix-stacks

This program post-processes ("fixes") the stack frames produced by MozFormatCodeAddress(), which often lack one or more of: function name, file name, line number. It relies on the symbolic and goblin crates to read debug info from files.

It reads from standard input and writes to standard output. Lines matching the special stack frame format are modified appropriately. For example, a line like this in the input:

#01: ???[tests/example +0x43a0]

is changed to something like this in the output:

#01: main (/home/njn/moz/fix-stacks/tests/example.c:24)

Lines that do not match the special stack frame format are passed through unchanged.

By default, fix-stacks uses native debug info present in binary files. In this case, because the stack frames produced by MozFormatCodeAddress() refer to build files (such as libxul), fix-stacks must run on the same machine that produced the stack frames and the build files. Furthermore, the build files must not have changed since the stack frames were produced. Otherwise, source locations in the output may be missing or incorrect.

Alternatively, you can use the -b option to tell fix-stacks to read Breakpad symbol files, as packaged by Firefox. The argument must contain two paths, separated by a comma: the first path points to the Breakpad symbols directory, the second path points to the fileid executable in the Firefox objdir. In this case, the processed output will contain square brackets instead of parentheses, to make it detectable from the output that breakpad symbols were used.

fix-stacks works on Linux, Windows, and Mac.

Shortcomings

On Linux, use with debuginfo sections in separate files is untested and probably does not work.

Android notes

In order to fix stacks in logcat output from an Android device, you need to tell fix-stacks how to map the remote file paths to files on the host machine, using the --local option.

The following command will check the file objdir/dist/bin/<fileName> on the host machine for any files which can't be found at the original path.

adb logcat | fix-stacks --local objdir/dist/bin

fix-stacks's Issues

test_mac fails when enabled

% cargo test
    Finished test [unoptimized + debuginfo] target(s) in 0.04s
     Running unittests src/main.rs (target/debug/deps/fix_stacks-a64e346390a79ade)

running 8 tests
test tests::test_regex ... ok
test tests::test_linux_breakpad ... ok
test tests::test_files ... ok
test tests::test_linux_breakpad_fallback ... ok
test tests::test_mac ... FAILED
test tests::test_linux ... ok
test tests::test_windows_breakpad ... ok
test tests::test_windows ... ok

failures:

---- tests::test_mac stdout ----
thread 'tests::test_mac' panicked at 'assertion failed: `(left == right)`
  left: `"#00: main (/Users/njn/moz/fix-stacks/tests/mac-normal.c:17)"`,
 right: `"#00: main (tests/mac-multi + 0xd70)"`', src/tests.rs:244:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::test_mac

test result: FAILED. 7 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.58s

error: test failed, to rerun pass '--bin fix-stacks'

Doesn't fix addresses for assembly functions?

I have a stack fixed by fix-stacks that contains these:

#04: ??? (/tmp/gecko/obj-x86_64-pc-linux-gnu/dist/bin/libxul.so + 0x68fa03e)
#19: ??? (/tmp/gecko/obj-x86_64-pc-linux-gnu/dist/bin/libxul.so + 0x68fa16b)

However, addr2line is able to resolve them:

$ addr2line -f -e /tmp/gecko/obj-x86_64-pc-linux-gnu/dist/bin/libxul.so 0x68fa03e
NS_InvokeByIndex
/tmp/gecko/xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_x86_64_unix.S:110
$ addr2line -f -e /tmp/gecko/obj-x86_64-pc-linux-gnu/dist/bin/libxul.so 0x68fa16b
SharedStub
xptcstubs_x86_64_linux.cpp:?

Remove dependence on `fileid`

#16 introduces Breakpad symbol support, but requires that a fileid executable from the Firefox build be provided. It's used to choose between symbol files when a binary has multiple symbol files. It would be better if fix-stacks could do fileid's job itself.

https://searchfox.org/mozilla-central/source/testing/tools/fileid has the code for fileid.

Get it working on Mac.

There are at least two challenges here.

• Fat binaries can have both 32-bit and 64-bit code in them. This requires using Symbolic's Archive type, which isn't hard.
• Debug info on Mac is normally (always?) stored separately, in a .dSYM folder.

Automatically fall back to using native symbols when breakpad symbols are unavailable or outdated

It is rather common on local machines to have leftover Breakpad symbols from old builds. Our tests harnesses has no way to tell if the breakpad symbols it finds are outdated or not so it will just instruct fix-stacks to use them which will fail if they are indeed outdated WRT the build. Given fix-stacks knows how to find native symbols automatically we should interpret the --breakpad option as a preference for Breakpad symbols but still fall back to native ones in case they're missing our outdated.

Move back to `getsentry:symbolic` when possible

We currently use a fork of symbolic, necessary for things to work on Mac. If/when my PR lands, we can go back to using the proper repo.

Add support for separate debug info on Linux

fix_linux_stack.py has code to read separate debug info files. This can give better stack frames involving system libraries if you have the appropriate debug info packages installed on your Linux machine. The code has a pointer to this useful documentation about separate debug info.

David Baron told me the following:

• Installing the debug info packages on Fedora involves changes some lines in /etc/yum.repos.d/*.conf from enabled=0 to enabled=1.
• For Ubuntu consult this or perhaps this; it's the -dbgsym packages that cover all the libraries.

Resolve android tombstone crash

This is Android's tombstone crash output.

05-07 04:17:57.046 14540 14555 F MOZ_Assert: Assertion failure: mSelection->Collapsed(), at /mozilla/gecko-mobile/widget/ContentCache.cpp:446
05-07 04:17:57.048 14540 14555 F MOZ_Assert: #01: ???[/data/app/org.mozilla.geckoview_example-1/lib/x86_64/libxul.so +0x77529d3]
05-07 04:17:57.048 14540 14555 F MOZ_Assert: #02: ???[/data/app/org.mozilla.geckoview_example-1/lib/x86_64/libxul.so +0x77519f3]
05-07 04:17:57.048 14540 14555 F MOZ_Assert: #03: ???[/data/app/org.mozilla.geckoview_example-1/lib/x86_64/libxul.so +0x77514d0]
05-07 04:17:57.048 14540 14555 F MOZ_Assert: #04: ???[/data/app/org.mozilla.geckoview_example-1/lib/x86_64/libxul.so +0x7764f4f]
05-07 04:17:57.049 14540 14555 F MOZ_Assert: #05: ???[/data/app/org.mozilla.geckoview_example-1/lib/x86_64/libxul.so +0x7769053]
05-07 04:17:57.049 14540 14555 F MOZ_Assert: #06: ???[/data/app/org.mozilla.geckoview_example-1/lib/x86_64/libxul.so +0x7748f47]
...

When using fix-stack, it is looking for host's /data/app/org.mozilla.geckoview_example-1/lib/x86_64/libxul.so, not remote's. So it cannot resolve symbols like the following.

fix-stacks: error: failed to read `/data/app/org.mozilla.geckoview_example-1/lib/x86_64/libxul.so`
fix-stacks: No such file or directory (os error 2)

So I would like something options to use remote file (or using adb to get remote *.so) for resolving symbols.

Dump inline frames

I'm not sure how easy this is or whether symbolic supports it but it would be nice to have.

As an aside: I was able to get inline frames working for my hacked together https://github.com/jrmuizel/pdb-addr2line but needed to work around getsentry/pdb#78

Wrong stack when hitting assertion on a linux64 debug build

I get this:

 4:16.44 GECKO(1292365) [Parent 1292446, Main Thread] ###!!! ASSERTION: Unexpected UpdateTransformLayer hint: '!(aChange & nsChangeHint_UpdateTransformLayer) || aFrame->IsTransformed() || aFrame->StyleDisplay()->HasTransformStyle()', file /home/emilio/src/moz/gecko-4/layout/base/RestyleManager.cpp, line 1168
 4:16.44 GECKO(1292365) #01: nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_CopyWithMemutils>::Length() const (/home/emilio/src/moz/gecko-4/obj-debug/dist/include/nsTArray.h:339)
 4:16.44 GECKO(1292365) #02: mozilla::PresShell::DoFlushPendingNotifications(mozilla::ChangesToFlush) (/home/emilio/src/moz/gecko-4/layout/base/PresShell.cpp:4080)
 4:16.44 GECKO(1292365) #03: ~RefPtr (/home/emilio/src/moz/gecko-4/obj-debug/dist/include/mozilla/RefPtr.h:80)
 4:16.44 GECKO(1292365) #04: nsINode::GetBoolFlag(nsINode::BooleanFlag) const (/home/emilio/src/moz/gecko-4/dom/base/nsINode.h:1704)
 4:16.45 GECKO(1292365) #05: mozilla::dom::Element::GetBoundingClientRect() (/home/emilio/src/moz/gecko-4/dom/base/Element.cpp:879)
 4:16.45 GECKO(1292365) #06: already_AddRefed<mozilla::dom::DOMRect>::take() (/home/emilio/src/moz/gecko-4/obj-debug/dist/include/mozilla/AlreadyAddRefed.h:146)
 4:16.45 GECKO(1292365) #07: bool mozilla::dom::binding_detail::GenericMethod<mozilla::dom::binding_detail::NormalThisPolicy, mozilla::dom::binding_detail::ThrowExceptions>(JSContext*, unsigned int, JS::Value*) (/home/emilio/src/moz/gecko-4/dom/bindings/BindingUtils.cpp:3217)
 4:16.45 GECKO(1292365) #08: CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), js::CallReason, JS::CallArgs const&) (/home/emilio/src/moz/gecko-4/js/src/vm/Interpreter.cpp:477)
 4:16.45 GECKO(1292365) #09: js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) (/home/emilio/src/moz/gecko-4/js/src/vm/Interpreter.cpp:569)
 4:16.45 GECKO(1292365) #10: Interpret(JSContext*, js::RunState&) (Interpreter.cpp:?)
 4:16.45 GECKO(1292365) #11: js::RunScript(JSContext*, js::RunState&) (/home/emilio/src/moz/gecko-4/js/src/vm/Interpreter.cpp:449)
 4:16.45 GECKO(1292365) #12: js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) (/home/emilio/src/moz/gecko-4/js/src/vm/Interpreter.cpp:604)
 4:16.45 GECKO(1292365) #13: js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>, js::CallReason) (/home/emilio/src/moz/gecko-4/js/src/vm/Interpreter.cpp:649)
 4:16.45 GECKO(1292365) #14: js::jit::InvokeFunction(JSContext*, JS::Handle<JSObject*>, bool, bool, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) (/home/emilio/src/moz/gecko-4/js/src/jit/VMFunctions.cpp:265)
 4:16.45 GECKO(1292365) #15: js::jit::InvokeFromInterpreterStub(JSContext*, js::jit::InterpreterStubExitFrameLayout*) (/home/emilio/src/moz/gecko-4/js/src/jit/VMFunctions.cpp:285)
 4:16.45 GECKO(1292365) #16: ??? (???:???)

Frames 1, 3, 4, and 5 are clearly wrong.

There are STR here for example (I don't think it's a very useful assertion: https://bugzilla.mozilla.org/show_bug.cgi?id=1332588).

Update to `symbolic` 7.0.0 once it's available

The code currently is using a pre-release version of 7.0.0. This is because the current release version (6.1.4) has an API botch: ObjectDebugSession::functions() is private.

Once 7.0.0 is out, we should update to use it.

Update symbolic dependencies to the latest version

Ideally we should keep it in line with dump_syms which is currently using symbolic version 8.

Offset non-leaf-frame addresses?

I don't know if this belongs in fix-stacks or in the code in Gecko that's printing the stacks, but getting the correct line info for non-leaf frames requires offsetting the return address to somewhere inside the preceding call instruction. Otherwise, the resolved line is for the code after the call, which might also be before it in the source due to optimizations; I'm currently having this problem with fix-stacks and a crashed Firefox debug build. Usually subtracting 1 is enough, but for ARM in Thumb mode it's necessary to subtract 2 (to clear the Thumb bit and then subtract 1).

See also bug 1642516.

Breakpad PUBLIC symbols not resolved

Incorrect breakpad stack fixing - only functions resolved, public symbols not resolved
FileInfo

impl FileInfo {
    fn new(debug_session: ObjectDebugSession) -> FileInfo {
        // Build the `FileInfo` from the debug session.
        let mut interner = Interner::default();
        let mut func_infos: Vec<_> = debug_session
            .functions()
            .filter_map(|function| {
                let function = function.ok()?;
                Some(FuncInfo::new(&mut interner, function, 0))
            })
            .collect();
        func_infos.sort_unstable_by_key(|func_info| func_info.address);
        func_infos.dedup_by_key(|func_info| func_info.address);

        FileInfo {
            interner,
            func_infos,
        }
    }

BreakpadDebugSession<'d>

impl<'d> BreakpadDebugSession<'d> {
    /// Returns an iterator over all functions in this debug file.
    pub fn functions(&self) -> BreakpadFunctionIterator<'_> {
        BreakpadFunctionIterator {
            file_map: &self.file_map,
            func_records: self.func_records.clone(),
        }
    }

Breakpad docs:
stack_walking.md
If no match was found in the function table, another table of publicly exported symbols may be consulted (PUBLIC lines from the symbol file). Public symbols contain only a start address, so the lookup simply looks for the nearest symbol that is less than the provided RVA.

Rust-minidump

if let Some(ref func) = self.functions.get(addr) {
            frame.set_function(&func.name, func.address + module.base_address());
            // See if there's source line info as well.
            func.lines.get(addr).map(|ref line| {
                self.files.get(&line.file).map(|ref file| {
                    frame.set_source_file(file, line.line, line.address + module.base_address());
                })
            });
        } else if let Some(ref public) = self.find_nearest_public(addr) {
            // Settle for a PUBLIC.
            frame.set_function(&public.name, public.address + module.base_address());
        }

/// Find the nearest `PublicSymbol` whose address is less than or equal to `addr`.
    pub fn find_nearest_public(&self, addr: u64) -> Option<&PublicSymbol> {
        for p in self.publics.iter().rev() {
            if p.address <= addr {
                return Some(p);
            }
        }

        return None;
    }

basic_source_line_resolver.cc

void BasicSourceLineResolver::Module::LookupAddress(StackFrame *frame) const {
  MemAddr address = frame->instruction - frame->module->base_address();

  // First, look for a FUNC record that covers address. Use
  // RetrieveNearestRange instead of RetrieveRange so that, if there
  // is no such function, we can use the next function to bound the
  // extent of the PUBLIC symbol we find, below. This does mean we
  // need to check that address indeed falls within the function we
  // find; do the range comparison in an overflow-friendly way.
  linked_ptr<Function> func;
  linked_ptr<PublicSymbol> public_symbol;
  MemAddr function_base;
  MemAddr function_size;
  MemAddr public_address;
  if (functions_.RetrieveNearestRange(address, &func, &function_base,
                                      NULL /* delta */, &function_size) &&
      address >= function_base && address - function_base < function_size) {
    frame->function_name = func->name;
    frame->function_base = frame->module->base_address() + function_base;

    linked_ptr<Line> line;
    MemAddr line_base;
    if (func->lines.RetrieveRange(address, &line, &line_base, NULL /* delta */,
                                  NULL /* size */)) {
      FileMap::const_iterator it = files_.find(line->source_file_id);
      if (it != files_.end()) {
        frame->source_file_name = files_.find(line->source_file_id)->second;
      }
      frame->source_line = line->line;
      frame->source_line_base = frame->module->base_address() + line_base;
    }
  } else if (public_symbols_.Retrieve(address,
                                      &public_symbol, &public_address) &&
             (!func.get() || public_address > function_base)) {
    frame->function_name = public_symbol->name;
    frame->function_base = frame->module->base_address() + public_address;
  }
}

Get it working on Windows.

In theory, this should be straightforward because Symbolic supports PDB.