eliben / pyelftools

Parsing ELF and DWARF in Python

pyelftools

pyelftools is a pure-Python library for parsing and analyzing ELF files and DWARF debugging information. See the User's guide for more details.

Pre-requisites

As a user of pyelftools, one only needs Python 3 to run. While there is no reason for the library to not work on earlier versions of Python, our CI tests are based on the official Status of Python versions.

Installing

pyelftools can be installed from PyPI (Python package index):

> pip install pyelftools

Alternatively, you can download the source distribution for the most recent and historic versions from the Downloads tab on the pyelftools project page (by going to Tags). Then, you can install from source, as usual:

> python setup.py install

Since pyelftools is a work in progress, it's recommended to have the most recent version of the code. This can be done by downloading the master zip file or just cloning the Git repository.

Since pyelftools has no external dependencies, it's also easy to use it without installing, by locally adjusting PYTHONPATH.

How to use it?

pyelftools is a regular Python library: you import and invoke it from your own code. For a detailed usage guide and links to examples, please consult the user's guide.

Contributing

See the Hacking Guide.

License

pyelftools is open source software. Its code is in the public domain. See the LICENSE file for more details.

pyelftools's Issues

Dwarf 4 support

Pyelftools does not support dwarf 4 and fails parsing when it encounters some of the new types (e.g. DW_FORM_exprloc). Even if full support is not provided, pyelftool should be able to skip the abbrev table it does not understand.

Many Linux distributions are starting to use DWARF4 with their toolchains now.

UnicodeDecodeError with python 3.4.3 on OS X

$ python
Python 3.4.3 (default, Feb 25 2015, 21:28:45) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from elftools.elf.elffile import ELFFile
>>> e = ELFFile(open("some_elf_file"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/site-packages/elftools/elf/elffile.py", line 50, in __init__
    self._identify_file()
  File "/usr/local/lib/python3.4/site-packages/elftools/elf/elffile.py", line 200, in _identify_file
    magic = self.stream.read(4)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/codecs.py", line 319, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 32-33: invalid continuation byte

There's no such an error with python 2.7.10.

A minor issue related to 0 section headers

Hi, Dear Eliben
First thanks a lot to your excellent tool, I had been using it since 2012. -- It helps a lot on my reverse engineering works.
Here is the issue I met:
When analyzing an elf file with 0 section headers contained, pyelftool reports an error of "UnboundLocalError: local variable 'stringtable' referenced before assignment".

Tracking back in call stack I found in dynamic.py, stringtable is NOT initialized in init function for the sake of "Zero" section.
As the segment is a Dynamic one, elftool tries to look up a corresponding "dynamic" section for more relevant data. -- but it fails anyway.

Secion information may be removed from most of proprietary elf files delivered to out-side world. Would you kindly consider to add some lines as a tolerance to this case?
e.g. a section amount check and a dummy assignment to stringtable before iterating section in init function.
def init(self, header, stream, elffile):
if elffile.num_sections() == 0:
stringtable = "dummy"

Thanks a lot.
And best regards to your work :)

Billy

The construct module is out of date!!! Problem will raise under the envirmonent python3.3

I try to write back some data into elf file use the pyelftools like that:
from elftools.elf.elffile import ELFFile
from elftools.elf.strucs import ELFStructs
elf=ELFFile(open('an elf file','rb'))
struct=ELFStructs()
struct.Elf_Ehdr.build(elf.header)
my python is in version 3.3 and it return a TypeError
and then I replace the construct module with the latest construct module and it going fun.

Easy way to get the binary image

Hi,

Is there an easy way to get the binary image from an ELF file to be programmed into a microcontroller flash? I currently have a bootloader written in python based on IntelHex to send an intel hex file to an ARM Cortex-M microcontroller. But I would like to extend it to support ELF files. Is there something like this available or do I have to start writing it myself?

I would have asked this on a forum or mailing list if I could find one, but I only found this issue tracker. My apologies if I'm using the wrong channel for asking questions.

High level functionalities

There is no high level functions to play with the DWARF symbols. You need to do everything from the original structures which is not always easy. Is this a personnal choice not to provide any or do you accept some high level pull request to enhance this ? (For example, utils to link structure declarations in code to easily access their member's values)

Problems rewinding to the beginning of a DebugSectionDescriptor stream

The DWARF parsing code will occasionally try to rewind to the start of a DebugSectionDescriptor stream via stream.seek(0, os.SEEK_SET) but this does not work if the section is located somewhere else in the stream. This forces the user to implement their own file-like sub-stream, or copy the file into memory.

Maybe it should use the global_offset property that the user provides to the constructor?

Incorrect parsing of CORE types

CORE files use notes to store register information. In particular, NT_GNU_ABI_TAG and NT_PRSTATUS are the same value.

Currently, pyelftools assumes the former and discards the data stream.

This should not be done when elftype is "CORE".

Bug in interpreter of DW_CFA_remember_state

Dear all

I believe there is a small bug in the interpreter of DW_CFA_remember_state. The interpreter pushes the current line on a stack, but, due to sharing, subsequent bytecode instructions can modify the pushed line whenever they modify the current line.

For instance, this FDE entry:

00000088 000000000000001c 0000005c FDE cie=00000030 pc= 0000000000400980.. 00000000004009d1
DW_CFA_advance_loc: 4 to 0000000000400984
DW_CFA_def_cfa_offset: 24
DW_CFA_advance_loc1: 64 to 00000000004009c4
DW_CFA_remember_state
DW_CFA_def_cfa_offset: 8
DW_CFA_advance_loc: 4 to 00000000004009c8
DW_CFA_restore_state

should generate:

00000060 000000000000001c 00000034 FDE cie=00000030 pc=0000000000400980..00000000004009d1
LOC CFA ra
0000000000400980 rsp+8 c-8
0000000000400984 rsp+24 c-8
00000000004009c4 rsp+8 c-8
00000000004009c8 rsp+24 c-8

(observe the last +24, due to remembering the line for 400984), while it results in:

00000060 000000000000001c 0000000000000034 FDE cie=00000030 pc=0000000000400980..00000000004009d1
LOC CFA ra
0000000000400980 rsp+8 c-8
0000000000400984 rsp+24 c-8
00000000004009c4 rsp+8 c-8
00000000004009c8 rsp+8 c-8

The final +8 is due due the update to the current line in 4009c4.

I believe that deep-copying the current line before pushing it on the stack + updating the pc when pulling is a simple fix (patch below).

I cannot submit you a binary to reproduce easily as the FDE entry was obtained by parsing eh_frame info via an eh_frame parser I implemented (that I will eventually contribute to pyelftools) and not by parsing debug_frames.

-francesco

Index: pyelftools/elftools/dwarf/callframe.py
===================================================================
--- pyelftools/elftools/dwarf/callframe.py  (original)
+++ pyelftools/elftools/dwarf/callframe.py  (working copy)
@@ -501,9 +501,11 @@
                 else:
                     cur_line.pop(instr.args[0], None)
             elif name == 'DW_CFA_remember_state':
-                line_stack.append(cur_line)
+                line_stack.append(copy.deepcopy(cur_line))
             elif name == 'DW_CFA_restore_state':
+                pc = cur_line['pc'] 
                 cur_line = line_stack.pop()
+                cur_line['pc'] = pc

         # The current line is appended to the table after all instructions
         # have ended, in any case (even if there were no instructions).

add to support DWARF macro info

dwarf4 support MACRO info ,
could you add to support it ?
for .debug_macro section

Trying to get LMA and VMA addresses

Moved from bitbucket issue #21:

Hi,

I'm using your library to parse a powerpc ELF file and that seems to work great except for VMA address extraction.

I use this code to get VMA address:

with open( elf_file_name, "rb" ) as f:
elf_file = ELFFile( f )
for i in range( elf_file.num_sections() ):
section = elf_file.get_section( i )
segment_header = elf_file._get_segment_header( i )
print "Name = %s" % section.name
print "VMA = 0x%x" % segment_header[ "p_paddr" ]
print "LMA = 0x%x\n" % section[ "sh_addr" ]

(yeah, i know, it's ugly to use private method "_get_segment_header" but i found nothing better). The output is:

Name =
VMA = 0xfc100000
LMA = 0x0

Name = .text
VMA = 0xfc200000
LMA = 0x0

Name = .rodata
VMA = 0xfc300000
LMA = 0x273d8

Name = .data
VMA = 0xfc400000
LMA = 0x800000

Name = .stacks
VMA = 0xfc500000
LMA = 0x800a80

Name = .bss
VMA = 0xfc600000
LMA = 0x812a80
...

and objdump -h output:

main: file format elf32-powerpc

Sections:
Idx Name Size VMA LMA File off Algn
0 .text 000273d8 00000000 fc100000 00001000 22
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .rodata 000041a0 000273d8 fc200000 000283d8 23
CONTENTS, ALLOC, LOAD, READONLY, DATA
2 .data 00000a7c 00800000 fc300000 0002d000 23
CONTENTS, ALLOC, LOAD, DATA
3 .stacks 00012000 00800a80 fc300a80 0002da80 24
CONTENTS, ALLOC, LOAD, DATA
4 .bss 00089c88 00812a80 fc312a80 0003fa80 2**5
ALLOC
...

So section's name and VMA address are incorrect :( It's the correct way to do that or it's a bug ?

Thx

Non-determinism of dict enumeration results in non-deterministic parsing

If we enumerate the segments of an ARM EABI ELF file and print the p_type field of each, we print different results on different runs with the same input. Both PT_ARM_UNWIND and PT_AARCH64_UNWIND have the value 0x70000001; we decode this value using the Enum facility of the Construct library. Depending on the order in which we enumerate the ENUM_P_TYPE entries, Enum can prefer either PT_ARM_UNWIND or PT_AARCH64_UNWIND.

Enumeration mappings we provide to Construct should never be ambiguous!

Rename readelf.py to pyreadelf (or something similar)

Hi,
I'm preparing the package for Debian and there is this rule that programs in /usr/bin should not have extension (https://lintian.debian.org/tags/script-with-language-extension.html). I propose to change it to pyreadelf - I'll prepare a PR if you agree.

Cheers,
Tomasz

CompileUnit._parse_DIEs() misreads DIE size in rare instances

0x0010adc0:     TAG_base_type [11]  
AT_name( "uint32" )
AT_encoding( DW_ATE_unsigned )
AT_byte_size( 0x04 )
Unknown DW_AT constant: 0x2900( 0x0a )

0x0010adcb: TAG_base_type [11]
AT_name( "uint8" )
AT_encoding( DW_ATE_unsigned )
AT_byte_size( 0x01 )
Unknown DW_AT constant: 0x2900( 0x08 )

0x0010add5: TAG_pointer_type [17]
AT_name( "*runtime.typeAlg" )
AT_type( {0x000000000010adf0} ( runtime.typeAlg ) )
Unknown DW_AT constant: 0x2900( 0x16 )

0x0010adf0: TAG_structure_type [21] *
AT_name( "runtime.typeAlg" )
AT_byte_size( 0x00000010 )
Unknown DW_AT constant: 0x2900( 0x19 )

0x0010ae03: TAG_member [6]
AT_name( "hash" )
AT_data_member_location( +0 )
AT_type( {0x000000000010ae40} ( func(unsafe.Pointer, uintptr) uintptr ) )

0x0010ae14: TAG_member [6]
AT_name( "equal" )
AT_data_member_location( +8 )
AT_type( {0x000000000010aec6} ( func(unsafe.Pointer, unsafe.Pointer) bool ) )

0x0010ae26: NULL

0x0010ae27: TAG_typedef [22]
AT_name( "runtime.typeAlg" )
AT_type( {0x000000000010adf0} ( runtime.typeAlg ) )

Arm abbrev_code

Hi

it seems that with arm binary the abbrev_code is wrong, pyelftools catchs only the last byte, the abbrev_code is on 2 bytes.

ex:
<1><6a126>: Abbrev Number: 6 (DW_TAG_typedef)

DW_TAG_typedef is 0x16

Cheers,

Need way to determine raw value of enum

Today, when we decode ELF enum fields, we represent the decoded value as a string. There's no easy way to recover the numeric value from which that string was built. Instead of using str directly, we should subclass str and provide an attribute on the subclasses object that contains the original, un-decoded value.

elffile.get_dwarf_info() Failed

I was trying to parse a linux kernel object which has debug information. The function elffile.get_dwarf_info() failed. I've test with some other linux kernel object, all failed with the same error.

python dwarf_die_tree.py ~/new-kernel/linux-3.12.1/vmlinux-3.12.1
Processing file: /home/cs3612/new-kernel/linux-3.12.1/vmlinux-3.12.1
Traceback (most recent call last):
File "dwarf_die_tree.py", line 65, in
process_file(filename)
File "dwarf_die_tree.py", line 31, in process_file
dwarfinfo = elffile.get_dwarf_info()
File "../elftools/elf/elffile.py", line 140, in get_dwarf_info
relocate_dwarf_sections)
File "../elftools/elf/elffile.py", line 357, in _read_dwarf_section
section_stream, reloc_section)
File "../elftools/elf/relocation.py", line 124, in apply_section_relocations
self._do_apply_relocation(stream, reloc, symtab)
File "../elftools/elf/relocation.py", line 181, in _do_apply_relocation
value_struct.build_stream(relocated_value, stream)
File "../elftools/construct/core.py", line 211, in build_stream
self._build(obj, stream, Container())
File "../elftools/construct/core.py", line 358, in _build
raise FieldError(ex)
elftools.construct.core.FieldError: integer out of range for 'L' format code

test/run_readelf_tests.py failed due to 32bit system and 64bit readelf-binary

There is a problem with 32bit systems and the run_readelf_tests.py.
Your readelf binary in test/external_tools/readelf is 64bit due to this the run_readelf_tests.py failed. I copied my readelf binary to this location but this causes a failing test, because youre hardcoded expectations are not equal to the output of the readelf 32bit binary.

Fail-Log with 64bit readelf binary on 32bit system:

Test file 'test/testfiles_for_readelf/update32.o.elf'
Traceback (most recent call last):
  File "test/run_readelf_tests.py", line 214, in <module>
    sys.exit(main())
  File "test/run_readelf_tests.py", line 203, in main
    verbose=options.verbose)
  File "test/run_readelf_tests.py", line 60, in run_test_on_file
    rc, stdout = run_exe(exe_path, args)
  File "/tmp/yaourt-tmp-user/aur-python2-pyelftools/src/pyelftools-0.23/test/utils.py", line 30, in run_exe
    proc = subprocess.Popen(popen_cmd, stdout=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception

Fail-Log with 32bit readelf binary on 32bit system:

user@trudy ..python2-pyelftools/src/pyelftools-0.23 % python2 test/run_readelf_tests.py
Test file 'test/testfiles_for_readelf/exe_simple32.elf'
Test file 'test/testfiles_for_readelf/exe_simple64.elf'
.......................FAIL
....for option "--debug-dump=frames"
....Output #1 is readelf, Output #2 is pyelftools
@@ Mismatch on line #1:
>>00000000 0000000000000014 ffffffff cie<<
>>00000000 0000000000000014 00000000ffffffff cie<<
 ([('equal', 0, 26, 0, 26), ('insert', 26, 26, 26, 34), ('equal', 26, 38, 34, 46)])
@@ Output #1 dumped to file: /tmp/out1_cuI3KG.stdout
@@ Output #2 dumped to file: /tmp/out2_FUr_dM.stdout
.......................FAIL
....for option "--debug-dump=frames-interp"
....Output #1 is readelf, Output #2 is pyelftools
@@ Mismatch on line #1:
>>00000000 0000000000000014 ffffffff cie "" cf=1 df=-8 ra=16<<
>>00000000 0000000000000014 00000000ffffffff cie "" cf=1 df=-8 ra=16<<
 ([('equal', 0, 26, 0, 26), ('insert', 26, 26, 26, 34), ('equal', 26, 58, 34, 66)])
@@ Output #1 dumped to file: /tmp/out1_WiRZEN.stdout
@@ Output #2 dumped to file: /tmp/out2_qwsiW9.stdout
Conclusion: FAIL
python2 test/run_readelf_tests.py  6.34s user 0.64s system 96% cpu 7.250 total

elf: _default_=Pass for enums

Why is _default_=Pass set for the enums?

If there is a bogus value in E_MACHINE it will not be caught, then when accessing elffile['e_machine'] we will get the unmap value which is an int, but we expect a str.

20f07e8 add _default_=Pass to the enums, but without any explanation. Could you explain the reason?

Potential Bug in pyelftools (DW_AT_type decoding)

I'm using pyelftools to decode an elf file from the XC32 complier from microchip.
My main goal is to decode variables into address so I can peak and poke them over a CAN interface.

What I found is that the DW_AT_type is not correctly being returned in some cases.

I've created files which I'll describe here (this very narrow window makes the formatting horrendous)

Good_decode.txt - shows what I want the program to do. It finds a variable and then iterates over and over until it gets to a DIE that doesn't have a DW_AT_type. It should always find at least one type that describes a variable.

Bad_decode.txt - shows two entries from my python script that is calling the pyelftools. In these two cases the DW_AT_type is not correct.

objdump.txt - shows the output of the XC32-objdump -w tool. Its a big file but if you search for 1c7d you'll find the first die for "adc_raw_data" and if you search for 1c8f you'll find the one for "adc_sensors_ready".

compare.txt - shows just the 1c7d "adc_raw_data" above and below for comparison. As you can see the only difference is in the DW_AT_type field and you can see that they disagree about the address for DW_AT_type.
0x18e9 = 6377 != 5874

Files are located here:
https://gist.github.com/bshaffer82/35f6e35be1157fc1776c#file-bad_decode-txt
Thanks

pyelftools incorrectly decodign names of Note Types.

NoteTypes are subtypes of NoteName. i.e. its only valid to use the current enum if the NoteName is GNU.
I've added a patch to fix this to the fork I made .
master...dethrophes:master

A also improved the handling of the scenario when the PT_NOTE segment aren't exactly the size of the contained notes.

Problem in Evaluating Dwarf Location Expression

Hi,
I want to evaluate the DW_AT_Location attribute.
So, first I intialiazed Compile unit and then I instantiated GenericExprVisitor object and then using that reference, calling process_expr(value) procedure. But it is retuning 'none' every time.

Code flow is as follows

Get CU instance
Gref = GenericExprVisitor(CU.structs)
x, d = die.attributes.items()
if x == 'DW_AT_Location'
Gref.process_expr(d.value)

STB_GNU_UNIQUE support. Patch included

readelf.py shows STB_GNU_UNIQUE variables as unknown. What a shame

I offer a patch that fixes the problem

diff -urN a/descriptions.py b/descriptions.py
--- a/descriptions.py   2014-11-08 16:41:54.000000000 +0300
+++ b/descriptions.py   2015-12-04 17:01:23.000000000 +0300
@@ -323,6 +323,7 @@
     STB_LOCAL='LOCAL',
     STB_GLOBAL='GLOBAL',
     STB_WEAK='WEAK',
+    STB_LOOS='UNIQUE',
 )

 _DESCR_ST_VISIBILITY = dict(

get_dwarf_info() call takes several seconds to complete

For a moderately sized binary with about 25000 DIEs this library takes several seconds to return from a call to get_dwarf_info(), even with pypy, which makes it unsuitable for several purposes. Please optimize this. I'm only interested in a a few of the DIEs with certain tags so there must be many, may cycles wasted on indexing and creating objects that are never looked at once.

Problem with _data_member_location_extra

I got a python error when running readelf.py on a basic demo C file. I was able to debug it and put a kludge in your code to work around it, but you probably know a better way to make a more permanent fix.

To recreate the problem, create a hello.out file using the source files I have included (hello.c/h):
gcc -g -o hello.out hello.c
(or just use the .out file I have submitted)

Then run
readelf.py --debug-dump=info hello.out
which gives the following python error:

Traceback (most recent call last):
File "readelf.py", line 1177, in
main()
File "readelf.py", line 1156, in main
readelf.display_debug_dump(options.debug_dump_what)
File "readelf.py", line 658, in display_debug_dump
self._dump_debug_info()
File "readelf.py", line 880, in _dump_debug_info
attr, die, section_offset)))
File "/home/bseifers/.local/lib/python2.6/site-packages/elftools/dwarf/descriptions.py", line 38, in describe_attr_value
extra_info = extra_info_func(attr, die, section_offset)
File "/home/bseifers/.local/lib/python2.6/site-packages/elftools/dwarf/descriptions.py", line 442, in _data_member_location_extra
return describe_DWARF_expr(attr.value, die.cu.structs)
File "/home/bseifers/.local/lib/python2.6/site-packages/elftools/dwarf/descriptions.py", line 149, in describe_DWARF_expr
dwarf_expr_dumper.process_expr(expr)
File "/home/bseifers/.local/lib/python2.6/site-packages/elftools/dwarf/dwarf_expr.py", line 119, in process_expr
self.stream = BytesIO(bytelist2string(expr))
File "/home/bseifers/.local/lib/python2.6/site-packages/elftools/common/utils.py", line 19, in bytelist2string
return b''.join(int2byte(b) for b in bytelist)
TypeError: 'int' object is not iterable

I was able to fix the problem by modifying elftools/dwarf/descriptions.py as follows

def _data_member_location_extra(attr, die, section_offset):
# According to section 5.5.6 of the DWARF spec v4, a data member location
# can be an integer offset, or a location description.
#
if attr.form in ('DW_FORM_data1', 'DW_FORM_data2',
'DW_FORM_data4', 'DW_FORM_data8'):
return '' # No extra description needed

Start bseifers kludge

elif attr.form in ('DW_FORM_sdata'):
return str(attr.value)

End bseifers kludge

else:
    return describe_DWARF_expr(attr.value, die.cu.structs)

I'm having trouble attaching the C files, so I'll include the source text here:

hello.c:

include <stdio.h>

include "hello.h"

const int GLOBAL_CONST;

int tryGlobal;
struct def hiLo;

int main()
{
int abc;
printf("Hello World\n");
return 0;
}

hello.h:

ifndef hello_h

define hello_h

struct def
{
int ijk;
char c;
long long lint;
float mno;
int bit1 : 1;
int bit3 : 3;
int bit2 : 2;
int bit4 : 4;
//};
}attribute((packed));

endif

file:///home/bseifers/hello.c
file:///home/bseifers/hello.h
file:///home/bseifers/hello.out

DWARF V4 Line Programs not supported

The current code doesn't support DWARF v4 Line program headers, specifically it doesn't understand the maximum_operations_per_instruction field resulting in everything getting out of alignment and the program crashing out.

I have a fix for this on my work laptop which I could push if desired?

Should `get_string` of `StringTableSection` return string instead of bytes?

I tried to use

efile.get_section_by_name('.init')

to get the '.init' section but it return None. I found it that name is bytes instead of string in Python3.

so now it should use

efile.get_section_by_name(b'.init')

to successfully get the right section. But I think that it is more easy and more readable way to get it by string.

What's more, the method name is get_string but not get_cstring

Compute Elf File Size

Hello Eli!
I love your project and because I need to be able to extract elf files from a stream of data I want to be able to calculate the file size by parsing the headers+program sections.
I had in mind two ways to calculate the effective file size but none work so far.
The first approach is sum up: header + program headers + section headers+ program sections.
The second approach is sum up: last program section + file size + header table size.
I have tried the function against your elf samples and they underestimate the file size.
Could you tell me what I am missing in my calculation?

def compute_file_size(self):
    self._emitline('File size:')
    header_size=self.elffile.header['e_ehsize'];
    table_program_size=self.elffile.header['e_phentsize']*self.elffile.header['e_phnum'];
    table_headers_size=self.elffile.header['e_shnum']*self.elffile.header['e_shentsize'];

    if self.elffile.num_segments() == 0:
        self._emitline('There are no program headers in this file.')
        return
    program_size=0
    #get the last segmment
    last_offset=0
    last_size=0
    for segment in self.elffile.iter_segments():
        program_size+=segment['p_filesz']
        if segment['p_offset']>last_offset:
            last_offset=segment['p_offset']
            last_size=segment['p_filesz']
    print "Last Offset ", self._format_hex(segment['p_offset'], fieldsize=6)
    print "Last Size", self._format_hex(segment['p_filesz'], fieldsize=5)
    total_size=header_size+table_program_size+table_headers_size+program_size
    print "Total Size via segments %d bytes \n"% (total_size)
    print "Total Size via offset   %d bytes \n"% (last_offset+last_size+table_headers_size)

Support compressed debug info sections.

elffile.py currently assumes that all debug related sections are start with .debug* but objcopy --compress-debug-sections renames them by adding 'z' prefix, so instead of .debug_info, the section becomes .zdebug_info.

List callable function in shared library

Is it possible to use pyelftools to list functions from shared library? For example:

#include <stdio.h>

void hello() {
    printf("hello()\n");
}

$ readelf.py <--magic-option> ./libhello.so
hello

The best output I could find is with nm -D:

$ nm -D --defined-only libhello.so                                                              
0000000000201038 B __bss_start
0000000000201038 D _edata
0000000000201040 B _end
00000000000006d8 T _fini
0000000000000578 T _init
00000000000006c5 T hello

but as you may see, there is a lot of excessive stuff here.

Over broad exception catching (except Exception)

I have an issue where I'm running pyelftools and need to interrupt processing. I'm calling pyelftools from a Celery task. Celery cancels tasks by raising an celery.exceptions.Terminated exception. pyelftools unfortunately catches this and as a result, my task does not get cancelled.

As a fix, all occurences of except Exception should be changed to catch more specific exceptions, such as IOError, ValueError etc.

I did not track which of the exceptions I'm hitting yet, but here's one example of offending try-except: https://github.com/eliben/pyelftools/blob/master/elftools/construct/core.py#L352

Any chance someone familiar with the pyelftools internals could determine which exceptions actually need to be caught and make the changes? Here's a list.

$ git grep --line-number "except Exception"
elftools/construct/core.py:238:        except Exception as e:
elftools/construct/core.py:352:        except Exception as ex:
elftools/construct/core.py:357:        except Exception as ex:
elftools/construct/core.py:903:                except Exception:
elftools/construct/debug.py:113:        except Exception:
elftools/construct/debug.py:124:        except Exception:

Malformed ELF causes RuntimeError on recursion

Template file from metasploit causes maximum recursion depth to exceed when given to pyelftools.

--- snip
[antti]% base64 -i template_x64_linux_dll.bin
f0VMRgIBAQAAAAAAAAAAAAMAPgABAAAAogEAAAAAAABAAAAAAAAAALAAAAAAAAAAAAAAAEAAOAACAEAAAgABAAEAAAAHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA776t3gAAAADvvq3eAAAAAAAQAAAAAAAAAgAAAAcAAAAwAQAAAAAAADABAAAAAAAAMAEAAAAAAABwAAAAAAAAAHAAAAAAAAAAABAAAAAAAAABAAAABgAAAAAAAAAAAAAAMAEAAAAAAAAwAQAAAAAAAHAAAAAAAAAAAAAAAAAAAAAIAAAAAAAAAHAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAKABAAAAAAAAoAEAAAAAAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAAogEAAAAAAAAEAAAAAAAAAAAAAAAAAAAABQAAAAAAAACgAQAAAAAAAAYAAAAAAAAAoAEAAAAAAAAKAAAAAAAAAAIAAAAAAAAACwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==
--- snip

Traceback:
--- snip
[antti]% readelf.py -d template_x64_linux_dll.bin
Traceback (most recent call last):
File "/usr/local/bin/readelf.py", line 1136, in
main()
File "/usr/local/bin/readelf.py", line 1103, in main
readelf.display_dynamic_tags()
File "/usr/local/bin/readelf.py", line 333, in display_dynamic_tags
for section in self.elffile.iter_sections():
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 92, in iter_sections
yield self.get_section(i)
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 72, in get_section
return self._make_section(section_header)
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 265, in _make_section
return DynamicSection(section_header, name, self.stream, self)
File "/usr/local/lib/python2.7/site-packages/elftools/elf/dynamic.py", line 103, in init
...
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 232, in _get_section_header
stream_pos=self._section_offset(n))
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 208, in _section_offset
return self['e_shoff'] + n * self['e_shentsize']
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 176, in getitem
return self.header[name]
RuntimeError: maximum recursion depth exceeded while calling a Python object

readelf.py --debug-dump=decodedline fails

Running python scripts/readelf.py --debug-dump=decodedline test/testfiles_for_unittests/arm_with_form_indirect.elf fails with the following error:

Traceback (most recent call last):
  File "scripts/readelf.py", line 1136, in <module>
    main()
  File "scripts/readelf.py", line 1115, in main
    readelf.display_debug_dump(options.debug_dump_what)
  File "scripts/readelf.py", line 635, in display_debug_dump
    self._dump_debug_line_programs()
  File "scripts/readelf.py", line 872, in _dump_debug_line_programs
    cu_filename = bytes2str(lineprogram['file_entry'][0].name)
TypeError: 'NoneType' object is not subscriptable

elf relocation looks incorrect in elf_relocation example script

hello,

it seems that elf_relocation example returns wrong offsets:

Processing file: /bin/ls
.rela.dyn section with 7 relocations
Relocation (RELA)
offset = 6406136
Relocation (RELA)
offset = 6407808
Relocation (RELA)
offset = 6407824
Relocation (RELA)
offset = 6407840
Relocation (RELA)
offset = 6407848
Relocation (RELA)
offset = 6407856
Relocation (RELA)
offset = 6407864

with reafelf:

Relocation section '.rela.dyn' at offset 0x1640 contains 7 entries:
Offset Info Type Sym. Value Sym. Name + Addend
00000061bff8 004200000006 R_X86_64_GLOB_DAT 0000000000000000 gmon_start + 0
00000061c680 007500000005 R_X86_64_COPY 000000000061c680 __progname + 0
00000061c690 007200000005 R_X86_64_COPY 000000000061c690 stdout + 0
00000061c6a0 007e00000005 R_X86_64_COPY 000000000061c6a0 optind + 0
00000061c6a8 008000000005 R_X86_64_COPY 000000000061c6a8 optarg + 0
00000061c6b0 007a00000005 R_X86_64_COPY 000000000061c6b0 __progname_full + 0
00000061c6b8 007d00000005 R_X86_64_COPY 000000000061c6b8 stderr + 0

ELF file for ARM is not properly supported.

I have a dwarf file produced from ARM (android) and readelf.py fails:

$ readelf.py --debug-dump=info module_dwarf.ko
ELF error: Unsupported relocation type: 2

The reason for that is because there are no relocation recipes for ARM in elftools/elf/relocation.py only ones exist for MIPS and X64 and X86.

The complete list is here:
http://infocenter.arm.com/help/topic/com.arm.doc.ihi0044e/IHI0044E_aaelf.pdf table 4-8.

As a quick hack we can lie and say that this is a MIPS file and this works ok (at least on this file) but it would be good to have proper support.

class ELFFile(elffile.ELFFile):
    def get_machine_arch(self):
        result = super(ELFFile, self).get_machine_arch()
        if result == "ARM":
            result = "MIPS"

        return result

elffile.ELFFile = ELFFile

Support for shared lineprograms

Moving from bitbucket issue #22:

Attached is an ELF file for which readelf from pyelftools behaves differently than GNU readelf.

The reason is that the file consists of multiple compilation units that share a single lineprogram. When executing readelf --debug-dump=decodedline , GNU readelf prints the source lines a single time, whereas readelf from pyelftools prints the same source lines once for every compilation unit.

Should we treat lineprograms as entities that are independent of compilation units?

---

My comments:

Yeah, I can reproduce the difference... Though it's just different behavior from readelf; all information seems to be parsed correctly. I'll take a look at how to make it behave similarly.

[...]

OK, I see what the problem is.

pyelftools enumerates line programs by following each CU and looking at its line program. readelf just goes over the line program section, and extracts the CU name from the line program itself.

This shouldn't be hard to fix by adding a method to DWARFInfo that enumerates line programs in order from the section and not looking up by CU.

Patches welcome!

examples/dwarf_location_lists.py and examples/dwarf_range_lists.py fail with Jython

examples/dwarf_location_lists.py and examples/dwarf_range_lists.py fail with Jython.
Other examples/*.py tests pass with Jython.
All examples/*.py tests pass with CPython.

$ jython2.7 test/run_examples_test.py
Example './examples/dwarf_decode_address.py'
Example './examples/dwarf_location_lists.py'
.......FAIL comparison
@@ Output #1 dumped to file: /tmp/out1_bbpziI.stdout
Example './examples/examine_dwarf_info.py'
Example './examples/elfclass_address_size.py'
Example './examples/elf_show_debug_sections.py'
Example './examples/elf_low_high_api.py'
Example './examples/dwarf_range_lists.py'
.......FAIL comparison
@@ Output #1 dumped to file: /tmp/out1_WfqAXl.stdout
Example './examples/elf_relocations.py'
Example './examples/dwarf_die_tree.py'

Conclusion: FAIL

UnboundLocalError: local variable 'stringtable' referenced before assignment

I tried running the following command: python readelf.py meander.debug -e

and I received the Traceback shown below:

Traceback (most recent call last):
File "../../lib/pyelftools-0.22/scripts/readelf.py", line 1136, in
main()
File "../../lib/pyelftools-0.22/scripts/readelf.py", line 1101, in main
show_heading=not do_file_header)
File "../../lib/pyelftools-0.22/scripts/readelf.py", line 170, in display_program_headers
for segment in self.elffile.iter_segments():
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 109, in iter_segments
yield self.get_segment(i)
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 103, in get_segment
return self._make_segment(segment_header)
File "/usr/local/lib/python2.7/site-packages/elftools/elf/elffile.py", line 222, in _make_segment
return DynamicSegment(segment_header, self.stream, self)
File "/usr/local/lib/python2.7/site-packages/elftools/elf/dynamic.py", line 123, in init
Dynamic.init(self, stream, elffile, stringtable, self['p_offset'])
UnboundLocalError: local variable 'stringtable' referenced before assignment

license mismatch in elftools.construct

Hello,
I noticed that elftools.construct is a 3rd party library which is, as far as I can tell, under MIT license. This is confusing and quite probably violation of license terms, since the main LICENSE file claims the code to be public domain.

Cheers,
Tomasz

latest pyelftools fails on Linux with cannot import name SUNWSyminfoTableSection when installed

it seems to work when running the examples inside the source dir. Some missing file in setup.py?

I reverted the solaris changed for now in my copy

describe_DWARF_expr() raises on some inputs.

$ python scripts/readelf.py --debug-dump=info /tmp/module_dwarf.ko 
....
 <2><359>: Abbrev Number: 13 (DW_TAG_member)
    <35a>   DW_AT_name        : (indirect string, offset: 0xab00): counter  
    <35e>   DW_AT_decl_file   : 5   
    <35f>   DW_AT_decl_line   : 176 
    <360>   DW_AT_type        : <0x70>  
Traceback (most recent call last):
  File "scripts/readelf.py", line 1170, in <module>
    main()
  File "scripts/readelf.py", line 1149, in main
    readelf.display_debug_dump(options.debug_dump_what)
  File "scripts/readelf.py", line 658, in display_debug_dump
    self._dump_debug_info()
  File "scripts/readelf.py", line 881, in _dump_debug_info
    attr, die, section_offset)))
  File "./elftools/dwarf/descriptions.py", line 38, in describe_attr_value
    extra_info = extra_info_func(attr, die, section_offset)
  File "./elftools/dwarf/descriptions.py", line 388, in _location_list_extra
    return describe_DWARF_expr(attr.value, die.cu.structs)
  File "./elftools/dwarf/descriptions.py", line 149, in describe_DWARF_expr
    dwarf_expr_dumper.process_expr(expr)
  File "./elftools/dwarf/dwarf_expr.py", line 119, in process_expr
    self.stream = BytesIO(bytelist2string(expr))
  File "./elftools/common/utils.py", line 19, in bytelist2string
    return b''.join(int2byte(b) for b in bytelist)
TypeError: 'int' object is not iterable

The sample file is at:
https://drive.google.com/file/d/0B9hc84IflFGbSDJ5NnNQRFotZDA/view?usp=sharing

Issue with CompileUnit._parseDIEs()

Currently, the way that a CompileUnit finds all of the DIEs contained within itself is via the following code:

# Compute the boundary (one byte past the bounds) of this CU in the
# stream
cu_boundary = ( self.cu_offset +
            self['unit_length'] +
            self.structs.initial_length_field_size())

# First pass: parse all DIEs and place them into self._dielist
die_offset = self.cu_die_offset
while die_offset < cu_boundary:
    die = DIE(
        cu=self,
        stream=self.dwarfinfo.debug_info_sec.stream,
        offset=die_offset)
    self._dielist.append(die)
    die_offset += die.size

This algorithm assumes that all DIE are aligned consecutively. That is, the next DIE starts where the current one ends. And this is fine for most cases. But I've run into an executable where this assumption does NOT hold true. Instead, there's one point where there's a single byte between one DIE and the next DIE . As a result, the second DIE doesn't get parsed properly.

Is there any way that this can be fixed? Is this even a bug, or is it definitively an issue with the executable?

Thanks!

support parsing of dynamic ELFs w/out section headers

Moving from bitbucket pull request #11 by Mike Frysinger:

support parsing of dynamic ELFs w/out section headers

At runtime, ELFs do not use the section headers at all. Instead, only
the program segments and dynamic tags get used. This means you can
strip the section table completely from an ELF and have it still work.

In practice, people rarely do this, but it's not unheard of. Make the
Dynamic tags work even in these cases by loading the strings table the
same way the runtime loader does:

• parse the symtab address from DT_STRTAB
• locate the file offset via the program segments

In order to avoid circular deps (parsing a dyntag requires walking parsed
dyntags), add a set of internal funcs for returning the raw values.
ave it still work.

Error trying to obtain dwarf info: 'structs' is not defined

Hello,
I'm trying to run the example examine_dwarf_info in an ELF file from Microvision but I receive the following error, could you please tell me if I'm doing something wrong or if there is a way to fix it? The example runs properly with the elf file provided as example. I'm using Python 2.7.3. Thank you.

Processing file: uvision/O0/obj/UserManualExample.axf
  Found a compile unit at offset 0, length 15044
Traceback (most recent call last):
  File "examine_dwarf_info.py", line 50, in <module>
    process_file(filename)
  File "examine_dwarf_info.py", line 42, in process_file
    top_DIE = CU.get_top_DIE()
  File "/usr/lib/python2.7/site-packages/elftools/dwarf/compileunit.py", line 76, in get_top_DIE
    return self._get_DIE(0)
  File "/usr/lib/python2.7/site-packages/elftools/dwarf/compileunit.py", line 95, in _get_DIE
    self._parse_DIEs()
  File "/usr/lib/python2.7/site-packages/elftools/dwarf/compileunit.py", line 119, in _parse_DIEs
    offset=die_offset)
  File "/usr/lib/python2.7/site-packages/elftools/dwarf/die.py", line 90, in __init__
    self._parse_DIE()
  File "/usr/lib/python2.7/site-packages/elftools/dwarf/die.py", line 185, in _parse_DIE
    value = self._translate_attr_value(form, raw_value)
  File "/usr/lib/python2.7/site-packages/elftools/dwarf/die.py", line 207, in _translate_attr_value
    structs.Dwarf_dw_form[form], self.stream)
NameError: global name 'structs' is not defined

Ability to read the full list of ARM build attributes such as Tag_CPU_arch Tag_CPU_name from the .ARM.attributes section

There doesn't seem to be a way to access these attributes for ARM ELF binaries.

http://llvm.org/docs/doxygen/html/ARMBuildAttributes_8h_source.html

The readelf tool does the correct thing here:
readelf -A ... where ... is some ARM ELF binary will show these tags.

Disagreement between objdump and pyelftools on DW_TAG_typedef::DW_AT_type

I am trying to use pyelftools to parse both TriCore and C166 ELF files (just TriCore for now) to get a list of variables and addresses including being aware of structure members. The application will be for a remote watch window for the embedded target. I have fiddled with the dwarf_die_tree.py example but ran into an issue where I am unable to connect between typedef’s and the unnamed structure definitions they reference. Or so it seems to my DWARF-ignorant brain (in case my previous comments did not in some way make that obvious).

To avoid any issues associated with my particular architecture ELF I also tried my script against test/testfiles_for_unittests/sample_exe64.elf and observed the same thing. When I run objdump -W as a reference I get, amongst other things:

<1><1d6>: Abbrev Number: 3 (DW_TAG_typedef)
    <1d7>   DW_AT_name        : (indirect string, offset: 0xcd): size_t
    <1db>   DW_AT_decl_file   : 2
    <1dc>   DW_AT_decl_line   : 214
    <1dd>   DW_AT_type        : <0x1e1>

My pyelftools script results in (again, just a snippet):

  DIE tag=DW_TAG_typedef
    Name: size_t
    Offset: 470
    File: 2
    Line: 214
    Type: 63
    Attributes: OrderedDict([('DW_AT_name', AttributeValue(name='DW_AT_name', form='DW_FORM_strp', value=b'size_t', raw_value=205, offset=471)), ('DW_AT_decl_file', AttributeValue(name='DW_AT_decl_file', form='DW_FORM_data1', value=2, raw_value=2, offset=475)), ('DW_AT_decl_line', AttributeValue(name='DW_AT_decl_line', form='DW_FORM_data1', value=214, raw_value=214, offset=476)), ('DW_AT_type', AttributeValue(name='DW_AT_type', form='DW_FORM_ref4', value=63, raw_value=63, offset=477))])
    DIE: ['__class__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_children', '_parent', '_parse_DIE', '_translate_attr_value', 'abbrev_code', 'add_child', 'attributes', 'cu', 'dwarfinfo', 'get_full_path', 'get_parent', 'has_children', 'is_null', 'iter_children', 'iter_siblings', 'offset', 'set_parent', 'size', 'stream', 'tag']

What seems to me to be an issue is that objdump shows DW_AT_type as 0x1e1 (481) as opposed to pyelftools which returns 63 (0x3f). The other values I have compared seem to correspond. Is this a simple lack of understanding of DWARF or a misuse of pyelftools, or is there an issue here? I started to dig in the code a bit but with my limited knowledge I didn’t find anything that looking glaringly wrong.

Here’s my system info (Python3 within Cygwin64 within Win7 64):

CYGWIN_NT-6.1 GUS-CZJCBS1 1.7.28(0.271/5/3) 2014-02-09 21:06 x86_64 Cygwin

Python 3.2.5 (default, Oct  2 2013, 22:58:11)
[GCC 4.8.1] on cygwin

Installed pyelftools from Git rev c9594acd0e1a1b87ab9a8b1de2b22c1411d617ff

Thank you for any time you choose to spend helping me.
-kyle

Parsing 12 Mb of DIE's takes 1 minute

Hi ,

Performance seems pretty poor when parsing an entire DWARF section for an elf. The DWARF size is about 12 meg and parsing takes 1 minute.

Attaching perf call graph. Commercial debuggers are able to parse the dwarf in < 2 seconds.

RepeatUntil parse method seems to be eating most the time. Is it the object copying that is taking too long?

Any suggestion on how to speed up performance?

C++ name demangling

Hi,

so far I know you are interested in the C++ world. Was looking for a python elf reader and bumped into this project. Having seen your blog, had also the hope to find C++ demangling in your library, but unfortunately it is not the case. Are you planning to implement something in this direction or you would just use the libiberty implementation.

Anyway thanks for the great source code.

add support for modifying and writing back to file

It would be great to be able to modify certain segments and sections and be able to write dump out the same to a file.

memory consumption and performance problems

Hi,

The following code never finishes (I killed it after 30 minutes) and consumes more than 3 GiB (and growing amount) of RAM.

$ cat fff.py
from __future__ import print_function

import libarchive
import sys
import os
import resource
from six.moves import cStringIO

try:
    from elftools.elf.elffile import ELFFile
except ImportError as exc:
    print(str(exc), file=sys.stderr)
    sys.exit(-1)


def get_producer(debugfile):
    elffile = ELFFile(debugfile)
    dwarfinfo = elffile.get_dwarf_info()

    producers = set()

    for CU in dwarfinfo.iter_CUs():
        # Start with the top DIE, the root for this CU's DIE tree
        top_DIE = CU.get_top_DIE()
        try:
            attrs = top_DIE.attributes['DW_AT_producer']
            if attrs.form == 'DW_FORM_GNU_strp_alt':
                print("XXX")
            elif attrs.form == 'DW_FORM_strp':  # lucky ;)
                print("XXX")
            else:
                print(attrs.form)
        except:
            pass

    return producers


def process_file(debugfile):
    elffile = ELFFile(debugfile)

    if not elffile.has_dwarf_info():
        assert 0
    else:
        return get_producer(debugfile)

pid = os.getpid()

a = libarchive.Archive(sys.argv[1])

for entry in a:
    print(entry.pathname)
    memusage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    sys.stderr.write("%s -> %s -> %s -> %s\n"
                     % (pid, sys.argv[1], entry.pathname, memusage))

    try:
        data = a.read(entry.size)
        memusage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        sys.stderr.write("[AR] %s -> %s -> %s -> %s\n"
                         % (pid, sys.argv[1], entry.pathname, memusage))

        fh = cStringIO(data)
        sys.stderr.write("[ARP] %s -> %s -> %s -> %s\n"
                         % (pid, sys.argv[1], entry.pathname, memusage))

        process_file(fh)
    except:
        pass

a.close()

$  python fff.py seamonkey-debuginfo-2.24-1.fc21.x86_64.rpm  # sit back ;(

You can download the problematic RPM from this URL.

python-libarchive is responsible for consuming 600 MiB (which is OK).