A modified pycparser to parse 010 templates
License: Other
py010parser is a python library that can parse 010 templates. It is a modified fork of Eli Bendersky's pycparser project.
Sweetscape's 010 editor is a binary-format editor and parser. Many templates can be found online for most binary formats.
This project (py010parser) is an effort to make 010 scripts parseable from python, with the intent to build additional tools using py010parser, such as pfp, an 010 template interpreter.
Variables/declaration scope doesn't seem to be working correctly in struct call parameters within nested structs when variables farther down the scope stack are referenced. E.g., this fails:
E.g., this fails to be parsed correctly:
struct PNG_CHUNK_BKGD (int32 colorType) {
};
int blah[4];
typedef struct {
PNG_CHUNK_BKGD bkgd(blah[1]);
} PNG_CHUNK;Bug if we declare a PNG_CHUNK_BKGD instance outside of the PNG_CHUNK struct, with the same parameters, it works:
struct PNG_CHUNK_BKGD (int32 colorType) {
};
int blah[4];
PNG_CHUNK_BKGD bkgd(blah[1]);Traceback (most recent call last):
File "/home/james/__ws__/dev/py010parser/tests/test_struct.py", line 118, in test_basic_struct_with_args_calling_non_simple_params
""", optimize=True, predefine_types=True)
File "tests/../py010parser/__init__.py", line 137, in parse_string
keep_scopes = keep_scopes
File "tests/../py010parser/c_parser.py", line 180, in parse
debug=debuglevel)
File "tests/../py010parser/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "tests/../py010parser/ply/yacc.py", line 1047, in parseopt_notrack
tok = self.errorfunc(errtoken)
File "tests/../py010parser/c_parser.py", line 1972, in p_error
column=self.clex.find_tok_column(p)))
File "tests/../py010parser/plyparser.py", line 54, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
ParseError: /tmp/tmpixU8tf:7:29: before: [
See the Structs With Arguments section at http://www.sweetscape.com/010editor/manual/Structs.htm
E.g.
struct VarSizeStruct (int arraySize)
{
int id;
int array[arraySize];
};
// and
typedef struct (int arraySize)
{
int id;
int array[arraySize];
} VarSizeStruct;DEXTemplate.bt uses these
I'm not sure why this happened, but after cleaning up and pushing a new version of py010parser, pfp is failing with the error:
File "../pfp/__init__.py", line 18, in <module>
PARSER = py010parser.c_parser.CParser()
File "/home/james/__ws__/dev/pfp/venv2/local/lib/python2.7/site-packages/py010parser/c_parser.py", line 76, in __init__
lextab=lextab)
File "/home/james/__ws__/dev/pfp/venv2/local/lib/python2.7/site-packages/py010parser/c_lexer.py", line 69, in build
self.lexer = lex.lex(object=self, **kwargs)
File "/home/james/__ws__/dev/pfp/venv2/local/lib/python2.7/site-packages/py010parser/ply/lex.py", line 898, in lex
lexobj.readtab(lextab,ldict)
File "/home/james/__ws__/dev/pfp/venv2/local/lib/python2.7/site-packages/py010parser/ply/lex.py", line 239, in readtab
titem.append((re.compile(lre[i][0],lextab._lexreflags | re.VERBOSE),_names_to_funcs(lre[i][1],fdict)))
File "/home/james/__ws__/dev/pfp/venv2/lib/python2.7/re.py", line 194, in compile
return _compile(pattern, flags)
File "/home/james/__ws__/dev/pfp/venv2/lib/python2.7/re.py", line 249, in _compile
p = sre_compile.compile(pattern, flags)
File "/home/james/__ws__/dev/pfp/venv2/lib/python2.7/sre_compile.py", line 583, in compile
"sorry, but this version only supports 100 named groups"
AssertionError: sorry, but this version only supports 100 named groups
Using the regex library instead of the built-in re library within lex.py solves this
courtesy of @vit9696 at d0c-s4vage/pfp#24:
Multiword types like "unsigned short" or "unsigned char" are causing a parse error when used in enum declarations. E. g.
enum <unsigned short> MY_ENUM {
VAL_1 = 0,
VAL_2 = 1
};
If one creates a typedef and replaces the type by a single word, it will be ok. I. e.
typedef unsigned short u16;
enum <u16> MY_ENUM {
VAL_1 = 0,
VAL_2 = 1
};
This will currently fail and be treated as a function prototype:
typedef struct (int param1) {
...
} struct_type;
local int val = 10;
struct_type test(val); // will be treated as a FuncDeclFor example, from https://www.sweetscape.com/010editor/repository/files/EXE.bt:
WORD IMAGE_FILE_LARGE_ADDRESS_AWARE:1 <comment="0x0020 App can handle >2gb addresses">;
The > sign inside of the string causes a parsing error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/james/__ws__/dev/pfp/pfp/__init__.py", line 91, in parse
printf=printf,
File "/home/james/__ws__/dev/pfp/pfp/interp.py", line 733, in parse
self._ast = self._parse_string(template, predefines)
File "/home/james/__ws__/dev/pfp/pfp/interp.py", line 878, in _parse_string
keep_scopes=predefines,
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/__init__.py", line 137, in parse_string
keep_scopes = keep_scopes
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/c_parser.py", line 178, in parse
debug=debuglevel)
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/ply/yacc.py", line 1047, in parseopt_notrack
tok = self.errorfunc(errtoken)
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/c_parser.py", line 1952, in p_error
column=self.clex.find_tok_column(p)))
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/plyparser.py", line 54, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
py010parser.plyparser.ParseError: tests/templates/EXE.bt:163:77: before: 2
courtesy of @vit9696 at d0c-s4vage/pfp#28:
true and false keywords are unsupported by pfp.
Parsing of the EXE template (https://www.sweetscape.com/010editor/repository/files/EXE.bt) fails with
In [2]: dom = pfp.parse(data_file="/home/user/samples/d22563d7b913f3c64176301f3de41a16f604bde5e4d335405574485d1817
...: 3844", template_file="/home/user/Documents/SweetScape/010 Templates/Repository/EXE.bt")
---------------------------------------------------------------------------
ParseError Traceback (most recent call last)
<ipython-input-2-0ad089ec8cd1> in <module>
----> 1 dom = pfp.parse(data_file="/home/user/samples/d22563d7b913f3c64176301f3de41a16f604bde5e4d335405574485d18173844", template_file="/home/user/Documents/SweetScape/010 Templates/Repository/EXE.bt")
~/.venv/pfp/lib/python3.7/site-packages/pfp/__init__.py in parse(data, template, data_file, template_file, interp, debug, predefines, int3, keep_successful, printf)
89 orig_filename=orig_filename,
90 keep_successful=keep_successful,
---> 91 printf=printf,
92 )
93
~/.venv/pfp/lib/python3.7/site-packages/pfp/interp.py in parse(self, stream, template, predefines, orig_filename, keep_successful, printf)
731 self._template = template
732 self._template_lines = self._template.split("\n")
--> 733 self._ast = self._parse_string(template, predefines)
734 self._dlog("parsed template into ast")
735
~/.venv/pfp/lib/python3.7/site-packages/pfp/interp.py in _parse_string(self, string, predefines)
876 parser=self._parser,
877 # only keep the scopes if we ran the predefines
--> 878 keep_scopes=predefines,
879 )
880 res.ext = exts + res.ext
~/.venv/pfp/lib/python3.7/site-packages/py010parser/__init__.py in parse_string(text, parser, filename, optimize, predefine_types, use_cpp, cpp_path, cpp_args, keep_scopes)
135 filename,
136 predefine_types = predefine_types,
--> 137 keep_scopes = keep_scopes
138 )
~/.venv/pfp/lib/python3.7/site-packages/py010parser/c_parser.py in parse(self, text, filename, debuglevel, predefine_types, keep_scopes)
176 input=text,
177 lexer=self.clex,
--> 178 debug=debuglevel)
179
180 res.ext = predefined_ext + res.ext
~/.venv/pfp/lib/python3.7/site-packages/py010parser/ply/yacc.py in parse(self, input, lexer, debug, tracking, tokenfunc)
263 return self.parseopt(input,lexer,debug,tracking,tokenfunc)
264 else:
--> 265 return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
266
267
~/.venv/pfp/lib/python3.7/site-packages/py010parser/ply/yacc.py in parseopt_notrack(self, input, lexer, debug, tracking, tokenfunc)
1045 if errtoken and not hasattr(errtoken,'lexer'):
1046 errtoken.lexer = lexer
-> 1047 tok = self.errorfunc(errtoken)
1048 del errok, token, restart # Delete special functions
1049
~/.venv/pfp/lib/python3.7/site-packages/py010parser/c_parser.py in p_error(self, p)
1950 'before: %s' % p.value,
1951 self._coord(lineno=p.lineno,
-> 1952 column=self.clex.find_tok_column(p)))
1953 else:
1954 self._parse_error('At end of input', '')
~/.venv/pfp/lib/python3.7/site-packages/py010parser/plyparser.py in _parse_error(self, msg, coord)
52
53 def _parse_error(self, msg, coord):
---> 54 raise ParseError("%s: %s" % (coord, msg))
55
ParseError: /tmp/tmp3ckztvn9:163:77: before: 2
requirements.txt should be way more permissive
Hello everyone,
We are currently building a fuzzer for binary formats on top of py010parser.
On my Mac (MacOS 10.15.6), py010parser fails, reporting a syntax error as soon as the first '//' comment is parsed in the .bt file.
This is because on my Mac, "cpp" (/usr/bin/cpp) by default does not strip comments. One needs to invoke it as "cpp -xstdc++".
I can work around this by setting cpp_args="-xstdc++" for parsing, but you may wish to set up the default for cpp_args accordingly.
Cheers,
Andreas
This will help template scripts know if they can use the pfp-specific functionality or not.
Probably add something in c_parser.py in _define_010_typedefs:
https://github.com/d0c-s4vage/py010parser/blob/master/py010parser/c_parser.py#L185
courtesy of @vit9696 at d0c-s4vage/pfp#26:
pfp seems to work very bad with structure typedefing.
Something like that would work with both template parsers (010 Editor and pfp):
typedef struct MY_STRUCT_S {
char magic[4];
unsigned int filesize;
} MY_STRUCT;
LittleEndian();
MY_STRUCT s;
The rest will fail with all kinds of errors in pfp:
struct MY_STRUCT {
char magic[4];
unsigned int filesize;
};
LittleEndian();
MY_STRUCT s;
typedef struct MY_STRUCT {
char magic[4];
unsigned int filesize;
};
LittleEndian();
MY_STRUCT s;
struct MY_STRUCT {
char magic[4];
unsigned int filesize;
};
LittleEndian();
struct MY_STRUCT s;
struct MY_STRUCT {
char magic[4];
unsigned int filesize;
};
typedef struct MY_STRUCT ME;
LittleEndian();
ME s;
And I suppose it is not the end. Pretty much the same happens to enums, for example.
This test from pfp (https://github.com/d0c-s4vage/pfp/blob/d6e9bad/tests/test_struct_union.py#L136-L152) is now failing, while the same test succeeds in py010parser (https://github.com/d0c-s4vage/py010parser/blob/011e6c0/tests/test_basic_parse.py#L27-L40).
The test:
typedef union {
int some_int;
struct {
char a;
char b;
char c;
char d;
} some_chars;
} blah;
blah some_union;It might have something to do with predefine_types being True/False, or maybe with the preprocessor. The code hasn't changed since the tests were passing ~a year ago, but the tests fail when run on a new laptop; this makes me think it's more likely a change in the preprocessor.
The full error text:
======================================================================
ERROR: test_union (__main__.TestStructUnion)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_struct_union.py", line 151, in test_union
"""
File "/home/james/__ws__/dev/pfp/tests/utils.py", line 14, in _test_parse_build
dom = pfp.parse(six.StringIO(data), template, debug=debug, predefines=predefines)
File "../pfp/__init__.py", line 64, in parse
dom = interp.parse(data, template, predefines=predefines, orig_filename=orig_filename, keep_successful=keep_successful)
File "../pfp/interp.py", line 672, in parse
self._ast = self._parse_string(template, predefines)
File "../pfp/interp.py", line 816, in _parse_string
keep_scopes=predefines
File "/usr/local/lib/python2.7/dist-packages/py010parser/__init__.py", line 113, in parse_string
return parser.parse(text, filename, predefine_types=predefine_types, keep_scopes=keep_scopes)
File "/usr/local/lib/python2.7/dist-packages/py010parser/c_parser.py", line 168, in parse
debug=debuglevel)
File "/usr/local/lib/python2.7/dist-packages/py010parser/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/usr/local/lib/python2.7/dist-packages/py010parser/ply/yacc.py", line 1047, in parseopt_notrack
tok = self.errorfunc(errtoken)
File "/usr/local/lib/python2.7/dist-packages/py010parser/c_parser.py", line 1962, in p_error
column=self.clex.find_tok_column(p)))
File "/usr/local/lib/python2.7/dist-packages/py010parser/plyparser.py", line 54, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
ParseError: /tmp/tmpl6dM9f:12:32: before: ;
Example:
typedef struct(int variable)
{
unsigned short a;
unsigned short b;
} RESOURCE_DIRECTORY_TABLE;
struct RESOURCE_DIRECTORY_TABLE test(1);
Yields:
Traceback (most recent call last):
File "/home/james/__ws__/dev/pfp/tests/test_struct_union.py", line 111, in test_struct_with_parameters4
""",
File "/home/james/__ws__/dev/pfp/tests/utils.py", line 53, in _test_parse_build_with_string
return self._test_parse_build_orig(*args, **kwargs)
File "/home/james/__ws__/dev/pfp/tests/utils.py", line 73, in _test_parse_build
data, template, debug=debug, predefines=predefines, printf=printf
File "/home/james/__ws__/dev/pfp/tests/../pfp/__init__.py", line 91, in parse
printf=printf,
File "/home/james/__ws__/dev/pfp/tests/../pfp/interp.py", line 733, in parse
self._ast = self._parse_string(template, predefines)
File "/home/james/__ws__/dev/pfp/tests/../pfp/interp.py", line 878, in _parse_string
keep_scopes=predefines,
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/__init__.py", line 137, in parse_string
keep_scopes = keep_scopes
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/c_parser.py", line 178, in parse
debug=debuglevel)
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/ply/yacc.py", line 1047, in parseopt_notrack
tok = self.errorfunc(errtoken)
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/c_parser.py", line 1952, in p_error
column=self.clex.find_tok_column(p)))
File "/home/james/__ws__/dev/pfp/venv3/lib/python3.6/site-packages/py010parser/plyparser.py", line 54, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
py010parser.plyparser.ParseError: /tmp/tmp9xmb09le:8:38: before: 1
The following code is valid in 010 editor and fails to parse with py010parser:
struct TEST_STRUCT(int arraySize, int arraySize2)
{
uchar b[arraySize];
uchar c[arraySize2];
};
local int bytes = 4;
typedef struct TEST_STRUCT NEW_STRUCT;
NEW_STRUCT l(bytes, 3);This should work, but errors because the struct name isn't added to the defined types list:
struct SomeStruct;
struct OtherStruct {
SomeStruct a; // py010parser errors here
};
struct SomeStruct {
int a;
int b;
};E.g.:
import py010parser
print(py010parser.__version__) # 2.10This should lead to some performance gains, with speed and memory usage.
The parser aborts in line 234 of the template http://www.sweetscape.com/010editor/templates/files/EXETemplate.bt because of an empty structure.
py010parser.plyparser.ParseError: /var/folders/r9/lsnl52fx7tzc884cyvyrlpdr0000gn/T/tmp6G1Evt:234:40: before: }
In the code below, a struct call statement exists to declare the variable value inside of a typedef'd struct:
typedef struct (int size, int type) {
} EncodedValue <read=EncodedValueRead, optimize=false>;
typedef struct {
local int value_type = 0;
local int value_arg = 0;
EncodedValue value(value_arg, value_type);
} TEST;
TEST test1;The generated AST shows that instead of a StructCallTypeDecl node in the AST existing, a FuncDecl node exists instead.
Typedef: TEST, [], ['typedef']
TypeDecl: TEST, []
Struct: None
Decl: value_type, ['local'], [], []
TypeDecl: value_type, ['local']
IdentifierType: ['int']
Constant: int, 0
Decl: value_arg, ['local'], [], []
TypeDecl: value_arg, ['local']
IdentifierType: ['int']
Constant: int, 0
Decl: value, [], [], []
FuncDecl:
ParamList:
ID: value_arg
ID: value_type
TypeDecl: value, []
IdentifierType: ['EncodedValue']
Decl: test1, [], [], []
TypeDecl: test1, []
IdentifierType: ['TEST']
Conversely, if we change the struct to not be a typedef'd struct, everything works as expected:
typedef struct (int size, int type) {
} EncodedValue <read=EncodedValueRead, optimize=false>;
struct {
local int value_type = 0;
local int value_arg = 0;
EncodedValue value(value_arg, value_type);
} TEST;with the resulting, relevant node from the AST being:
Decl: TEST, [], [], []
TypeDecl: TEST, []
Struct: None
Decl: value_type, ['local'], [], []
TypeDecl: value_type, ['local']
IdentifierType: ['int']
Constant: int, 0
Decl: value_arg, ['local'], [], []
TypeDecl: value_arg, ['local']
IdentifierType: ['int']
Constant: int, 0
Decl: value, [], [], []
StructCallTypeDecl: value, []
IdentifierType: ['EncodedValue']
Hopefully I am right with the project now :)
local char str[12] = "Test";
void printStr(string s) {
Printf("%s\n", s);
}
string getStr() {
return str;
}
printStr(getStr());
In 010 Editor this will print "Test" in pfp this will say something like:
pfp.errors.PfpError: AttributeError: 'Array_Char_12' object has no attribute '_pfp__value'
Here is call trace up until another module interp.py
File "/usr/local/lib/python2.7/dist-packages/py010parser/__init__.py", line 90, in parse_string keep_scopes = keep_scopes File "/usr/local/lib/python2.7/dist-packages/py010parser/c_parser.py", line 178, in parse debug=debuglevel) File "/usr/local/lib/python2.7/dist-packages/py010parser/ply/yacc.py", line 265, in parse return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc) File "/usr/local/lib/python2.7/dist-packages/py010parser/ply/yacc.py", line 1047, in parseopt_notrack tok = self.errorfunc(errtoken) File "/usr/local/lib/python2.7/dist-packages/py010parser/c_parser.py", line 1952, in p_error column=self.clex.find_tok_column(p))) File "/usr/local/lib/python2.7/dist-packages/py010parser/plyparser.py", line 54, in _parse_error raise ParseError("%s: %s" % (coord, msg)) py010parser.plyparser.ParseError: <string>:1:1: before: /
Here is example of code
dom = pfp.parse(data_file=data,template=temp_)
I will attach the Template file as well for this error
MP4.txt
See pfp
When parsing http://www.sweetscape.com/010editor/templates/files/CLASSTemplate3.bt the “goto" in line 235 is considered a statement and not one of the enum values.
File "./testpfp.py", line 46, in
png = pfp.parse(data_file="ArchiveSelectionWindow.class", template=template, debug=False)
File "pfp/pfp/init.py", line 64, in parse
dom = interp.parse(data, template, predefines=predefines, orig_filename=orig_filename, keep_successful=keep_successful)
File "pfp/pfp/interp.py", line 684, in parse
self._ast = self._parse_string(template, predefines)
File "pfp/pfp/interp.py", line 828, in _parse_string
keep_scopes=predefines
File "py010parser/py010parser/init.py", line 113, in parse_string
return parser.parse(text, filename, predefine_types=predefine_types, keep_scopes=keep_scopes)
File "py010parser/py010parser/c_parser.py", line 168, in parse
debug=debuglevel)
File "py010parser/py010parser/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "py010parser/py010parser/ply/yacc.py", line 1047, in parseopt_notrack
tok = self.errorfunc(errtoken)
File "py010parser/py010parser/c_parser.py", line 1962, in p_error
column=self.clex.find_tok_column(p)))
File "py010parser/py010parser/plyparser.py", line 54, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
010 Editor allows space to be allowed between the field decl and the metadata (special attributes). E.g.
int field0<hello=blah>; // this works right now
int field1 <hello=blah>; // this works right now
int field2 < hello = blah >; // this doesn't work, but should
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.