Extensions for Eli Bendersky's pycparser
Home Page: http://pypi.python.org/pypi/pycparserext
License: Other
Extended functionality for Eli Bendersky's pycparser, in particular support for parsing GNU extensions and OpenCL.
See also the github code repository.
For some reason, double pointers sometimes cause issues with the parser. This is especially the case for casts to any double (or triple, etc) pointer type, as well as double pointers as (anonymous) arguments for function prototypes. More precisely, this code fails:
int main() {
int **a = (int **) 0;
}
As well as this one:
void foo(int **);
But strangely, this prototype does not fail:
int ** foo(int **a);
And neither does this code:
int main() {
int **a = 0;
}
Everything works with "vanilla" CParser.
I have tried to fix this issue but have not been able to do so. However, there is a workaround: if you comment out the function "p_xxx_declarator_2" in ext_c_parser.py, the double pointers work again. However, I guess, you might have issues with pointers and attributes in this case...
There are issues to parse opencl types: uint, ushort, ulong etc. Specifcally the scalar (non-vector) unsigned types.
To test this simply add a declaration uint a = 1; to any test, e.g.:
def test_opencl():
from pycparserext.ext_c_parser import OpenCLCParser
src = """
__kernel void zeroMatrix(__global float _A, int n, __global float * B)
{
uint a = 1;
int i = get_global_id(0);
for (int k=0; k<n; k++)
A[i_n+k] = 0;
}
"""
Expected: test passed, got a parse error. Note that changing it to int immediately fixes the error.
Edit: vector types parse fine.
Tested on newest version, installed by pip.
Using the example file func_calls.py from pycparser, with the only modification of instantiating a GnuCParser:
def show_func_calls(filename, funcname):
gnu_parser = GnuCParser()
ast = parse_file(filename, use_cpp=False, parser=gnu_parser)
v = FuncCallVisitor(funcname)
v.visit(ast)
Basic "asm" expressions are parsed correctly, such as
asm ("foo" ::: "bar");
or
__asm__ ("foo" ::: "bar", "bar");
However, any asm expression with i/o operands will fail with the error: AttributeError: 'Constant' object has no attribute 'name'.
This is the test file I'm using:
#include <stdio.h>
void main(void)
{
// Sample: asm volatile("" ::: "memory")
// => WORKS
asm ("foo" ::: "bar");
// Sample: __asm__ __volatile__("sub pc, pc, -4" ::: "memory", "cc")
// => WORKS
__asm__ ("foo" ::: "bar", "bar");
// Sample: __asm__ ("brev\t%0" : "+r" (__value) : : "cc")
// => ERROR
__asm__ ("foo" : "bar" (5) : : "bar");
// Sample: __asm__ __volatile__ ("ssrf\t%0" : : "i" (OFFSET))
// => ERROR
__asm__ ("foo" : : "bar" (5));
// Sample: __asm__ ("max\t%0, %1, %2" : "=r" (__value) : "r" (__arg_a), "r" (__arg_b))
// => ERROR
__asm__ ("foo" : "bar" (5) : "bar" (6), "baz" (7));
}
The samples are lines of code taken from a real application, I don't have much knowledge about the syntax of these GCC extension but these lines compile and work correctly with GCC 4.4.7 so I assume they are valid variations of the asm function.
This code is OK:
int __attribute__((__nothrow__)) fct() {}but this one failed:
int * __attribute__((__nothrow__)) fct() {}Error given is:
pycparser.plyparser.ParseError: :1:7: before: __attribute__
I tried to fix this bug myself, without success.
I discovered that when trying to parse stdio.h from Mingw.
Preprocessed sample:
__attribute__ ((__dllimport__)) FILE *__attribute__((__cdecl__)) _fsopen(const char *_Filename,const char *_Mode,int _ShFlag);dear author, I encountered a error
p = GnuCParser()
File "/usr/local/lib/python3.6/dist-packages/pycparserext/ext_c_parser.py", line 47, in init
debug=yacc_debug, write_tables=False)
File "/home/longting/.local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 3426, in yacc
raise YaccError('Unable to build parser')
pycparser.ply.yacc.YaccError: Unable to build parser
when I use the example you gave in test. my code as following:
def test_asm_volatile_2(src):
from pycparserext.ext_c_parser import GnuCParser
p = GnuCParser()
ast = p.parse(src)
ast.show()
from pycparserext.ext_c_generator import GnuCGenerator
print(GnuCGenerator().visit(ast))
src=open('880.cpp','r').read()
test_asm_volatile_2(src)
in which src is a cpp code
could you please give me some advice.
An exception is thrown when trying to instantiate a GnuCParser:
>>> from pycparserext.ext_c_parser import GnuCParser
>>> p = GnuCParser()
ERROR: /home/j0ker/pycparserext/pycparserext/ext_c_lexer.py:11: No regular expression defined for rule 't_FLOAT_CONST'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/j0ker/pycparserext/pycparserext/ext_c_parser.py", line 32, in __init__
self.clex.build()
File "/usr/lib/python3.4/site-packages/pycparser-2.12-py3.4.egg/pycparser/c_lexer.py", line 66, in build
File "/usr/lib/python3.4/site-packages/pycparser-2.12-py3.4.egg/pycparser/ply/lex.py", line 894, in lex
SyntaxError: Can't build lexer
This also leads to 4 of 6 tests failing and makes the whole library unusable.
I am using Python 2.7.9 on an up-to-date Arch Linux. pycparserext and its dependencies were installed from a freshly cloned repo via setup.py.
typedef int wchar_t;
extern wchar_t *wcscpy (wchar_t **restrict __dest,
const wchar_t __restrict __src) __attribute ((nothrow , leaf));
python example.py test1.c
WARNING: /usr/lib/python2.7/dist-packages/pycparser/plyparser.py:41: Rule 'initializer_list_opt' defined, but not used
WARNING: There is 1 unused rule
WARNING: Symbol 'initializer_list_opt' is unreachable
Traceback (most recent call last):
File "example.py", line 12, in <module>
ast = p.parse(text)
File "/home/jamesmikedupont/py/pycparserext/pycparserext/ext_c_parser.py", line 64, in parse
return self.cparser.parse(text, lexer=self.clex, debug=debuglevel)
File "/usr/lib/python2.7/dist-packages/ply/yacc.py", line 269, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/usr/lib/python2.7/dist-packages/ply/yacc.py", line 975, in parseopt_notrack
p.callable(pslice)
File "/home/jamesmikedupont/py/pycparserext/pycparserext/ext_c_parser.py", line 282, in p_declarator_2
% type(p[1]))
NotImplementedError: cannot attach attributes to nodes of type '<class 'pycparser.c_ast.PtrDecl'>'
Hello,
I tried to use the OpenCL parser but there were some small mistakes in the code.
In ext_c_parser.py in the class OpenCLCParser, the first line reads
from pycparserext.ext_c_lexer import GnuCLexer as lexer_class
it should be ---------------------------------------------v v
from pycparserext.ext_c_lexer import OpenCLCLexer as lexer_class
After this it complains about not having the 'ASM' token so I added 'asm' to the lexer keywords in ext_c_lexer.py. That is:
--------------------------------------------------------- v v
add_lexer_keywords(GnuCLexer, [
'attribute', 'asm', 'asm', 'asm', '__typeof',
'real', 'imag', 'builtin_types_compatible_p',
'__const', '__restrict', '__inline', '__inline',
'extension', 'asm'])
_CL_KEYWORDS = ['kernel', 'constant', 'global', 'local', 'private',
"read_only", "write_only", "read_write"]
add_lexer_keywords(OpenCLCLexer, [
'attribute', 'asm', 'asm', 'asm']
+ _CL_KEYWORDS + [""+kw for kw in _CL_KEYWORDS])
Also, I wrote a small OpenCL test program:
def test_OpenCL():
from pycparserext.ext_c_parser import OpenCLCParser, c_ast
src = """
__kernel void zeroMatrix(__global float _A, int n, __global float * B)
{
int i = get_global_id(0);
for (int k=0; k<n; k++)
A[i_n+k] = 0;
}
"""
p = OpenCLCParser()
ast = p.parse(src)
ast.show()
from pycparserext.ext_c_generator import OpenCLCGenerator
print OpenCLCGenerator().visit(ast)
test_OpenCL()
Perhaps you could incorporate these changes.
Best Regards,
Jacob
I added some support to handle Snapdragon processors, header files. Basically, trivial support for volatile I could have used a #define but this seems to be a better method. I think GCC supports both __XXX and XXX for all of those key words so this is really a missing feature.
mhoffman-linux02$ git diff
diff --git a/pycparserext/ext_c_lexer.py b/pycparserext/ext_c_lexer.py
index e501a61..ed7ae67 100644
--- a/pycparserext/ext_c_lexer.py
+++ b/pycparserext/ext_c_lexer.py
@@ -63,8 +63,8 @@ def add_lexer_keywords(cls, keywords):
add_lexer_keywords(GnuCLexer, [
'__attribute__', '__asm__', '__asm', '__typeof__',
'__real__', '__imag__', '__builtin_types_compatible_p',
- '__const', '__restrict', '__inline', '__inline__',
- '__extension__', 'asm', '__attribute'])
+ '__const', '__restrict', '__restrict__', '__inline', '__inline__',
+ '__extension__', 'asm', '__attribute', '__volatile__'])
_CL_KEYWORDS = ['kernel', 'constant', 'global', 'local', 'private',
"read_only", "write_only", "read_write"]
diff --git a/pycparserext/ext_c_parser.py b/pycparserext/ext_c_parser.py
index c9a014a..218d118 100644
--- a/pycparserext/ext_c_parser.py
+++ b/pycparserext/ext_c_parser.py
@@ -359,6 +359,7 @@ class _AsmMixin(object):
def p_asm_volatile(self, p):
""" asm_volatile : VOLATILE
+ | __VOLATILE__
| empty
"""
p[0] = p[1]
@@ -434,6 +435,7 @@ class GnuCParser(_AsmAndAttributesMixin, CParserBase):
def p_type_qualifier_gnu(self, p):
""" type_qualifier : __CONST
| __RESTRICT
+ | __RESTRICT__
| __EXTENSION__
"""
p[0] = p[1]
File "./xxxxyyy.py", line 4, in
import viap_svcomp
File "/Users/X/Downloads/X/viap_svcomp.py", line 88, in
from pycparserext.ext_c_parser import GnuCParser
File "/Users/X/Downloads/X/packages/pycparserext/pycparserext/ext_c_parser.py", line 185, in
Example of this at the bottom. I'm not sure why attributes on arrays are not supported and whether this is a deliberate choice. I can't promise anything soon, but if you're happy to accept a pull request for this I can have a go at implementing it sometime.
By the way, impressive work with this project!
pycparserext: 2013.2
python-ply: 3.4-3ubuntu2
python-pycparser: 2.10+dfsg-1ubuntu2
$ cat foo.c
int x[10] __attribute__((unused));
$ cat test.py
from pycparserext.ext_c_parser import GnuCParser
f = open('foo.c', 'r').read()
ast = GnuCParser().parse(f)
$ python test.py
Traceback (most recent call last):
File "test.py", line 3, in <module>
ast = GnuCParser().parse(f)
File "/home/matthew/.local/lib/python2.7/site-packages/pycparserext/ext_c_parser.py", line 65, in parse
return self.cparser.parse(text, lexer=self.clex, debug=debuglevel)
File "/usr/lib/python2.7/dist-packages/ply/yacc.py", line 269, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/usr/lib/python2.7/dist-packages/ply/yacc.py", line 975, in parseopt_notrack
p.callable(pslice)
File "/home/matthew/.local/lib/python2.7/site-packages/pycparserext/ext_c_parser.py", line 232, in p_declarator_1
% type(p[1]))
NotImplementedError: cannot attach attributes to nodes of type '<class 'pycparser.c_ast.ArrayDecl'>'Pycparser breaks on this example:
diff --git a/test/test_pycparserext.py b/test/test_pycparserext.py
index 183769f..357061c 100644
--- a/test/test_pycparserext.py
+++ b/test/test_pycparserext.py
@@ -245,6 +245,7 @@ def test_array_attributes():
src = """
int x[10] __attribute__((unused));
int y[20] __attribute((aligned(10)));
+ long long z __attribute__((aligned(__alignof__(long long))));
"""
from pycparserext.ext_c_parser import GnuCParserThe error vanishes if I simplify the __attribute__ or replace __alignof__ by (for example) sizeof.
__alignof__ is supported by GNU : https://gcc.gnu.org/onlinedocs/gcc-6.2.0/gcc/Alignment.html
I was not able to engineer a good solution so far. Can we have a generic __attribute__((whatever(foo(long long)))) solution or do we have to manually add __alignof__ keywords and the like?
As the title says, the line number in the Coord object is always set to 0 for Asm nodes. This happens because lineno in the following lines
pycparserext/pycparserext/ext_c_parser.py
Line 300 in b602b4d
pycparserext/pycparserext/ext_c_parser.py
Line 306 in b602b4d
pycparserext/pycparserext/ext_c_parser.py
Line 313 in b602b4d
pycparserext/pycparserext/ext_c_parser.py
Line 320 in b602b4d
tries to get the 'lineno' attribute from p.slice[1] that isn't there so getattr() will return the default value of 0 instead of the actual line number.
I figured out that p.slice[2] contains the appropriate 'lineno' attribute. Changing the last param of the Asm constructor calls from
coord=self._coord(p.lineno(1)))
to
coord=self._coord(p.lineno(2)))
makes the code to return correct file:line position for Asm nodes. Coord.column seems often to be set to None while pycparser fills it with meaningful values.
Would you kindly verify whether that simply fix is okay?
I cannot provide any test case right now because it would require a file with several lines. I confirm that the above mentioned fix works for several real-world C sources I fed in so far.
I'm trying to parse this fragment of C code:
void foo(const int *bar, int * const baz);
using GnuCParser, part of pycparserext.
Based on this answer I expected to see some PtrDecls, but here's what I get from ast.show() on the resulting parse tree:
FileAST:
Decl: foo, [], [], []
FuncDecl:
ParamList:
Decl: bar, ['const'], [], []
TypeDeclExt: bar, ['const']
IdentifierType: ['int']
Decl: baz, [], [], []
TypeDeclExt: baz, []
IdentifierType: ['int']
TypeDecl: foo, []
IdentifierType: ['void']
Notice how baz, a "const pointer to int", doesn't have a trace of const (or "pointeryness") in the data printed by ast.show(). Is this difference because of GnuCParser?
How do I figure out the type of baz from the AST? My actual C code needs the GNU parser. I'm using pycparserext version 2016.1.
This question is a verbatim quote from Stack Overflow; http://stackoverflow.com/q/40094339/28169. I decided to post an issue here since it might be more efficient, and after getting Mr Bendersky's confirmation (in an answer to that question) that the problem seems to be with GnuCParser.
Trying to parse this old file I had laying around as a test case: http://pastebin.com/yYr12cMH
This is the very simple parsing code (just uses pycparser's parse_file with the parser=GnuCParser() kwarg): http://pastebin.com/7Si6e3q1
It fails with this IPython trace: http://pastebin.com/5SFZnXiU
This is some related debugger output: http://pastebin.com/Veb7n9a7
I can link you my exact libio.h file, but unfortunately I don't know what it looks like after pycparser's pre processor call.
Any thoughts? (Is it possible to at least make the parser print which line causes the syntax error, since it keeps the coordinate?)
Do you have a roadmap or an ETA for the next release?
Any issues that you want/need to tackle first?
First of all, thanks for your great project, it helps me very much with parsing microsoft function definitions in their headers.
In spite of the last point of #59 I have some concerns regarding the test suite: it does not seem to check the resultant AST or generated code correctness.
I think it might be worthwhile to add such checks to all the tests. As of now I am considering replacing all the ast.show() and print(GnuCGenerator().visit(ast)) in the tests with checks against known good values.
The problem I see is that ast.show() results are incomplete (they lack attributes in some cases, for example), so it might not be the best idea to use it as-is.
What do you think regarding this issue? Maybe there is a better way to approach this issue?
My preprocessed source has a gazillion things like attribute((nothrow)) from the C library... can we get attribute in function declarations that just have a list of IDs?
I am trying to parse some code which includes libxml2 and it seems to fail on
typedef void *(__attribute__((alloc_size(1))) *xmlMallocFunc)(long unsigned int size);
xmlMallocFunc mallocFunc;
If I take out the __attribute__((alloc_size(1))) this parses fine. I added debug statements and one thing I noticed different was when it worked I saw
Stack : decl_body SEMI . LexToken(TYPEID,'xmlMallocFunc',2,120)
and when it did not work I saw
Stack : decl_body SEMI . LexToken(ID,'xmlMallocFunc',2,151)
the difference being TYPEID vs ID. I'm not sure if this is the root of the problem but I'm not sure where the ID is coming from vs TYPEID and was hoping you could help me here.
Hello,
I'm currently trying to investigate an issue related to the build process of another tool and I'm getting these errors related to pycparserduring the build:
$ /usr/local/bin/python2 tools/generate_pins.py NUCLEO_F207ZG
ERROR: /usr/local/lib/python2.7/site-packages/pycparser/c_parser.py:630: Symbol 'id_init_declarator_list_opt' used, but not defined as a token or a rule
ERROR: /usr/local/lib/python2.7/site-packages/pycparser/c_parser.py:709: Symbol 'declaration_specifiers_no_type_opt' used, but not defined as a token or a rule
ERROR: /usr/local/lib/python2.7/site-packages/pycparser/c_parser.py:714: Symbol 'declaration_specifiers_no_type_opt' used, but not defined as a token or a rule
ERROR: /usr/local/lib/python2.7/site-packages/pycparser/c_parser.py:719: Symbol 'declaration_specifiers_no_type_opt' used, but not defined as a token or a rule
WARNING: /usr/local/lib/python2.7/site-packages/pycparser/plyparser.py:42: Rule 'specifier_qualifier_list_opt' defined, but not used
WARNING: /usr/local/lib/python2.7/site-packages/pycparser/plyparser.py:42: Rule 'declaration_specifiers_opt' defined, but not used
WARNING: There are 2 unused rules
WARNING: Symbol 'id_init_declarator' is unreachable
WARNING: Symbol 'id_init_declarator_list' is unreachable
WARNING: Symbol 'specifier_qualifier_list_opt' is unreachable
WARNING: Symbol 'declaration_specifiers_opt' is unreachable
ERROR: Infinite recursion detected for symbol 'direct_declarator'
ERROR: Infinite recursion detected for symbol 'declaration_specifiers_no_type'
Traceback (most recent call last):
File "tools/generate_pins.py", line 247, in <module>
main()
File "tools/generate_pins.py", line 237, in main
pins = enumerate_pins(pins_file, ['./tools'] + list(includes), defines)
File "tools/generate_pins.py", line 154, in enumerate_pins
parser=GnuCParser())
File "/usr/local/lib/python2.7/site-packages/pycparserext/ext_c_parser.py", line 47, in __init__
debug=yacc_debug, write_tables=False)
File "/usr/local/lib/python2.7/site-packages/pycparser/ply/yacc.py", line 3426, in yacc
raise YaccError('Unable to build parser')
pycparser.ply.yacc.YaccError: Unable to build parser
As @eliben pointed it out in eliben/pycparser#208, this seems more related to pycparserext, do you have any advice to investigate this issue any further?
Hello again,
The patch for #19 worked -- thank you for the quick fix!
pycparserext seems to be appending an empty __attribute__ to declarations of functions that return pointer types (*). This is similar to #19 but slightly different. This bug can be reproduced by this POC with pycparserext installed from the master branch:
#!/usr/bin/env python
from pycparserext import ext_c_parser, ext_c_generator
src = '''
char* foo() {
return "";
}
int main() {
return 0;
}
'''
parser = ext_c_parser.GnuCParser()
ast = parser.parse(src)
gen = ext_c_generator.GnuCGenerator()
print gen.visit(ast)
Running this program results in the following output:
char *foo() __attribute__(())
{
return "";
}
int main()
{
return 0;
}
Add a file named LICENSE at the base of this repository so that users and contributors know how you would like the software licensed (in some counties, an implicit copyright is assumed, so this is important).
pycparserext fails to reduce function declarations of the form
void* foo(int a, int b) attribute((nonnull(1));
The script crashed with the following bt:
Traceback (most recent call last):
File "", line 1, in
File "pycparserext/ext_c_parser.py", line 64, in parse
return self.cparser.parse(text, lexer=self.clex, debug=debuglevel)
File "/home/vijay/script/pycparser/pycparser/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/home/vijay/script/pycparser/pycparser/ply/yacc.py", line 971, in parseopt_notrack
p.callable(pslice)
File "/home/vijay/script/pycparser/pycparser/c_parser.py", line 638, in p_decl_body
typedef_namespace=True)
File "/home/vijay/script/pycparser/pycparser/c_parser.py", line 389, in _build_declarations
self._parse_error('Invalid declaration', coord)
File "/home/vijay/script/pycparser/pycparser/plyparser.py", line 55, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: ?: Invalid declaration
A simple script to reproduce the issue:
from pycparserext.ext_c_parser import GnuCParser
src="""void* memcpy(const void *a, const void *b, int len) attribute((nonnull(1))) attribute((nonnull(2)));"""
p = GnuCParser()
p.parse(src).show()
However the parser succeeds If the declaration doesn't contain pointer as return type
src="""void foo(int a, int b) attribute((nonnull(1)));"""
p.parse(src).show()
FileAST:
Decl: foo, [], [], []
FuncDecl:
ParamList:
Decl: a, [], [], []
TypeDecl: a, []
IdentifierType: ['int']
Decl: b, [], [], []
TypeDecl: b, []
IdentifierType: ['int']
TypeDeclExt: foo, []
IdentifierType: ['void']
I haven't found time to figure out the exact production that is reducing the expression in the above two cases. It will be great if you can fix this or point out the error.
typedef struct {
__attribute__((__deprecated__)) int test1;
} test2;
fails with plyparser.ParseError: :3:9: before: __attribute__
PLY: PARSE DEBUG START
State : 0
Stack : . LexToken(TYPEDEF,'typedef',2,5)
Action : Shift and goto state 65
State : 65
Stack : TYPEDEF . LexToken(STRUCT,'struct',2,13)
Action : Reduce rule [storage_class_specifier -> TYPEDEF] with ['typedef'] and goto state 22
Result : <str @ 0x7f027d5ccab0> ('typedef')
State : 22
Stack : storage_class_specifier . LexToken(STRUCT,'struct',2,13)
Action : Reduce rule [empty -> <empty>] with [] and goto state 121
Result : <NoneType @ 0x7f02812c6280> (None)
State : 121
Stack : storage_class_specifier empty . LexToken(STRUCT,'struct',2,13)
Action : Reduce rule [declaration_specifiers_no_type_opt -> empty] with [None] and goto state 124
Result : <NoneType @ 0x7f02812c6280> (None)
State : 124
Stack : storage_class_specifier declaration_specifiers_no_type_opt . LexToken(STRUCT,'struct',2,13)
Action : Reduce rule [declaration_specifiers_no_type -> storage_class_specifier declaration_specifiers_no_type_opt] with ['typedef',None] and goto state 26
Result : <dict @ 0x7f027d837500> ({'qual': [], 'storage': ['typedef'], 'ty ...)
State : 26
Stack : declaration_specifiers_no_type . LexToken(STRUCT,'struct',2,13)
Action : Shift and goto state 73
State : 73
Stack : declaration_specifiers_no_type STRUCT . LexToken(LBRACE,'{',2,20)
Action : Reduce rule [struct_or_union -> STRUCT] with ['struct'] and goto state 39
Result : <str @ 0x7f027d5cd7d0> ('struct')
State : 39
Stack : declaration_specifiers_no_type struct_or_union . LexToken(LBRACE,'{',2,20)
Action : Shift and goto state 145
State : 145
Stack : declaration_specifiers_no_type struct_or_union LBRACE . LexToken(__ATTRIBUTE__,'__attribute__',3,30)
Action : Reduce rule [brace_open -> LBRACE] with ['{'] and goto state 148
Result : <str @ 0x7f02813fc070> ('{')
State : 148
Stack : declaration_specifiers_no_type struct_or_union brace_open . LexToken(__ATTRIBUTE__,'__attribute__',3,30)
ERROR: Error : declaration_specifiers_no_type struct_or_union brace_open . LexToken(__ATTRIBUTE__,'__attribute__',3,30)
I'm trying to fix, but don't understand why function_specifier doesn't hit here, @inducer if you'd have a moment to take a quick look and point me into a right direction, I'd provide a PR asap.
Hi,
I like this software though I found an issue when parsing a preprocessed file with stddef.h included.
The software seems to have issues with:
typedef struct {
long long __max_align_ll __attribute__((__aligned__(__alignof__(long long))));
long double __max_align_ld __attribute__((__aligned__(__alignof__(long double))));
} max_align_t;
My code:
class FunctionList(pycparser.c_ast.NodeVisitor):
def __init__( self, source):
self.funcs = set()
parser = ext_c_parser.GnuCParser()
self.visit(parser.parse(source))
def visit_FuncDef(self, node):
self.funcs.add(node.decl.name)
With source as the preprocessed file with args = ['gcc','-E','-P' ]
Any solutions to this issue?
Thanks!
The following test code is generating an assertion error:
from pycparserext import GnuCParser
text = """
struct a {
int *b[1][1];
};
"""
parser = GnuCParser()
parser.parse(text, "test.c")
Traceback (most recent call last):
File "test.py", line 10, in
parser.parse(text, "test.c")
File "/home/jdevine/projects/python/pycparserext/ext_c_parser.py", line 64, in parse
return self.cparser.parse(text, lexer=self.clex, debug=debuglevel)
File "/home/jdevine/projects/python/pycparser/ply/yacc.py", line 331, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
File "/home/jdevine/projects/python/pycparser/ply/yacc.py", line 1106, in parseopt_notrack
p.callable(pslice)
File "/home/jdevine/projects/python/pycparserext/ext_c_parser.py", line 280, in p_declarator_2
decl.type = TypeDeclExt.from_pycparser(decl.type)
File "/home/jdevine/projects/python/pycparserext/ext_c_parser.py", line 162, in from_pycparser
assert isinstance(td, c_ast.TypeDecl)
AssertionError
When i want to generate AST of a function, it displays the following error mesages:
Traceback (most recent call last): File "slice_uniq.py", line 227, in <module> split_if(path) File "slice_uniq.py", line 131, in split_if ast = parser.parse(cont, filename='<none>')#, debuglevel=1) File "/usr/local/lib/python3.6/dist-packages/pycparserext-2021.1-py3.6.egg/pycparserext/ext_c_parser.py", line 33, in parse File "/usr/local/lib/python3.6/dist-packages/pycparser/ply/yacc.py", line 331, in parse return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc) File "/usr/local/lib/python3.6/dist-packages/pycparser/ply/yacc.py", line 1118, in parseopt_notrack p.callable(pslice) File "/usr/local/lib/python3.6/dist-packages/pycparser/plyparser.py", line 126, in param_rule func(self, p) File "/usr/local/lib/python3.6/dist-packages/pycparserext-2021.1-py3.6.egg/pycparserext/ext_c_parser.py", line 427, in p_xxx_declarator_2 NotImplementedError: cannot attach asm or attributes to nodes of type '<class 'pycparser.c_ast.PtrDecl'>'
I guess it's because the return value of the function is a pointer. I want to define it in the command line like "-D'char*=char'"
However, another error occurred：
Traceback (most recent call last): File "slice_uniq.py", line 227, in <module> split_if(path) File "slice_uniq.py", line 131, in split_if ast = parser.parse(cont, filename='<none>')#, debuglevel=1) File "/usr/local/lib/python3.6/dist-packages/pycparserext-2021.1-py3.6.egg/pycparserext/ext_c_parser.py", line 33, in parse File "/usr/local/lib/python3.6/dist-packages/pycparser/ply/yacc.py", line 331, in parse return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc) File "/usr/local/lib/python3.6/dist-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack tok = call_errorfunc(self.errorfunc, errtoken, self) File "/usr/local/lib/python3.6/dist-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc r = errorfunc(token) File "/usr/local/lib/python3.6/dist-packages/pycparser/c_parser.py", line 1934, in p_error column=self.clex.find_tok_column(p))) File "/usr/local/lib/python3.6/dist-packages/pycparser/plyparser.py", line 67, in _parse_error raise ParseError("%s: %s" % (coord, msg)) pycparser.plyparser.ParseError: /home/nkamg/ifcut/pycparser/utils/fake_libc_include/_fake_typedefs.h:176:17: before: *
How can I solve this problem？
When I was tried to parse source with several asm instructions (described at http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html)
I faced with fails to parse similar pieces:
1 . void read_tsc(void) {
long val;
asm("rdtsc" : "=A" (val));
}
whereas asm(''nothing") mistakenly recognized as function call.
2 . void read_tsc(void) {
long val;
asm volatile("rdtsc" : "=A" (val));
}
3 . void read_tsc(void) {
long fpenv;
asm(""mtfsf 255,%0"
: :
"f" (fpenv));
}
I suppose that highlighted parts are different states for context-free grammar used by pycparserext in whole.
export PYTHONPATH="/home/ebolsur/desktop/pycparser/lib/python2.7/site-packages/"
python setup.py install --prefix /home/ebolsur/desktop/pycparser
running install
.
.
.
.
.
.
.
Installed /home/ebolsur/desktop/pycparser/lib/python2.7/site-packages/pycparserext-2016.2-py2.7.egg
Processing dependencies for pycparserext==2016.2
Searching for ply>=3.4
Reading http://pypi.python.org/simple/ply/
Couldn't find index page for 'ply' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for ply>=3.4
Best match: None
Traceback (most recent call last):
File "setup.py", line 30, in
packages=["pycparserext"])
File "/app/python/2.7.3/LMWP3/lib/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/app/python/2.7.3/LMWP3/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/app/python/2.7.3/LMWP3/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/install.py", line 76, in run
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/install.py", line 104, in do_egg_install
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 211, in run
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 427, in easy_install
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 478, in install_item
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 519, in process_distribution
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 563, in resolve
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 799, in best_match
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 811, in obtain
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 434, in easy_install
File "/app/python/2.7.3/LMWP3/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/package_index.py", line 475, in fetch_distribution
AttributeError: 'NoneType' object has no attribute 'clone'
Hi,
Thanks for this project!
I see that you've implemented designated range initializers. Would you consider implementing the somewhat related case ranges (https://gcc.gnu.org/onlinedocs/gcc/Case-Ranges.html)?
I am using pycparserext to parse, edit and then re-generate OpenCL kernels. During the re-generation phase, with some kernels, I get the following error:
File "<path>/test.py", line 131, in instrument_file
f.write(gen.visit(ast))
File "<path>/env/lib/python3.7/site-packages/pycparser/c_generator.py", line 28, in visit
return getattr(self, method, self.generic_visit)(node)
File "<path>/env/lib/python3.7/site-packages/pycparserext/ext_c_generator.py", line 155, in visit_FileAST
s += self.visit(ext)
File "<path>/env/lib/python3.7/site-packages/pycparser/c_generator.py", line 28, in visit
return getattr(self, method, self.generic_visit)(node)
... (long list of calls here) ...
File "<path>/env/lib/python3.7/site-packages/pycparser/c_generator.py", line 122, in visit_Cast
s = '(' + self._generate_type(n.to_type, emit_declname=False) + ')'
TypeError: _generate_type() got an unexpected keyword argument 'emit_declname'
This error happens only when a call to visit_Cast occurs somewhere along the way.
Whats more, the only place where pycparserext is being used (and not pycparser) seems to be the second call in the trace above. Note that gen comes from pycparserext. From somewhere near the start of my script:
from pycparserext.ext_c_generator import OpenCLCGenerator
...
gen = OpenCLCGenerator()
It seems to me that your _generate_type() is called and not the one from pycparser, despite the fact that the trace says otherwise, given that it is the implementation of _generate_type() from pycparserext (and not from pycparser) that does not have the keyword argument emit_declname.
Parserext fails when GCC attributes are attached as suffixes instead of prefix. Is this a known issue ?
At the end of this note is a small C snippet that can be used to reproduce the below crash.
BackTrace:
Traceback (most recent call last):
File "gcc_E.py", line 37, in
main(sys.argv[1:])
File "gcc_E.py", line 33, in main
cpp_args=cpp_args)
File "/root/pycparser/pycparser/pycparser/init.py", line 96, in parse_file
return parser.parse(text)
File "/root/pycparser/pycparserext-master/pycparserext/ext_c_parser.py", line 64, in parse
return self.cparser.parse(text, lexer=self.clex, debug=debuglevel)
File "/root/pycparser/pycparser/pycparser/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/root/pycparser/pycparser/pycparser/ply/yacc.py", line 971, in parseopt_notrack
p.callable(pslice)
File "/root/pycparser/pycparserext-master/pycparserext/ext_c_parser.py", line 245, in p_declarator_1
% type(p[1]))
NotImplementedError: cannot attach attributes to nodes of type '<class 'pycparser.c_ast.FuncDecl'>
Code snippet:
extern void int happy(void) attribute((unused));
int main()
{
}
I have written the following program using pycparser and pycparserext:
from pycparser import parse_file,c_parser, c_ast, c_generator
from pycparserext.ext_c_parser import GnuCParser
content="int main() { int x = 1; int y = 0; while (y < 1000 && __VERIFIER_nondet_int()) { x = x + y; y = y + 1; } __VERIFIER_assert(x >= y); return 0;}"
text = r""" """+content
parser = GnuCParser()
ast = parser.parse(text)
generator = c_generator.CGenerator()
print str(generator.visit(ast))When I run the code in Mac it returns the correct output. But when I run the same code in Ubuntu 16.04.3 it returns the following incorrect output (that is missing the 'main()':
int{
int x = 1;
int y = 0;
while ((y < 1000) && __VERIFIER_nondet_int())
{
x = x + y;
y = y + 1;
}
__VERIFIER_assert(x >= y);
return 0;
}
What is causing this incorrect output?
Is there any functionality in pycparserext regarding basic blocks? I.e. parsing the input file and getting a list of basic blocks for each function, or a specific function. If not, is there any way to do this easily with the existing functionalities? If not again, would you be interested in me trying to implement this feature and send a PR?
This is a GNU extension, see Designated Inits in the GCC manual. However, the GnuCParser doesn't cope:
>>> from pycparserext.ext_c_parser import GnuCParser
>>> p=GnuCParser()
>>> p.parse("int widths[] = { [0 ... 9] = 1, [10 ... 99] = 2, [100] = 3 };")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python27\lib\site-packages\pycparserext\ext_c_parser.py", line 64, in parse
return self.cparser.parse(text, lexer=self.clex, debug=debuglevel)
File "C:\Program Files (x86)\Python27\lib\site-packages\pycparser\ply\yacc.py", line 331, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
File "C:\Program Files (x86)\Python27\lib\site-packages\pycparser\ply\yacc.py", line 1181, in parseopt_notrack
tok = call_errorfunc(self.errorfunc, errtoken, self)
File "C:\Program Files (x86)\Python27\lib\site-packages\pycparser\ply\yacc.py", line 193, in call_errorfunc
r = errorfunc(token)
File "C:\Program Files (x86)\Python27\lib\site-packages\pycparser\c_parser.py", line 1721, in p_error
column=self.clex.find_tok_column(p)))
File "C:\Program Files (x86)\Python27\lib\site-packages\pycparser\plyparser.py", line 55, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: :1:21: before: ...
Tested with pycparserext 2016.2.
No idea if this is supposed to be supported, but I had an instance of this in my code so I ran into it.
Hello,
pycparserext seems to be inserting an empty __attribute__ list when * should occur. This can be reproduced by this POC with pycparserext installed from the master branch:
#!/usr/bin/env python
from pycparserext import ext_c_parser, ext_c_generator
src = '''
struct foo {
const char *bar;
const char *baz;
};
int main () {
return 0;
}
'''
parser = ext_c_parser.GnuCParser()
ast = parser.parse(src)
gen = ext_c_generator.GnuCGenerator()
print gen.visit(ast)
Running this program results in the following output:
struct foo
{
const char bar __attribute__(());
const char baz __attribute__(());
};
int main()
{
return 0;
}
I'm not very familiar with the codebase, but for what its worth I think the bug may occur during parsing rather than generation.
Thanks for all the good work on this project so far.
Hi,
given this simple test file:
$ cat kernel.cl
void A() {}The output of parse_file() is different when using the OpenCLCparser vs the pycparser default:
>>> parse_file("kernel.cl", use_cpp=True).children()
(('ext[0]', <pycparser.c_ast.FuncDef at 0x7f986ea2e2c8>),)
>>> parse_file("kernel.cl", use_cpp=True, parser=OpenCLCParser()).children()
(('ext[0]', [<pycparserext.ext_c_parser.PreprocessorLine at 0x7f986ea31240>]),
('ext[1]', [<pycparserext.ext_c_parser.PreprocessorLine at 0x7f986ea314e0>]),
('ext[2]', [<pycparserext.ext_c_parser.PreprocessorLine at 0x7f986ea271d0>]),
('ext[3]', [<pycparserext.ext_c_parser.PreprocessorLine at 0x7f986eacc898>]),
('ext[4]', [<pycparserext.ext_c_parser.PreprocessorLine at 0x7f986ea31470>]),
('ext[5]', [<pycparserext.ext_c_parser.PreprocessorLine at 0x7f986ea31128>]),
('ext[6]', <pycparser.c_ast.FuncDef at 0x7f986ed6b908>))This breaks the behaviour of the show() method:
>>> parse_file("kernel.cl", use_cpp=True, parser=OpenCLCParser()).show()
Traceback (most recent call last)
...
AttributeError: 'list' object has no attribute 'show'What is the meaning of the PreprocessorLine objects? Is there a way I can disable their creation, or have I misunderstood the API?
$ pip freeze | grep pycp
pycparser==2.17
pycparserext==2016.2I have a .h file which I am trying to parse.
#include <stdio.h>
#define MAX_VAL 100
struct {
int x;
char a[MAX_VAL];
} mahesh_t;
after preprocessing it I get the following lines
cpp -E test.h > test.hh
excerpt from the file:
#1 "/usr/lib/gcc/x86_64-redhat-linux/4.4.6/include/stddef.h" 1 3 4
#211 "/usr/lib/gcc/x86_64-redhat-linux/4.4.6/include/stddef.h" 3 4
typedef long unsigned int size_t;
#35 "/usr/include/stdio.h" 2 3 4
when I try to run the script I get this error
$ python c-to-c.py ./test.hh
Traceback (most recent call last):
File "c-to-c.py", line 60, in <module>
translate_to_c(sys.argv[1])
File "c-to-c.py", line 24, in translate_to_c
ast = parse_file(filename, use_cpp=True)
File "/usr/lib/python2.6/site-packages/pycparser/__init__.py", line 93, in parse_file
return parser.parse(text, filename)
File "/usr/lib/python2.6/site-packages/pycparser/c_parser.py", line 138, in parse
debug=debuglevel)
File "/usr/lib/python2.6/site-packages/pycparser/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/usr/lib/python2.6/site-packages/pycparser/ply/yacc.py", line 1047, in parseopt_notrack
tok = self.errorfunc(errtoken)
File "/usr/lib/python2.6/site-packages/pycparser/c_parser.py", line 1613, in p_error
column=self.clex.find_tok_column(p)))
File "/usr/lib/python2.6/site-packages/pycparser/plyparser.py", line 54, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: /usr/include/_G_config.h:53:24: before: __attribute__
Any ideas on how do I fix this?
What wrong am I doing?
thanks
Is it possible to add support for the following syntax:
register typeName variableName asm("registerName");
Best regards,
Martin
I'm just trying out pycparser and pycparserext for the first time. I'm using python2.7 on 64bit Linux.
I ran the self test and hit:
self = <pycparserext.ext_c_generator.GnuCGenerator object at 0x1a15050>, node = [<pycparser.c_ast.FuncDef object at 0x19fc950>]
def generic_visit(self, node):
#~ print('generic:', type(node))
if node is None:
return ''
else:
return ''.join(self.visit(c) for c in node.children())E AttributeError: 'list' object has no attribute 'children'
../../../.local/lib/python2.7/site-packages/pycparserext-2012.1-py2.7.egg/pycparserext/c_generator.py:31: AttributeError
Hi, tried to parse the Linux Kernel Source by this program, in combination with your extension:
import os
import sys
import pprint
import mmap
import pycparser
from __future__ import print_function
temp_path="/public/linux-kernel/linux-2.6.32.41"
sys.path.extend(['.', '..'])
from pycparser import c_parser, c_ast, parse_file
class FuncDefVisitor(c_ast.NodeVisitor):
def visit_FuncDef(self, node):
print('%s at %s' % (node.decl.name, node.decl.coord))
def show_func_defs(filename):
# Note that cpp is used. Provide a path to your own cpp or
# make sure one exists in PATH.
ast = parse_file(filename, use_cpp=True,\
cpp_args=r'-Iutils/fake_libc_include')
v = FuncDefVisitor()
v.visit(ast)
if __name__ == "__main__":
start_dir=""
if len(sys.argv) > 1:
start_dir = sys.argv[1]
if len(start_dir) == 0:
start_dir=temp_path
for folder, subs, files in os.walk(start_dir):
for filename in files:
fpath=os.path.join(folder, filename)
filename_n, file_extension = os.path.splitext(fpath)
print file_extension
if file_extension in [ ".c", ".cc", ".cpp", ".h", ".hpp" ]:
show_func_defs(fpath)
But got as result: (file /linux-kernel/linux-2.6.32.41/Documentation/accounting/getdelays.c)
Traceback (most recent call last):
File "func_defs.py", line 61, in <module>
show_func_defs(fpath)
File "func_defs.py", line 42, in show_func_defs
cpp_args=r'-Iutils/fake_libc_include')
File "/usr/lib/python2.7/dist-packages/pycparser/__init__.py", line 93, in parse_file
return parser.parse(text, filename)
File "/usr/lib/python2.7/dist-packages/pycparser/c_parser.py", line 138, in parse
debug=debuglevel)
File "/usr/lib/python2.7/dist-packages/ply/yacc.py", line 269, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/usr/lib/python2.7/dist-packages/ply/yacc.py", line 1051, in parseopt_notrack
tok = self.errorfunc(errtoken)
File "/usr/lib/python2.7/dist-packages/pycparser/c_parser.py", line 1613, in p_error
column=self.clex.find_tok_column(p)))
File "/usr/lib/python2.7/dist-packages/pycparser/plyparser.py", line 54, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: /usr/lib/gcc/x86_64-linux-gnu/4.9/include/stdarg.h:40:27: before: __gnuc_va_list
Unfortunately Benderski closed the issue post before I got any new answer. Just don't want to dig into the code of the parser and the extension, so what's wrong - Thanks!
Support for either of these cases:
struct foo_t {
struct __attribute__((packed)) {
int a;
char b;
};
};
struct foo_t {
struct {
int a;
char b;
} __attribute__((packed));
};
I can work on an patch, but just posting an issue so it's documented.
Using GnuCParser() and subclassing c_ast.NodeVisitor, I cannot traverse the nodes as in the FuncDefs example in https://github.com/eliben/pycparser/blob/master/examples/func_defs.py
It results in the following error:
Traceback (most recent call last):
[...]
File "main_script.py", line 77, in verify
FuncDefVisitor().visit(self.ast)
File "/path/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
return visitor(node)
File "/path/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 165, in generic_visit
self.visit(c)
File "/path/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
return visitor(node)
File "/path/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 165, in generic_visit
self.visit(c)
File "/path/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
return visitor(node)
File "/path/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 164, in generic_visit
for c in node:
TypeError: 'FuncDeclExt' object is not iterable
I work on python3.8.6 x64 version, windows 7 64bit OS, I use pip to install pycparser 2.20, PLY 3.11 , then git clone pycparserext and python setup.py install. Is there anything I missed?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.