haoxiang47 / ply Goto Github PK

What steps will reproduce the problem?
Adding some @Decorator to t_RULE funcs completely breaks the funcs order which 
is significant for correct lexing and parsing.


What is the expected output? What do you see instead?
Lexing rule order is broken.

What version of the product are you using? On what operating system?
PLY 3.3 with Python 2.5.4 on MacOs 10.6.4 (but I think OS doesn't matter)


Please provide any additional information below.
I don't try how it works in Python 3 because I need to use Python 2.5 (but I 
guess that problem will be the same in py3) .
In Python 2.5 the problem reproduced.
The root cause of the problem is in ordering code of lex.py - it uses the 
following func_code() which return src code obj for specified func f, where f - 
actually is our t_RULE func (and later attr co_firstlineno of this obj - number 
of 1-st line of func is used for sorting)

if sys.version_info[0] < 3:
    def func_code(f):
        return f.func_code
else:
    def func_code(f):
        return f.__code__

so the problem is that when I use decorator:
@MyDecorator
t_MYRULE(t)
and if this MyDecorator actually decorates (substitutes) the t_MYRULE then 
f.func_code will return the src code object of this decorator-func but not 
original t_MYRULE. This breaks correct ordering of t_RULE funcs.

Unfortunately attr co_firstlineno is readonly, so possible workaround - to 
assign 
decorated_func.func_code.co_firstlineno=original_func.func_code.co_firstlineno 
- doesn't work.

Does somebody know workaround for this problem?
Thanks in advance.

What steps will reproduce the problem?
It isn't a problem, but enhancement proposal. Now lexing always starts in 
predefined INITIAL state. There is no way to start from my own state.

What version of the product are you using? On what operating system?
PLY 3.3

Please provide any additional information below.
As I saw from lex src code - the lexing always starts in INITIAL state. And if 
my grammar has many state and there no common rules which I can place in 
INITIAL state (or may be I just don't want to identify these common rules). So 
in other words - I identified my states and named them all - how can I start 
lexing in my own START state instead of predefined INITIAL state?

What steps will reproduce the problem?
1. Make a simple grammar with non-assoc, such as
exp -> exp LT exp
   | INT
with LT having non-assoc precedence
2. Lexer supplies INT LT INT LT INT
3. Run parser

What is the expected output? What do you see instead?

Expected output: parse error
Actual output: parses without complaint

What version of the product are you using? On what operating system?

ply-2.3 on linux (arch/os doesn't matter) 

Please provide any additional information below.

The bug is on line 1746 of yacc.py:

                                    if (slevel > rlevel) or ((slevel ==
rlevel) and (rprec != 'left')): # BUG HERE BUG HERE
                                        # We decide to shift here...
highest precedence to shift
                                        st_action[a] = j
                                        st_actionp[a] = p
                                        if not rlevel:
                                            n_srconflict += 1
                                            _vfc.write("shift/reduce
conflict in state %d resolved as shift.\n" % st)
                                            _vf.write("  ! shift/reduce
conflict for %s resolved as shift.\n" % a)
                                    elif (slevel == rlevel) and (rprec ==
'nonassoc'):
                                        st_action[a] = None

Brief visual inspection should confirm that the elif branch a few lines
down cannot ever execute, because if slevel=rlevel and rprec='nonassoc'
then the if branch on 1746 will execute instead. The solution is to change
line 1746 to:

                                    if (slevel > rlevel) or ((slevel ==
rlevel) and (rprec == 'right')):


Note that this also brings it more in line with the 1691 (except that on
1746 we want to shift on right associativity, not reduce on left as in 1691). 

I am attaching a patch that appears to fix the problem. 

While doing the braces example I discoverd some minor issues.
I took the liberty to correct them and to enhance the example into a working 
script (see below and attachment)

I don't know if the documentation is part of the open source project on google 
code so I sent a mail to Dave Baezly aswell


CHANGES:
--------------------------------------------------------------------------------
-----------------------------------------------------------
OLD  t.value = t.lexer.lexdata[t.lexer.code_start:t.lexer.lexpos+1]
NEW t.value = t.lexer.lexdata[t.lexer.code_start:t.lexer.lexpos-1]

OLD  r'(/\*(.|\n)*?*/)|(//.*)'
NEW r'\"([^\\\n]|(\\.))*?\"'

thanks to: http://d.hatena.ne.jp/noritsugu/20080401
--------------------------------------------------------------------------------
-----------------------------------------------------------

SCRIPT:
# ----------------------------------------
# bracelex.py
# 
# Demonstrates how to grab arbitray C code enclosed by braces.
# It resolves nested braces 
# and ignores braces in comments ( /* */ , //) 
# and strings (" ",' ')
# ----------------------------------------
# Import
import ply.lex as lex
import re
tokens = (
    'CCODE',
)
# Declare the state
states = (
  ('ccode','exclusive'),
)
# Match the first {. Enter ccode state.
def t_ccode(t):
    r'\{'
    t.lexer.code_start = t.lexer.lexpos        # Record the starting 
position
    t.lexer.level = 1                          # Initial brace 
level
    t.lexer.begin('ccode')                     # Enter 'ccode' 
state
# Rules for the ccode state
def t_ccode_lbrace(t):     
    r'\{'
    t.lexer.level +=1                
def t_ccode_rbrace(t):
    r'\}'
    t.lexer.level -=1
    # If closing brace, return the code fragment
    if t.lexer.level == 0:
         t.value = t.lexer.lexdata[t.lexer.code_start:t.lexer.lexpos-1]
         t.type = "CCODE"
         t.lexer.lineno += t.value.count('\n')
         t.lexer.begin('INITIAL')           
         return t
#C or C++ comment (ignore)    
def t_ccode_comment(t):
    r'(/\*(.|\n)*?\*/)|(//.*)'
    pass
# C string
def t_ccode_string(t):
   r'\"([^\\\n]|(\\.))*?\"'
# C character literal
def t_ccode_char(t):
   r'\'([^\\\n]|(\\.))*?\''
# Any sequence of non-whitespace characters (not braces, strings)
def t_ccode_nonspace(t):
   r'[^\s\{\}\'\"]+'
# Ignored characters (whitespace)
t_ccode_ignore = " \t\n"
# For bad characters, we just skip over it
def t_ccode_error(t):
    t.lexer.skip(1)
# Mandatory Error handling rule
def t_error(t):
    #print "?:%s" % t.value[0]
    t.lexer.skip(1)
    
#Execute
data = '''
// {this is some comment between braces}
function () {
    // {this is a single line comment between braces}
    this is interessting code
    /* {this is another single line comment between braces} */
    /*
    {this is a multiline comment between braces}
    */
    { code between nested braces 1
        { code between nested braces 2 }
    }
    '{string between braces}'
    "{anaother string between braces}"
    
}
'''
# create lexer
lexer = lex.lex(reflags = re.IGNORECASE)
    
# Tokenize (case insensitive matches)
lexer.input(data)
for tok in lexer:
    print tok.type.rjust(24), tok.value
    #print tok.type, tok.value, tok.lineno, tok.lexpos

What steps will reproduce the problem?
1. install python 3
2. import lex or yacc v.2.5

The new python implementation does no longer support '%' string formatting,
and no longer support raise, print, exec, etc. with unbracketed arguments.

I've found a problem with ply2.3 related to parsing table reloading. I am
using python2.5.1 (on windows2K :().

When we want to read the parsing tables from a dotted module name like
'pckg.subpckg.parsetab' (by adding 'tabmodule = "pckg.subpckg.parsetab" to
the parser constructor), the lr_read_tables function executes 'import
pckg.subpckg.parsetab as parsetab'. This is fairly ok, but that's not the
problem.

The problem is, when yacc writes the tables, it will write to a file named
'pckg.subpckg.parsetab.py' inside the specified outputdir (using the full
module name, and not only parsetab).
This isn't what we intended, so at reload, yacc will search for
'pckg.subpckg.parsetab' but it won't find a file named 'parsetab.py' inside
a 'pckg/subpckg' directory on python's module search path. 

So, instead of loading the tables from the previously generated file, it
will regenerate the parsing tables every time it gets loaded.

I've fixed the bug on my ply source. The solution is rather ugly (to blend
with the rest of yacc's source :P), but the problem seems to be gone. If
anyone's interested in my solution, please contact me be email (cptbinho at
gmail dot com) and i'll gladly send you my modified version.

Regards

What steps will reproduce the problem?
1. Make lexer with lex.lex(debug=True)
2. Run lexer.input('a string'); list(lexer)

What is the expected output? What do you see instead?

The documentation in section 4.13 says that the lexer in debug mode will print 
debug information about the "tokens generating during lexing".  I took this to 
mean that the lexer would log the tokens during the lexing process, but this 
does not occur. 

It's no real problem to print the tokens as they arrive:

for t in lexer:
    print t

but maybe the documentation would benefit from clarification?

What version of the product are you using? On what operating system?

SVN r86 OSX 10.6

Please provide any additional information below.

In yacc.py on line 202, the method set_lineno is

 def set_lineno(self,n,lineno):
     self.slice[n].lineno = n

instead of

 def set_lineno(self,n,lineno):
     self.slice[n].lineno = lineno

This results in the line number to be wrong - in the most common case,
likely ending up as 0.

What steps will reproduce the problem?
1. Use a slice in a production rule, eg:

def p_toplevel(p):
    """ toplevel : alpha beta gamma TOKEN delta """
    p[0] = p[1:4] + p[5:]

What is the expected output? What do you see instead?
Expected: p[0] is a list of the nonterminals alpha, beta, gamma, and delta.
Got:
  File "/usr/lib/python3.1/site-packages/ply/yacc.py", line 198, in __getitem__
    if n >= 0: return self.slice[n].value
TypeError: unorderable types: slice() >= int()

What version of the product are you using? On what operating system?

This is python 3.1, but this has been true since python 3.0: 
http://docs.python.org/py3k/whatsnew/3.0.html#operators-and-special-methods

Namely, instead of __getslice__ being called as it was in python 2.x, 
__getitem__ is called with a slice object, which is being compared with an int 
in YaccProduction's case. (Similarly, my testing shows that even in Python 2.7, 
a slice that includes a step, eg. a[1:4:2], sends a slice object to __getitem__ 
instead of to __getslice__.)

Simple patch included (Python 2 compatible) to fix this, but I note it allows 
the use of negative indices in slices in a way that doesn't match how negative 
indexing currently works.

What steps will reproduce the problem?
1. Run 'python ply_bug.py' (see attachment).

---

What is the expected output? What do you see instead?

The expected output is to have the line number the same each time, as we
parse the same input. However, it seems the line number is not reset each time.

---

What version of the product are you using? On what operating system?

Ply 3.3 on Linux.

---
Please provide any additional information below.

I included a patch (lex_bugfix.patch attachment) that resets the lineno
attribute of the lexer each time the input is set. If you apply the patch
and rerun the example, you will see that the line number of the parse error
is the same each time.

Also note that the lexpos attribute of the lexer is properly reset in the
input() function.

What steps will reproduce the problem?
1. Start calc.py
2. type 2=
3.

What is the expected output? What do you see instead?
Expected: Syntax error at '='
I get:
Syntax error at '='
Traceback (most recent call last):
  File "./calc.py", line 106, in <module>
    yacc.parse(s)
  File "../ply/yacc.py", line 303, in parse
  File "./calc.py", line 84, in p_expression_number
    p[0] = p[1]
  File "../ply/yacc.py", line 121, in __getitem__
IndexError: list index out of range


What version of the product are you using? On what operating system?
UBUNTU and python2.5, ply 2.3

Please provide any additional information below.

In 3.19 (Miscellaneous Issues) the rules that should be followed when
writing an external Lexer w.r.t. the tokens may be extended with the need
to have a 'linno' attribute for getting line number information.
You may also expect column information somewhere, but I never use that, so
I don't know... :)

With PLY 3.4 on Python 3.2 on Windows XP, if I type "pip install python", I get 
this:

Downloading/unpacking ply
  Real name of requirement ply is ply
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\python32\lib\threading.py", line 740, in _bootstrap_inner
    self.run()
  File "C:\python32\lib\threading.py", line 693, in run
    self._target(*self._args, **self._kwargs)
  File "C:\python32\lib\site-packages\pip-1.0.2-py3.2.egg\pip\index.py", line 23
9, in _get_queued_page
    page = self._get_page(location, req)
  File "C:\python32\lib\site-packages\pip-1.0.2-py3.2.egg\pip\index.py", line 32
4, in _get_page
    return HTMLPage.get_page(link, req, cache=self.cache)
  File "C:\python32\lib\site-packages\pip-1.0.2-py3.2.egg\pip\index.py", line 44
5, in get_page
    inst = cls(u(resp.read()), real_url, headers)
  File "C:\python32\lib\site-packages\pip-1.0.2-py3.2.egg\pip\backwardcompat.py"
, line 58, in u
    return s.decode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf8 in position 10783: inval
id start byte

  Downloading ply-2.5.tar.gz (119Kb): 119Kb downloaded
  Running setup.py egg_info for package ply

Installing collected packages: ply
  Running setup.py install for ply
      File "C:\python32\Lib\site-packages\ply\lex.py", line 187
        exec "import %s as lextab" % tabfile
                                 ^
    SyntaxError: invalid syntax

      File "C:\python32\Lib\site-packages\ply\yacc.py", line 174
        raise YaccError, "Can't directly instantiate Parser. Use yacc() instead.
"
                       ^
    SyntaxError: invalid syntax


Successfully installed ply
Cleaning up...


It looks like there are two separate issues: the Unicode error and the exec 
error. I haven't looked into the Unicode error, but lex.py is using exec as a 
statement, not as a function, which is clearly wrong for Python 3.

Using ply-3.3, if I process the following text using cpp.py:

// A comment
#define A_DIRECTIVE

then the directive line is _not_ detected as a directive. This occurs 
because the regex definition of a CPP_COMMENT (line 59) causes the \n to be 
included in the comment token, which then causes Preprocessor.group_lines 
to interpret the directive as part of the previous comment line. 

This can be easily fixed by changing the CPP_COMMENT definition to:
r'(/\*(.|\n)*?\*/)|(//[^\n]*)' 

What steps will reproduce the problem?
1. the regular expression for comment is bad. And comment is not detected.
Token TIMES and DIVIDE are found.
2.
3.

What is the expected output? What do you see instead?
Regular expression in clex.py for comment (line 143 to 146) : 
143 # Comments
144 def t_comment(t):
145     r' /\*(.|\n)*?\*/'
146     t.lineno += t.value.count('\n')

They are a space on first character.

The good regular expression is :
143 # Comments
144 def t_comment(t):
145     r'/\*(.|\n)*?\*/'
146     t.lineno += t.value.count('\n')


What version of the product are you using? On what operating system?
PLY 2.3 on Windows XP

Please provide any additional information below.

It seems there is no EOF rule in PLY right now, to match the end of file, 
dirty hacks must be used.

What version of the product are you using? On what operating system?
V3.3 on Linux

While poking around in the ANSI C example I noticed what looks like a typo 
in the second struct_declaration_list rule.  Instead of:

def p_struct_declaration_list_2(t):
    'struct_declaration_list : struct_declarator_list struct_declaration'
    pass


I think it should be:

    'struct_declaration_list : struct_declaration_list struct_declaration'
                                          ^^^^^^^

Regards,
Phil

What steps will reproduce the problem?
1. I checked out r39 and installed it.
2. ply/cpp.py seems to contain a syntax error.
3. I think the '... for ... in ...' needs '[' and ']' around it...

Dennis Hendriks

Output:

$ python setup.py install
running install
running bdist_egg
running egg_info
creating ply.egg-info
writing ply.egg-info/PKG-INFO
writing top-level names to ply.egg-info/top_level.txt
writing dependency_links to ply.egg-info/dependency_links.txt
writing manifest file 'ply.egg-info/SOURCES.txt'
reading manifest file 'ply.egg-info/SOURCES.txt'
writing manifest file 'ply.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-i686/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/ply
copying ply/__init__.py -> build/lib/ply
copying ply/lex.py -> build/lib/ply
copying ply/yacc.py -> build/lib/ply
copying ply/cpp.py -> build/lib/ply
creating build/bdist.linux-i686
creating build/bdist.linux-i686/egg
creating build/bdist.linux-i686/egg/ply
copying build/lib/ply/__init__.py -> build/bdist.linux-i686/egg/ply
copying build/lib/ply/lex.py -> build/bdist.linux-i686/egg/ply
copying build/lib/ply/yacc.py -> build/bdist.linux-i686/egg/ply
copying build/lib/ply/cpp.py -> build/bdist.linux-i686/egg/ply
byte-compiling build/bdist.linux-i686/egg/ply/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-i686/egg/ply/lex.py to lex.pyc
byte-compiling build/bdist.linux-i686/egg/ply/yacc.py to yacc.pyc
byte-compiling build/bdist.linux-i686/egg/ply/cpp.py to cpp.pyc
  File "build/bdist.linux-i686/egg/ply/cpp.py", line 742
    filename = "".join(x.value for x in tokens[1:i])
                                 ^
SyntaxError: invalid syntax
creating build/bdist.linux-i686/egg/EGG-INFO
copying ply.egg-info/PKG-INFO -> build/bdist.linux-i686/egg/EGG-INFO
copying ply.egg-info/SOURCES.txt -> build/bdist.linux-i686/egg/EGG-INFO
copying ply.egg-info/dependency_links.txt ->
build/bdist.linux-i686/egg/EGG-INFO
copying ply.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/ply-2.4-py2.3.egg' and adding 'build/bdist.linux-i686/egg' to it
removing 'build/bdist.linux-i686/egg' (and everything under it)
Processing ply-2.4-py2.3.egg
Copying ply-2.4-py2.3.egg to <***>/lib/python2.3/site-packages
Adding ply 2.4 to easy-install.pth file

Installed <***>/lib/python2.3/site-packages/ply-2.4-py2.3.egg
Processing dependencies for ply==2.4
Finished processing dependencies for ply==2.4

What steps will reproduce the problem?
1. Make a virtualenv with Python 2.7
2. Take some working example that uses precedence
3. Add `from __future__ import unicode_literals`

What is the expected output?
The behavior should be unchanged from the basic Python 2.7 behavior.

What do you see instead?
The error raise is `YaccError: Unable to build parser` and the verbose output 
is `precedence associativity must be a string`

What version of the product are you using? On what operating system?
PLY 3.4
Python 2.7 (probably true for any 2.X)
OSX 10.7.5 (probably irrelevant)

Please provide any additional information below.
Everything does work fine in Python 3. I am working on a project that should 
support both Python 2 and Python 3 simultaneously,

What steps will reproduce the problem?
1. Run calc.py and submit "10/2".

What is the expected output? What do you see instead?
Expected value is 5, but actual value is "None".

What version of the product are you using? On what operating system?
Copy hosted at http://www.dabeaz.com/ply/example.html, downloaded 3/19/2007.

Please provide any additional information below.
The patched function should read:

def p_expression_binop(t):
    '''expression : expression PLUS expression
                  | expression MINUS expression
                  | expression TIMES expression
                  | expression DIVIDE expression'''
    if t[2] == '+'  : t[0] = t[1] + t[3]
    elif t[2] == '-': t[0] = t[1] - t[3]
    elif t[2] == '*': t[0] = t[1] * t[3]
    elif t[2] == '/': t[0] = t[1] / t[3]

(The last line said "elif t[3] == '/'...")

What steps will reproduce the problem?
1. Use lex and make sure the table file is written.

What is the expected output? What do you see instead?
Line 143 in lex.py: "basetabfilename = tabfile.rsplit(".",1)[-1]". In
python 2.3, the rsplit method doesn't exist. It is new in python 2.4.

What version of the product are you using? On what operating system?
I'm using 2.4 prerelease version on python 2.3.

Please provide any additional information below.
Maybe use a regular split, which reduces performance. Alternative, an
rfind() can be used.

What steps will reproduce the problem?
1. Run lexer with states, where lexer finishes in state other than initial
2. Rerun lexer after lexer.input(txt); continues in previous state rather than 
initial state

See lex_state_history.py (attached)

What is the expected output?

[LexToken(SOMETHINGFIRST,'start@',1,0), LexToken(STATECHANGER,'change@',1,6), 
LexToken(SOMETHINGSTATEY,'later',1,13)]
[LexToken(SOMETHINGFIRST,'start@',1,0), LexToken(STATECHANGER,'change@',1,6), 
LexToken(SOMETHINGSTATEY,'later',1,13)]

What do you see instead?

[LexToken(SOMETHINGFIRST,'start@',1,0), LexToken(STATECHANGER,'change@',1,6), 
LexToken(SOMETHINGSTATEY,'later',1,13)]
Traceback (most recent call last):
  File "lex_state_history.py", line 31, in <module>
    print list(lexer)
  File "/Users/mb312/usr/local/lib/python2.6/site-packages/ply/lex.py", line 405, in next
    t = self.token()
  File "/Users/mb312/usr/local/lib/python2.6/site-packages/ply/lex.py", line 393, in token
    raise LexError("Illegal character '%s' at index %d" % (lexdata[lexpos],lexpos), lexdata[lexpos:])
ply.lex.LexError: Illegal character 's' at index 0

What version of the product are you using? On what operating system?

Current SVN checkout r86 OSX 10.6

Please provide any additional information below.

Suggested patch attached

This was also posted to ply-hack, but without the attached patch.  Sorry
for the duplication.

What steps will reproduce the problem?
1. specify a lexer using # comments and whitespace in regexps (re.VERBOSE).
2. don't pass any reflags to lex.  (But see below).
3. compile a lextab file.
4. do a second run from lextab file.

What is the expected output? What do you see instead?
This works when compiling the regexps from your source file, but fails when
compiling them from the lextab file.  The re.VERBOSE flag is set for the
first case, but not the second case.

What version of the product are you using? On what operating system?
ply 3.2 on linux, with python 2.6.

Please provide any additional information below.
There is actually a second problem because calling lex() with
reflags=re.VERBOSE doesn't get stored in the lextab file, so doesn't fix
this.  I'm supplying a patch to fix this second problem, but the first
problem remains if re.VERBOSE is not passed to lex().

When I type

  easy_install ply

using Python 3.3, I get the following error:

Traceback (most recent call last):
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/bin/easy_install", line 9, in <module>
    load_entry_point('distribute==0.6.27', 'console_scripts', 'easy_install')()
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 1915, in main
    with_ei_usage(lambda:
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 1896, in with_ei_usage
    return f()
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 1919, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/store/store/sandia/python3.3/ver-3.3.0/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/store/store/sandia/python3.3/ver-3.3.0/lib/python3.3/distutils/dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "/store/store/sandia/python3.3/ver-3.3.0/lib/python3.3/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 350, in run
    self.easy_install(spec, not self.no_deps)
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 590, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 620, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 814, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 1094, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/easy_install.py", line 1080, in run_setup
    run_setup(setup_script, args)
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/sandbox.py", line 31, in run_setup
    lambda: exec(compile(open(
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/sandbox.py", line 79, in run
    return func()
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/sandbox.py", line 34, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'})
  File "setup.py", line 29, in <module>
  File "/store/store/sandia/python3.3/ver-3.3.0/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/store/store/sandia/python3.3/ver-3.3.0/lib/python3.3/distutils/dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "/store/store/sandia/python3.3/ver-3.3.0/lib/python3.3/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 227, in run
    os.path.join(archive_root,'EGG-INFO'), self.zip_safe()
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 266, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 402, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/home/hudson/slave/workspace/PyUtilib_trunk_python3.3/python/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 429, in scan_module
    code = marshal.load(f);  f.close()
ValueError: bad marshal data (unknown type code)

Error installing package ply with easy_install


I'm installing on a 64-bit Linux box with RH 6.

--Bill

What steps will reproduce the problem?
1. after python setup.py install, run test/rununit.py
2.
3.

What is the expected output? What do you see instead?
result is 10 failures out of 52 tests. i assume failures are unexpected?
attached file is stderr of runinit.

What version of the product are you using? On what operating system?
ply-2.5.tar.gz
Linux localhost.localdomain 2.6.27.9-159.fc10.x86_64 #1 SMP Tue Dec 16
14:47:52 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
$ python
Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) 
[GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Please provide any additional information below.
note: this is an install of ply-2.5 under virtualenv.
a virtualenv install of ply-1.8 works (tests) ok:
$ python rununit.py
........................................
----------------------------------------------------------------------
Ran 40 tests in 2.736s

OK

What steps will reproduce the problem?
1. python cxx.py bug.cpp
2. python cxx.py -v bug.cpp
3.

What is the expected output? What do you see instead?
1. No output
2. Show the PLY trace, and terminate normally.

What version of the product are you using? On what operating system?

PLY 2.5, under Cygwin.

Please provide any additional information below.

I submitted a note about this to the ply-hack discussion, and I have
subsequently refined the C++ parser here.  I've tested the parser on a
variety of C++ test examples.  At this point, the remaining parsing issue
appears to lie with the table that PLY generates to parse nested ids, like
A::B::C.

In the bug.cpp example, the parser generates an error in state 124, when
processing a SCOPE token (which represents '::').  But state 124 is
summarized as:

state 124

    (5) id_scope -> id . SCOPE
    (8) nested_id -> id .

    SCOPE           shift and go to state 401

So, this state should shift the SCOPE token, but it doesn't.

Perhaps a clue to the issue with PLY is that there are two other
'equivalent' states in the table:

state 280

    (8) nested_id -> id .
    (5) id_scope -> id . SCOPE

state 641

    (5) id_scope -> id . SCOPE

Perhaps PLY is getting confused...???

What steps will reproduce the problem?
1. Use optimize mode
2. Generate tables
3. Unset write permission on the tables just generated

What is the expected output? What do you see instead?
Nothing. Why should lex.py be writing to the lextab file if it read them to
begin with?

Instead, I see an error message, that lex.py has tried to write to the
lextab file and failed.

What version of the product are you using? On what operating system?
3.2, linux (ubuntu 9.04, installed ply from pypi)

Please provide any additional information below.
A patch to fix this is attached

The documentation of PLY promises 'copious amounts of debugging during
parsing' with yacc.parse(debug=1) in the section '5.12 Yacc implementation
notes'.
debug=1 however only prints an additional line when an error is found.

To get the desired output, one should enter a number > 1 (ie
'yacc.parse(debug=2)'.

This probably holds also when re-directing output to file (the next item).

yacc.yacc() parses tables and returns a new LRParser. It would be nice to
be able to simply get a LRTable back, so that new LRParsers could be
created from them. Since parsers have state on their instances, it should
be easy to create one from the same tables, so concurrent parsing can occur.

It would also be nice if something similar could be done for lex.

What steps will reproduce the problem?
1. Build a parser that reports lexer.lineno on error
2. Trigger an error

What is the expected output? What do you see instead?

Expected output is the line number of the error.  Actual output is 1. 

What version of the product are you using? On what operating system?

PLY 3.3

Please provide any additional information below.

Inspection of the source code reveals that the lexer's lineno attribute is 
initialized to 1, but never incremented as characters are read.  The following 
patch appears to fix the problem.

--- a/lex.py
+++ b/lex.py
@@ -309,6 +309,8 @@ class Lexer:
         lexdata   = self.lexdata

         while lexpos < lexlen:
+            if lexdata[lexpos] == '\n':
+                self.lineno += 1
             # This code provides some short-circuit code for whitespace, tabs, and other ignored characters
             if lexdata[lexpos] in lexignore:
                 lexpos += 1

What steps will reproduce the problem?
1. Use a query production like:

def p_query(p):
    '''query : query_part'''
    p[0] = Query(p[-1])

What is the expected output? What do you see instead?
This used to work in PLY 2.5.  I found that it no longer works in PLY 3.3 when 
I upgraded today.  Instead, I get the exception:
    AttributeError: YaccSymbol instance has no attribute 'value'

It is possible to use p[1] rather than p[-1], and perhaps this is actually a 
better way of doing it, but I thought it was worth noting since this was a 
change that broke existing code.

What steps will reproduce the problem?
1. Create Tokens, where one is a broadly generic version of the second
2. Declare them both as functions (probably not necessary, but it
guarantees priority) with the specific case listed first.
3. Try parsing a file with this lexical structure, to see it in action, use
the attached files please.

What is the expected output? What do you see instead?
When reaching the line that says #begin environment support, I expect a
K_BEGIN_ENV_SUPPORT token.  Instead the Parser gets a K_ANYTHING token, and
uses the wrong rule.

What version of the product are you using? On what operating system?
Version 2.3, downloaded straight from the website. Running Fedora 7
(tragically), using eclipse for development and debugging (version 3.2 with
thelatest release of pydev 1.3.4)  I made a small change, so that I can see
all the groups from the Match object, in lex.  I'm including my copy of
lex.py and yacc.py

Please provide any additional information below.
As far as I can tell, lex seems to compile all the regexs into one massive
regex, with parameters.  I plan on making a patch where it will instead
iterate through the regexs until one matches.  This will probably be
slower.  Please let me know if this is desired, or if there will be another
solution in the works.

I didn't know until I checked the main PLY website just now that the source is 
now hosted on github instead of here. I think the project summary here should 
be updated to reflect this.

I needed to have a filename attached to each token.
One hacky way is to replace the LexToken class with my own version after import 
(ie "lex.LexToken = MyTokenClass").
A better way would be to have the lexer construct tokens from a variable 
holding the class rather than a hardcoded name.
The attached patch realizes this.

My token class can copy the file name as part of the constructor.

It might be better to also have a function call that delivers the token data to 
the new token (eg the token constructor??).
That would solve the problem of deciding when it is safe to read eg 
token.lineno. I don't know how that would affect lexer speed though, so I left 
this change out.