tonybaloney / cpython-book-samples

Sample scripts and examples for my CPython Internals book

cpython-book-samples

Sample scripts and examples for my CPython Internals book

cpython-book-samples's Issues

failure to compile when adding almost-equal operator (Lexing and Parsing)

Attempting to work the almost-equal example in the Lexing and Parsing section, I get an error.

Steps:

• modify Grammar/python.gram and Tokens (see diff below)
• execute make regen-token regen-pegen
• confirm Parser/token.c shows the expected change
• recompile with make -j2 -s
• get error:

Parser/pegen/parse.c:9034:51: error: use of undeclared identifier 'AlE'
            _res = _PyPegen_cmpop_expr_pair ( p , AlE , a );
                                                  ^
1 error generated.
make: *** [Parser/pegen/parse.o] Error 1
make: *** Waiting for unfinished jobs....

Note: experienced this error on both Mac and Linux machines.

Diff before regenerating headers:

$ git diff
diff --git a/Grammar/Tokens b/Grammar/Tokens
index 9de2da5d15..424e388cd6 100644
--- a/Grammar/Tokens
+++ b/Grammar/Tokens
@@ -53,6 +53,7 @@ ATEQUAL                 '@='
 RARROW                  '->'
 ELLIPSIS                '...'
 COLONEQUAL              ':='
+ALMOSTEQUAL            '~='
 
 OP
 AWAIT
diff --git a/Grammar/python.gram b/Grammar/python.gram
index 0e12b5cb96..21474deab8 100644
--- a/Grammar/python.gram
+++ b/Grammar/python.gram
@@ -410,6 +410,7 @@ compare_op_bitwise_or_pair[CmpopExprPair*]:
     | in_bitwise_or
     | isnot_bitwise_or
     | is_bitwise_or
+    | ale_bitwise_or
 eq_bitwise_or[CmpopExprPair*]: '==' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Eq, a) }
 noteq_bitwise_or[CmpopExprPair*]:
     | (tok='!=' { _PyPegen_check_barry_as_flufl(p, tok) ? NULL : tok}) a=bitwise_or {_PyPegen_cmpop_expr_pair(p, NotEq, a) }
@@ -421,6 +422,7 @@ notin_bitwise_or[CmpopExprPair*]: 'not' 'in' a=bitwise_or { _PyPegen_cmpop_expr_
 in_bitwise_or[CmpopExprPair*]: 'in' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, In, a) }
 isnot_bitwise_or[CmpopExprPair*]: 'is' 'not' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, IsNot, a) }
 is_bitwise_or[CmpopExprPair*]: 'is' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Is, a) }
+ale_bitwise_or[CmpopExprPair*]: '~=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, AlE, a) }
 
 bitwise_or[expr_ty]:
     | a=bitwise_or '|' b=bitwise_xor { _Py_BinOp(a, BitOr, b, EXTRA) }

test_tokens is missing

I purchased the paperback and it references a file called test_tokens.py, however I don't see it in this repo. Could you please add it?

Compilation of v3.9.0b1 fails on macOS

Followed the instructions in the CPython Internals book explicitly and unable to compile v3.9.0b1. Unable to continue with the book...

Steps & output as follows:

1. Cloned repo and checkout version stated in book

> git clone [email protected]:python/cpython
> cd cpython
> git checkout tags/v3.9.0b1 -b v3.9.0b1
Updating files: 100% (4849/4849), done.
Switched to a new branch 'v3.9.0b1'

2. Check Xcode command line tools installed

> xcode-select --install
xcode-select: error: command line tools are already installed, use "Software Update" in System Settings to install updates

> softwareupdate --history | grep Xcode
Command Line Tools for Xcode                       15.1       08/01/2024, 12:51:10

> clang --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

3. Check Homebrew installed

> brew --version
Homebrew 4.2.17

4. Install dependencies via homebrew

> brew install openssl xz zlib gdbm sqlite
Warning: openssl@3 3.2.1 is already installed and up-to-date.
Warning: xz 5.4.6 is already installed and up-to-date.
Warning: zlib 1.3.1 is already installed and up-to-date.
Warning: gdbm 1.23 is already installed and up-to-date.
Warning: sqlite 3.45.2 is already installed and up-to-date.

5. Run config script with homebrew zlib

> CPPFLAGS="-I$(brew --prefix zlib)/include" \
  LDFLAGS="-L$(brew --prefix zlib)/lib" \
  ./configure --with-openssl=$(brew --prefix openssl) --with-pydebug

##### I've truncated some output, see below #####

configure: creating ./config.status
config.status: creating Makefile.pre
config.status: creating Misc/python.pc
config.status: creating Misc/python-embed.pc
config.status: creating Misc/python-config.sh
config.status: creating Modules/ld_so_aix
config.status: creating pyconfig.h
creating Modules/Setup.local
creating Makefile

I've truncated the output from config script. See this gist for full output.

6. Check Makefile, etc. created

> ls -l
total 4272
-rw-r--r--@   1 Braden  staff     630 16 Apr 11:51 CODE_OF_CONDUCT.md
drwxr-xr-x@  31 Braden  staff     992 16 Apr 11:51 Doc
drwxr-xr-x@   5 Braden  staff     160 16 Apr 11:51 Grammar
drwxr-xr-x@ 103 Braden  staff    3296 16 Apr 11:51 Include
-rw-r--r--@   1 Braden  staff   12775 16 Apr 11:51 LICENSE
drwxr-xr-x@ 205 Braden  staff    6560 16 Apr 11:51 Lib
drwxr-xr-x@  11 Braden  staff     352 16 Apr 11:51 Mac
-rw-r--r--@   1 Braden  staff   79698 16 Apr 12:04 Makefile
-rw-r--r--@   1 Braden  staff   70340 16 Apr 12:04 Makefile.pre
-rw-r--r--@   1 Braden  staff   70063 16 Apr 11:51 Makefile.pre.in
drwxr-xr-x@  29 Braden  staff     928 16 Apr 12:04 Misc
drwxr-xr-x@ 134 Braden  staff    4288 16 Apr 12:04 Modules
drwxr-xr-x@  55 Braden  staff    1760 16 Apr 11:51 Objects
drwxr-xr-x@  45 Braden  staff    1440 16 Apr 11:51 PC
drwxr-xr-x@ 117 Braden  staff    3744 16 Apr 11:51 PCbuild
drwxr-xr-x@  18 Braden  staff     576 16 Apr 11:51 Parser
drwxr-xr-x@   6 Braden  staff     192 16 Apr 11:51 Programs
drwxr-xr-x@  79 Braden  staff    2528 16 Apr 11:51 Python
-rw-r--r--@   1 Braden  staff   10173 16 Apr 11:51 README.rst
drwxr-xr-x@  24 Braden  staff     768 16 Apr 11:51 Tools
-rw-r--r--@   1 Braden  staff   13421 16 Apr 11:51 aclocal.m4
-rwxr-xr-x@   1 Braden  staff   44166 16 Apr 11:51 config.guess
-rw-r--r--@   1 Braden  staff  852221 16 Apr 12:04 config.log
-rwxr-xr-x@   1 Braden  staff   42210 16 Apr 12:04 config.status
-rwxr-xr-x@   1 Braden  staff   36251 16 Apr 11:51 config.sub
-rwxr-xr-x@   1 Braden  staff  512111 16 Apr 11:51 configure
-rw-r--r--@   1 Braden  staff  172981 16 Apr 11:51 configure.ac
-rwxr-xr-x@   1 Braden  staff   15368 16 Apr 11:51 install-sh
drwxr-xr-x@   4 Braden  staff     128 16 Apr 11:51 m4
-rw-r--r--@   1 Braden  staff      82 16 Apr 11:51 netlify.toml
-rw-r--r--@   1 Braden  staff   47822 16 Apr 12:04 pyconfig.h
-rw-r--r--@   1 Braden  staff   45557 16 Apr 11:51 pyconfig.h.in
-rw-r--r--@   1 Braden  staff  111118 16 Apr 11:51 setup.py

7. Compile

> make -j2 -s
Objects/fileobject.c:245:9: warning: variable 'newlinetypes' set but not used [-Wunused-but-set-variable]
    int newlinetypes = 0;
        ^
1 warning generated.
Objects/dictobject.c:2235:16: warning: variable 'i' set but not used [-Wunused-but-set-variable]
    Py_ssize_t i, j;
               ^
1 warning generated.
Objects/obmalloc.c:2639:14: warning: variable 'j' set but not used [-Wunused-but-set-variable]
        uint j;
             ^
1 warning generated.
./Modules/getpath.c:942:40: warning: incompatible pointer types passing 'unsigned long *' to parameter of type 'uint32_t *' (aka 'unsigned int *') [-Wincompatible-pointer-types]
    if (_NSGetExecutablePath(execpath, &nsexeclength) != 0
                                       ^~~~~~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/mach-o/dyld.h:105:54: note: passing argument to parameter 'bufsize' here
extern int _NSGetExecutablePath(char* buf, uint32_t* bufsize)                 __OSX_AVAILABLE_STARTING(__MAC_10_2, __IPHONE_2_0);
                                                     ^
1 warning generated.
./Modules/posixmodule.c:9403:15: error: call to undeclared function 'sendfile'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        ret = sendfile(in_fd, out_fd, offset, &sbytes, &sf, flags);
              ^
1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....

7. Cannot proceed any further due to the -Wimplicit-function-declaration error.