Tools to process things using python chess library.
License: GNU General Public License v3.0
Tools to process things using python chess library. See scripts folder.
Is it possible to process "PGN File" from "PlayChess Games"? Tell me how to make please. Scripts - evalswing.
Now as the options work for setting the width/height in the graphs, there needs to be some auto-scaling of the fonts, I guess.
I did adapt the fonts manually for a new size (4:2.7 - I tried to keep the original ratio), but another problem is that now not enough horizontal grid lines are shown (max 3 in my resized example, probably due to a default value?).
Here is an example (min/max eval 7), it prints now only the -5 0 +5 lines and info, which
is too less, I think it should may be print 1 line at least for 1 eval point, or similar?
Hi Ferdinand,
I tested an older (2019) PGN and noticed that we need to add a few cases for some engines,
which create no output for forced or easy moves, otherwise plot.py would exit.
I could verify that it chokes on output, which just has e.g. {0s} or {0.001s} in it.
A regex search in my pgn revealed that (in this case) CuteChess had saved several
times a very low 3 digits time w/o eval/depth.
IIRC there must be a few dozen others which will send no PV output, but Cutechess
will sometimes still save a very low time in the move comment.
After a quick ugly hack I could create all graphs, before it stopped after game 1 (of 400)
(did not know how to do regex in py, sth like 0.00^[1-9])
# Get time.
# +13.30/12 0.020s
if comment == '':
tv = 0.0
elif comment == '0':
tv = 0.0
elif comment == '0.001':
tv = 0.0
elif comment == '0.002':
tv = 0.0
elif comment == '0.003':
tv = 0.0
elif comment == '0.004':
tv = 0.0
else:
try:
tv = float(comment.split()[1].split('s')[0])
except ValueError:
tv = 0.0
The second issue with this is that it would be better, if no eval is there at all (due to forced/easy move),
to use the eval from the previous move to avoid ugly discontinuities.
IIRC this was a problem also in various TCEC games and there was some discussion about how
to fix graphs for such engines.
// A third issue, which doesn't hurry or should be fixed before feeding the pgn to plot.py would be when
doing engine matches/tournaments with own books.
Here there are some programs which give special eval/depth output, while still in book.
But this has no priority at all and may be it would be even better in this case to fix the pgn files first with a script. //
The first two issues are very relevant though.
I have added some more info here:
https://github.com/rwbc/Python-Chess-Scripts/blob/main/additional_info
Here the pgn (the 's' in time was stripped by a script of mine, I didn't find the original pgn anymore)
20190708.zip
Would it be possible to just plot pre-selected graphs, may be by passing an array of numbers (= game number in pgn file)?
This would be a nice enhancement, of course plotting all games should remain default option.
It is crucial though that the pre-selected graphs are numbered after the given game number in the total pgn file.
I fiddled around with more testing of the eval_swing and found more things I had to correct in the FCPT 2021 pgn file.
After I removed all other strange things, I imported the result into a GoogleSheet and did a first filtering for the most interesting
and strange results and had the idea this could be used for just plotting a first selection of graphs.
https://docs.google.com/spreadsheets/d/1FoSw0MH6ahhX4jsLNzKIAR322mKD99HhgUuvYxs7jRE/edit#gid=1639412648
The resulting game numbers in Sheet3 would be the first to plot.
So if the pgn-graph could read a text file with an array of numbers, which is equivalent to the game to plot, e.g. {x y z...),
that would be nice. Just an idea. What do you think?
Example:
[Event "Blitz"]
[Site "St. Louis, MO USA"]
[Date "2016.04.29"]
[Round "18.1"]
[White "Fabiano Caruana"]
[Black "Garry Kasparov"]
[Result "0-1"]
[EventDate "2016.04.28"]
[WhiteElo "2795"]
[BlackElo "2812"]
[PlyCount "74"]
[WMaxEval= 0.00]
[BMaxEval = 0.00]
[WMinEval = 0.00]
[BMinEval = 0.00]
Sorry for having not properly reported this in an own issue before.
Depth 1 evals should be treated as no eval (skipped) and the previous non-depth 1 eval should be used for plotting.
Same for eval_swing. Depth 1 evals should not be considered for output.
(The times should be kept if not zero)
I have created a test.pgn with just 4 games from two sites. This is attached at the end of this post.
found depth 1 in pgn (you can see what evals made it into the plot and into the eval_swing output)
game 1
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/50: 40. Nd5 {+2.55/1 0} Qd8 {+0.79/26 16}
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/90: 80. e6 {+0.49/1 0} Nd6 {-0.30/29 30}
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/130: 120. Kf5 {-8.94/1 0} Bd5 {-98.00/32 1:07}
game 2
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/191: 17. Nxh7 {+3.37/24 17} Kxh7 {+1.26/1 5}
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/197: 23. Bd1 {+6.66/25 4} cxb2 {+5.29/1 5}
game 3
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/227: 5. f4 {+0.01/1 12} c5 {0.00/39 0}
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/230: 8. g3 {+0.01/1 2} O-O {+0.22/24 7}
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/235: 13. O-O {+0.01/1 8} cxd4 {+0.31/34 27}
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/255: 33. Rd3 {+0.01/1 4} Bb2 {+3.27/34 22}
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/271: 49. Rxb1 {+0.01/1 31} Bxb1 {+153.75/39 0}
game 4
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/341: 40. Rc6 {0.00/27 25} Ra2 {-0.95/1 0}
C:\ChessTools\QualityControl\PGN_Scripts\PGN\test.pgn/381: 80. Kf2 {0.00/46 38} Rd1 {-0.43/1 0}
# White Black Res WMaxMove WMaxEval WMinMove WMinEval BMaxMove BMaxEval BMinMove BMinEval
1 iCE 4.0 v853 Modern x64 Wasp 4.50 Modern x64 0-1 40 2.55 - NaN - NaN 107 -98.00
2 Fire 7.1 Senpai 2.0 1-0 - NaN 1 0.15 23 5.29 - NaN
3 Crafty 25.2 Stockfish 220818 0-1 23 0.92 - NaN - NaN 1 -0.13
4 Vajolet2 2.8 BMI2 x64 iCE 4.0 v853 Modern x64 1/2-1/2 9 0.28 34 -0.37 8 0.55 40 -0.95
It seems for WB pgn only reading of minute/hours time savings is necessary.
(I have never really used Arena, but may be it does the same?)
Example: move 11 White
[Event "RWBC Test34"]
[Site "ESPRESSO"]
[Date "2008.08.04"]
[Round "2.1"]
[White "Bright_03a"]
[Black "Delfi_54"]
[Result "1-0"]
[TimeControl "40/480"]
[Annotator "11. -0.46 10... +0.07"]
[Number "26"]
1. e4 e6 2. d4 d5 3. Nc3 Nf6 4. Bg5 Bb4 5. e5 h6 6. Be3 Ne4 7. Qg4 g6 8. a3
Bxc3+ 9. bxc3 c5 10. Bd3 h5 {+0.07/13 17} 11. Qf3 {-0.46/15 1:11}
Nxc3 {+0.09/13 11} 12. dxc5 {-0.43/15 30} Nc6 {+0.10/13 10} 13.
Qf6 {-0.43/14 25} Qxf6 {+0.56/15 10} 14. exf6 {-0.44/16 13}
d4 {+0.47/15 17} 15. Bd2 {-0.32/15 11} Na4 {+0.48/15 8} 16.
Nf3 {-0.27/15 9} Nxc5 {+0.51/14 11} 17. O-O {-0.30/15 16}
Nxd3 {+0.59/14 11} 18. cxd3 {-0.29/15 10} b6 {+0.68/15 14} 19.
Rfc1 {-0.26/16 16} Bb7 {+0.53/16 18} 20. a4 {-0.26/15 10} Rd8 {+0.64/13 8}
21. a5 {-0.15/14 14} bxa5 {+0.70/14 10} 22. Rab1 {-0.17/14 16}
Ba8 {+0.73/15 12} 23. Rb5 {-0.27/15 32} O-O {+0.81/14 7} 24.
Rbc5 {-0.38/14 10} Nb4 {+0.87/14 9} 25. Ne5 {+0.11/14 13} Rd5 {+0.48/14 8}
26. Rc8 {+0.78/15 11} Rdd8 {-0.24/15 13} 27. R8c7 {+1.12/15 14}
a6 {-0.54/15 12} 28. Bh6 {+1.46/14 9} Nd5 {-0.95/16 17} 29.
Ra7 {+1.66/15 14} Nc3 {-0.75/16 16} 30. Bxf8 {+1.66/15 14}
Rxf8 {-0.76/15 5} 31. Kf1 {+1.86/15 7} a4 {-0.89/16 11} 32.
Rxa6 {+1.95/15 10} Bd5 {-0.96/16 10} 33. Ra7 {+2.13/15 14} h4 {-0.94/16 11}
34. f3 {+2.10/14 8} Ba8 {-0.93/14 14} 35. Ke1 {+2.28/14 11}
Bd5 {-1.04/15 11} 36. Kd2 {+2.30/15 14} Ba8 {-1.20/15 19} 37.
g4 {+2.72/14 11} Bd5 {-1.57/15 10} 38. g5 {+2.86/15 9} a3 {-1.58/15 8} 39.
Rxa3 {+2.80/14 9} Rb8 {-2.35/15 8} 40. Raa1 {+2.87/14 9} Rb5 {-1.89/15 11}
41. f4 {+3.23/14 13} Bg2 {-2.22/14 7} 42. Ra7 {+4.49/15 13}
Rb2+ {-3.35/14 6} 43. Rc2 {+4.55/17 10} Rb1 {-3.65/17 8} 44.
Rxc3 {+4.66/17 15} dxc3+ {-3.86/16 1} 45. Kc2 {+4.66/19 14}
Rb2+ {-4.77/18 8} 46. Kxc3 {+6.85/18 11} Rb7 {-7.66/19 9} 47.
Ra8+ {+8.14/18 12} Kh7 {-7.66/18 2} 48. Rf8 {+9.00/18 13}
Rc7+ {-8.09/18 14} 49. Kd2 {+9.22/17 8} Bh3 {-8.76/19 4} 50.
Nxf7 {+12.24/18 11}
{Black resigns} 1-0
I add two historical WB pgn files
One is from RWBC (blitz) and one from WBEC (long time)
A suggestion for a tiny enhancement for later:
Create the images in a folder named after the pgn file.
Guenther
I did some stress test yesterday on a modified Shredder PGN from Franks tournament and after I figured out all necessary changes (Shredder GUI really has problems with near mate scores and creates all kind of strange eval output there, when trying to convert scores) it ran smoothly.
But there is a problem with WPOV (White point of view) evals saved. I guess a flag/setting for this is needed for inverting the Black scores.
Here is a snippet of 20 games to see the effect. I attached the test games too.
# White Black Res WMaxMove WMaxEval WMinMove WMinEval BMaxMove BMaxEval BMinMove BMinEval
1 Critter 1.6a x64 Xiphos 0.6 BMI2 x64 1/2-1/2 17 0.11 25 0.00 18 0.35 19 0.00
2 Fizbo 2.0 BMI2 x64 Texel 1.08a13 BMI2 x64 1/2-1/2 24 0.35 18 -0.72 26 0.06 15 -1.14
3 Booot 6.4 POPCNT x64 pirarucu 3.3.5 JAVA x64 1-0 - - 10 0.29 44 306.92 - -
4 Dragon by Komodo NN AVX2 x64 Vajolet2 2.8 BMI2 x64 1-0 - - 17 0.16 75 319.99 - -
5 Arasan 22.2 BMI2 x64 rofChade 2.3 BMI2 x64 1-0 - - 8 0.39 69 319.98 - -
6 Ethereal 12.75 PEXT x64 Topple 0.8.0 Modern x64 1-0 - - 10 0.94 68 319.98 - -
7 Chiron 4 x64 iCE 4.0 v853 Modern x64 1/2-1/2 13 0.04 26 -0.37 60 0.75 28 -0.92
8 Demolito 2020-12-24 PEXT x64 Wasp 4.50 Modern x64 0-1 32 1.43 - - - - 91 -319.99
9 Combusken 1.4.0 AMD x64 chess22k 1.14 JAVA x64 1-0 - - 9 -0.04 80 306.34 - -
10 Halogen 9 NN PEXT x64 Shredder 13 POPCNT x64 0-1 18 0.78 - - - - 66 -319.99
11 GullChess 3.0 Sy BMI2 x64 SlowChess Blitz 2.4 NN x64 0-1 17 0.97 - - - - 73 -319.99
12 Igel 2.9.0 NN BMI2 x64 Seer 1.2.1 NN Skylake x64 1-0 - - 31 0.00 81 256.00 - -
13 Laser 1.7 BMI2 x64 RubiChess 1.9 NN BMI2 x64 1-0 - - 11 -0.29 76 319.97 - -
14 Lc0 0.26.3 NN CPU x64 Protector 1.9.0 x64 1-0 - - 11 0.26 72 319.99 - -
15 Marvin 4.0.1 POPCNT x64 Pedone 3.0 NN BMI2 x64 1/2-1/2 16 0.05 27 -0.86 15 0.00 29 -0.88
16 Fritz 17 (Ginkgo) x64 Sf 2020-12-31_18 NN BMI2 x64 0-1 18 0.54 - - - - 139 -319.99
17 Koivisto 4.19 AVX x64 Schooner 2.2 XB SSE x64 1/2-1/2 38 0.59 27 -0.10 17 0.52 26 -0.08
18 Nemorino 6.04 NN PEXT x64 Nirvanachess 2.5 POPCNT x64 1-0 - - 10 0.09 60 319.99 - -
19 Minic 3.02 NaNi Skylake x64 Orion 0.8 NN FMA x64 1/2-1/2 32 0.86 92 -1.35 4 0.27 22 -0.94
20 Defenchess 2.3 dev BMI2 x64 Winter 0.9 NN BMI2 x64 1/2-1/2 42 1.15 20 0.00 11 0.33 219 -2.24
Done C:\ChessTools\QualityControl\PGN_Scripts\PGN\FCPT_2021\FCPT2021.pgn, Elapse (sec): 18.213
It would be nice if you could add the depths also into the time plot.
May be with a user settable factor for scaling them to fit better into the canvas?
(e.g. in fast games with 0.x - some seconds per move, depths still could be 15 - 20
and then a lot of space would be wasted on the image.)
It would help to see anomalities with relatively low depths despite higher time used.
(Mostly user introduced problems with the tournament setup)
What do you think?
I think with Lichess as the biggest online chess server (and open source too!) it would be nice to support its format too.
You probably know Lichess has a auto-analyze feature for checking played games later with a fast SF search and comment all games (if you have an account).
This is saved to thegames and available for download either by the Lichess api via filters or via the Lichess game database.
https://lichess.org/api#operation/apiGamesUser
https://database.lichess.org/
A few people even have set up an UI for making it easier to send requests for games download via the api commands.
https://lichessgamedownload.netlify.app/
https://fbserv.herokuapp.com/games.html?
Of course creating graphs only makes sense for the already analyzed games, but you can filter them already before downloading. (not with the total databse per month download)
Here is an example of its format:
[Event "RT-Schnellschachturnier 31"]
[Site "https://lichess.org/HPpt04kq"]
[Date "2020.12.16"]
[White "Bruto130199"]
[Black "hot_sun"]
[Result "0-1"]
[UTCDate "2020.12.16"]
[UTCTime "20:49:15"]
[WhiteElo "2140"]
[BlackElo "2425"]
[WhiteRatingDiff "-3"]
[BlackRatingDiff "+3"]
[Variant "Standard"]
[TimeControl "900+5"]
[ECO "B23"]
[Termination "Normal"]
1. e4 { [%eval 0.0] [%clk 0:15:00] } 1... c5 { [%eval 0.0] [%clk 0:15:00] } 2. Nc3 { [%eval 0.12] [%clk 0:14:58] } 2... a6 { [%eval 0.29] [%clk 0:15:03] } 3. a4 { [%eval -0.17] [%clk 0:15:02] } 3... e6 { [%eval 0.13] [%clk 0:15:06] } 4. Bc4 { [%eval -0.31] [%clk 0:15:06] } 4... Nc6 { [%eval -0.33] [%clk 0:15:07] } 5. d3 { [%eval -0.44] [%clk 0:15:07] } 5... Nf6 { [%eval -0.22] [%clk 0:15:08] } 6. f4 { [%eval -0.73] [%clk 0:15:03] } 6... d5 { [%eval -0.82] [%clk 0:15:11] } 7. Ba2 { [%eval -0.6] [%clk 0:15:06] } 7... dxe4 { [%eval -0.36] [%clk 0:14:34] } 8. Nxe4 { [%eval -0.64] [%clk 0:15:08] } 8... Nb4 { [%eval -0.16] [%clk 0:13:17] } 9. Bb3 { [%eval -0.28] [%clk 0:15:08] } 9... Be7 { [%eval -0.31] [%clk 0:12:34] } 10. Nf3 { [%eval -0.2] [%clk 0:15:10] } 10... Qc7 { [%eval 0.0] [%clk 0:12:38] } 11. O-O { [%eval -0.89] [%clk 0:15:13] } 11... Nxe4 { [%eval -0.74] [%clk 0:12:42] } 12. dxe4 { [%eval -0.85] [%clk 0:15:17] } 12... c4 { [%eval -1.08] [%clk 0:12:46] } 13. Ba2 { [%eval -1.0] [%clk 0:15:21] } 13... O-O { [%eval -0.92] [%clk 0:12:28] } 14. c3 { [%eval -1.58] [%clk 0:15:24] } 14... Nxa2 { [%eval -1.25] [%clk 0:12:31] } 15. Rxa2 { [%eval -1.24] [%clk 0:15:29] } 15... b6 { [%eval -1.23] [%clk 0:11:49] } 16. Be3 { [%eval -1.16] [%clk 0:15:30] } 16... Bb7 { [%eval -1.18] [%clk 0:11:48] } 17. e5 { [%eval -1.21] [%clk 0:15:09] } 17... Rad8 { [%eval -1.0] [%clk 0:11:28] } 18. Qe2 { [%eval -1.49] [%clk 0:15:10] } 18... Rd3 { [%eval -1.47] [%clk 0:11:04] } 19. a5 { [%eval -2.29] [%clk 0:14:57] } 19... bxa5 { [%eval -2.17] [%clk 0:10:23] } 20. Nd4 { [%eval -2.9] [%clk 0:14:55] } 20... Bc5 { [%eval -2.76] [%clk 0:10:07] } 21. Rfa1 { [%eval -3.42] [%clk 0:14:49] } 21... Bb6 { [%eval -3.31] [%clk 0:10:09] } 22. Rd1 { [%eval -4.99] [%clk 0:14:25] } 22... Bd5 { [%eval -2.82] [%clk 0:09:43] } 23. Raa1 { [%eval -2.74] [%clk 0:14:22] } 23... Rxd1+ { [%eval -1.87] [%clk 0:09:45] } 24. Qxd1 { [%eval -2.67] [%clk 0:14:18] } 24... f6 { [%eval -2.11] [%clk 0:09:21] } 25. exf6 { [%eval -2.49] [%clk 0:14:13] } 25... Rxf6 { [%eval -2.17] [%clk 0:09:26] } 26. Qd2 { [%eval -1.82] [%clk 0:14:10] } 26... h5 { [%eval -1.61] [%clk 0:09:19] } 27. Rf1 { [%eval -1.39] [%clk 0:14:03] } 27... h4 { [%eval -1.48] [%clk 0:09:21] } 28. h3 { [%eval -1.47] [%clk 0:14:07] } 28... Rg6 { [%eval -0.26] [%clk 0:08:50] } 29. Nf3 { [%eval -9.47] [%clk 0:14:08] } 29... Qxf4 { [%eval -9.64] [%clk 0:08:13] } 30. Bxb6 { [%eval -10.09] [%clk 0:12:28] } 30... Qxd2 { [%eval -9.9] [%clk 0:07:42] } 31. Nxd2 { [%eval -10.42] [%clk 0:12:30] } 31... Rxg2+ { [%eval -10.33] [%clk 0:07:47] } 32. Kh1 { [%eval -10.39] [%clk 0:12:34] } 32... Rxd2+ { [%eval -10.48] [%clk 0:07:51] } 33. Kg1 { [%eval -10.42] [%clk 0:12:37] } 33... Rg2+ { [%eval -10.56] [%clk 0:07:56] } 34. Kh1 { [%eval -10.62] [%clk 0:12:41] } 34... Rxb2+ { [%eval -10.35] [%clk 0:08:01] } 35. Kg1 { [%eval -10.04] [%clk 0:12:46] } 35... Rxb6 { [%eval -10.48] [%clk 0:08:05] } 0-1
A dozen of selected games (only rated rapid for a certain period)
lichess_hot_sun_2020-12-26.zip
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.