droher / baseball.computer.rs

Rust parser for the baseball.computer database.

baseball.computer - The Rust Part

Rust parser for the baseball.computer database.

See the website or the main repo for more info:

https://baseball.computer/

https://github.com/droher/baseball.computer

baseball.computer.rs's Issues

Data completeness tables

Data points

• Pitch sequence
  • Pitch total
  • Balls/strikes thrown
  • Balls/strikes count at event
  • Called vs swinging strikes
• Fielding
  • Putouts
  • Assists
  • Errors
  • Fielding opportunities (using hit-to-fielder) (covered in hit location)
• Hit location
  • GB/FB
  • Hit to fielder
  • Hit location spec
• PBP Account
  • PA covered
• Box score account
  • Basically all box score stats

Granularities (all split into batting, pitching, fielding)

• Season
• Game
• Team Season
• Player
• Player Season
• Team Game
• Player Game
• Event

Add file name and line number to game and event tables

Potentially add table to include actual full raw line?

Add scorekeeper/inputter metadata to game schema

Will be useful inputs for interpolation algorithms

Add box score lines to nested table

Misc data quality tracker

• Many old box score files missing park id
• Some Negro League games have null ump entries when they should just be not included e.g. https://github.com/droher/retrosheet/blob/master/nlb/allstar/1933EW.EB#L20
• Incorrect/dupe box phlines, blines
• Files/games duplicated across directories - rosters, Negro League games

Add back game-type imputation from file name

Fielder position not attached to fielders choice

Should be able to infer from the position hit to and/or the first fielder in the sequence.

Add info from new NLB data files

Add batter/pitcher handedness to event table

Add file info to game table

end_event_id not being populated for fielders in certain situations

Happens during at least some instances when a player changes positions but remains in the game

Create numeric keys for PKs and enums

Duckdb is running out of memory when trying to perform operations on tables that have multiple rows per event (particularly fielding. A game-event-baserunner key might have 20 bytes as a string, but only 4 bytes are needed to represent it uniquely.

• Assign numeric IDs to enums that make it to outputs
  • Alternative is to make them enums inside duckdb but that seems not worth it/harder
• Add global 4-byte identifiers for games, events, baserunners
  • This one might actually make more sense to do in dbt as long as there's a consistent translation

Update hit location codes

There are some undocumented hit location codes that are not getting parsed correctly - "L", "M", and "R" as further specificity in a numerical range.

Might also want to build a mapping of the codes to polar coordinates (maybe worth saving for the inference step).