foreverwintr / blueprints

A utility for aggregating data from disparate sources.

Articles I like:

• The law of leaky abstractions. Modern software and computers in general are an amazing tower of abstractions, one built upon another. In this article Joel Spolsky explains both the power of abstractions, and some of the problems that arise when they 'leak'.
• The Infinite Space Between Words. Puts the speed of a modern CPU in perspective by scaling a single clock cycle up to 1 second, and looking at how long other common operations take.

blueprints's Issues

Better names

• assembler
• data_factory
• blueprints. Project with similar name

ToDo

• Static frame examples
• Plan visualization
  • Using Dash?
    • Using a web browser as a UI has some caveats, like knowing when to stop the server. webui looks interesting.
• Factory API that supports distributed processing
  • This seems hard considering firewalls and such.
• set up mypy
• Allow missing
  • For series recipes, missing fill value can be a constant recipe or series of fill value if there is an index. FrameRecipe can handle.
• Dependency order. Args and kwargs?
• Blueprints. Hashable? - Seems unlikely you'd build the same blueprint twice.
• Factory Caching
• Make installable
• Use in real code. Note: need python 3.10
• Should the factory be able to produce missing as a final output (e.g., the return of factory.process_recipe)? That seems undesirable, but the user can decide whether to allow_missing on the final recipe, too. I'll say yes for now.
• Downstream effects? Satisfy a missing data fill value if something is reindexed.
• #9
• Exception Grouping

Recipe serialization and repr.

Also blueprint serialization. The difficulty that is making me hesitate is how to serialize recipes that contain functions. I think serializing the function name and requiring all functions to be present in a registry is acceptable.