juliadata / chunkedbase.jl Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Imagine we want to do type inference on columns of CSV. We'd start with some initial guess for each column and then process the file in chunks, as we always do. Now one of the workers discovers that a column that was until now considered an Int
has to be a Float64
. How should we spread this information to the other chunks? Should this be a responsibility of the consume
context? Should we define specific callback that would work as a barrier -- we'd stop parsing, wait for all results to enter the barrier, sync their schemas, and release them? Doing this in sync_tasks
is problematic, because that only synchronizes chunks belonging to one of the two buffers.
More design work is needed on this issue.
MIN_TASK_SIZE_IN_BYTES
is a global constant that we use to decide which amount of work (in bytes) is worth splitting into multiple tasks. Currently, this is set to 16KiB, but this number has not been researched at all. We should find a good default value for this and also make it easily configurable (maybe as a field of ChunkingContext
).
These can happen when the buffer is too small to fit the largest row in the input, or when we misconfigured the lexer, or if the file is not parseable. It's hard to know which of these is the problem, but we should try and do a bit of analysis on the current chunk (like looking for how strings are escaped and comparing it with our escape character, look if there are any newlines in the chunk...) to give our best guess (and evidence for it)
The repo contains a small tracing framework which is all commented out + there is a src/_traces.jl
file which visualizes the collected traces using GLMakie. Ideally, the tracing framework would be hidden behind a toggle that is easy to switch without modifying source code and the plotting capability would be implemented via package extensions.
This comment suggests a way to implement the on/off switch #7 (comment)
This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml
to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix
on this issue.
I'll open a PR within a few hours, please be patient!
We should keep specializing on the result buffer type, but the AbstractResultBuffer
doesn't carry any real interface with it, so we should abandon it and accept e.g. Dict
s and Vector
s as valid result buffer types.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.