Comments (4)
Simplest / quickest way I can think of doing- parser writes a .reg file which is "0" when started and running and "1" when completed. fab can simply collect all the .reg files- super simple- and then come up with register of global status
from bazaar.
@chrismre Have to run to dinner with Will (timeliness!) but quick rundown of process (bold is what I'm adding):
- Files split locally into segments
- Segments are distributed across servers / cores
- When a core picks up a segment, it writes "0" to
seg_id.reg
in addition to writing output toseg_id.parsed
- When a core completes a segment it writes "1" to
seg_id.reg
- Global status can be checked at any time via a simple 'collect' operation over
*.reg
infab
- At the end, the
*.parsed
files are collected and catted to get full output...
Any thoughts? This just seemed quick and simple to implement given the current setup
@raphaelhoffmann any thoughts?
from bazaar.
That is really cool! I like that you can do a global status check with a
simple command.
Thanks for building this.
On Thu, Aug 6, 2015 at 5:57 PM, Alex Ratner [email protected]
wrote:
@chrismre https://github.com/chrismre Have to run to dinner with Will
(timeliness!) but quick rundown of process:
- Files split locally into segments
- Segments are distributed across servers / cores
- When a core picks up a segment, it writes "0" to seg_id.reg in
addition to writing output to seg_id.parsed- When a core completes a segment it writes "1" to seg_id.reg
- Global status can be checked at any time via a simple 'collect'
operation over *.reg in fab- At the end, the *.parsed files are collected and catted to get full
output...Any thoughts? This just seemed quick and simple to implement given the
current setup—
Reply to this email directly or view it on GitHub
#13 (comment).
from bazaar.
@raphaelhoffmann Just need to merge my jsonreader-multi-key
branch into master, or the relevant part of it, for this to work
from bazaar.
Related Issues (12)
- Condor/AWS unified interface? HOT 2
- Launching a r3.8xlarge instance doesn't work HOT 1
- Instances can run out of HD space during parse operation HOT 1
- Something else causing fab parse to abort... HOT 2
- Medical Map HOT 1
- Exception and no TSV's HOT 2
- Default settings require empty config.properties
- Bazaar/Parser doesn't correctly escape some characters in TSV HOT 2
- Auto-calculate the batch size? HOT 3
- [error] error message in setting up pip
- Parser processes are killed (memory error)? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bazaar.