juditacs / wordcount Goto Github PK
View Code? Open in Web Editor NEWCounting words in different programming languages.
Counting words in different programming languages.
bash scripts/test.sh java -classpath java WordCount
---- java -classpath java WordCount ----
test1 fails
test2 fails
FAIL
or testing just one file:
cat data/test/test1.in | java -classpath java WordCount
aaa 3
abc 3
bbb 2
1
ccc 1
It looks like empty lines are counted as words.
There is that rule in the README:
- single-thread is preferred but you can add multi-threaded or multicore versions too
This reads as if there shall be a singlethreaded contribution for a language, and then multithreaded contributions can be added as seperate entries.
Now there was this cleanup which threw out quite a lot of programs in languages that had multiple entries.
Since Elixir is a language that bets on concurrency, I'd like to retry a concurrent version before starting this in Erlang.
So how do we handle this?
It would be very easy to add a CLI-switch/subcommand to the script which does enable/disable concurrency easily, but then still the question remains, how to handle it during the runs?
Using nodejs 5.x - since test9 deals with whitespace, it's probably fudging that.
bash scripts/test.sh node javascript/wordcount.js
---- node javascript/wordcount.js ----
test1 fails
test2 fails
test3 fails
test4 fails
FAIL
The test filed are located in data/test/test*
the ones ending in .in are the input and the corresponding .out is their supposed output.
Could you try and see if this gives any benefits in performance. Does it even make sense to use G1 GC? @svanoort
See 13:45ff Talk / Slides
Using "ubuntu" as base is not best practice. In my case it fails. I installed docker a long time ago, and my ubuntu image was still at 12.04
Let use
FROM ubuntu:14.04
I am looking at setting up an AWS environment (spun up on demand only) that will run tests in a fast and automated fashion, using my personal Jenkins host to trigger it when commits are pushed.
Work progress:
Hardware/specs:
Architecture:
Two options for how to set it up:
Open questions:
Yesterday I had good results tinkering with a spot-purchased c3.large instance for benchmarking, doing all I/O to the /media/ephemeral0 instance store. Pricing was only about $0.04/hour for the spot buy (purchased at 2x current spot price to prevent it being terminated after exceeding the price).
@kundralaci @szelpe I would like to delete wordcount2.js, it's slower than wordcount.js. Is it ok?
The full Hungarian wiki has ~4.3 GB of data, but ~2.5GB of unique string content:
cat data/huwiki-latest-pages-meta-current.xml | sed 's/[\t ]/\n/g' | grep -v ^$ | sort | uniq | wc -m
2507384541
There are ~25M unique tokens.
This means that we are generating gigantic hashtables with generally count = 1, and languages that store Unicode strings as 2-byte representations in memory suffer greatly due to memory overheads. Much of the memory used will simply be storing the unique strings.
Originally I wanted only one version in each language but simple/vanilla and optimized versions reasonably differ (Java is a prime example of this), so we should support more than one version. Should it be one simple and one optimized or maybe more?
This would be a very nice addition.
Due to numerous other engagement I am unable to continue maintaining the project and therefore looking for a new owner. I will try to fix the current issues mostly due to bitrot and after that the project will officially be discontinued until someone is willing take over.
Maybe it's time to update the leaderboard?
The Dockerfile is a mess right now.
Install commands should be grouped by the language they are required for.
Any help would be welcome.
Reorganize the setup of everything such that it is much easier to begin with everything.
I struggled a lot during the setup phase, and I'm still unsure if everything is working as it should. Probably either the documentation or the workflow itself should be updated in a way that makes the process more clear for potential contributors.
I think the memory usage as shown in the overview page is not comparable to other languages.
The bash script fork to different processes many times by piping outputs to other binaries. I think the memory usage of those separate processes is not accounted for.
My pull request was merged 22 days ago (25.04.2016), but results (leaderboard) still not updated.
When executed with an UTF-8 locale, it outputs completely different counts than e.g. the py2 / java versions. When executed with the C locale, it fails at the output phase with
UnicodeEncodeError: 'ascii' codec can't encode character '\xee' in position 0: ordinal not in range(128)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.