astralway / webindex Goto Github PK
View Code? Open in Web Editor NEWApache Fluo application that creates a web index using Common Crawl data
License: Apache License 2.0
Apache Fluo application that creates a web index using Common Crawl data
License: Apache License 2.0
While running a 3 day experiment with Webindex on EC2, I noticed that workers were constantly being killed for using two much memory. I started off with 128 threads and 4G per workers. I increased to 5G and then 6G and workers were still being killed or dying.
When the workers had 4G and were being killed I looked in the YARN logs to determine why and found it was memory usage. After increasing the memory, I noticed they were still dying (based on attempt count in task ID). However I did not inspect YARN logs again to verify it was memory related, I should have.
I tried running webindex copy
on EC2 and saw the following failure. I suspect the remote task is trying to access the paths which is on the machine that started the job but not on any of the workers.
Caused by: java.io.FileNotFoundException: File file:/home/ec2-user/webindex/paths/2015-18.wat.paths does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
Currently webindex does some odd things because it obtains Accumulo config from Fluo config for its external Accumulo index. I think it would be cleaner if Accumulo info were explicit webindex config. Then webindex could use its Accumulo config to configure Fluo and its external index table. With this approach I think all init could be done Java, avoiding editing the app props file and then calling fluo init
.
The new asyn commit manager uses a semaphore to limit the amount transactions committing asynchronously based on memory. I ran into a problem running webindex locally where a single transaction exceeded this limit and blocked forever. I wrote a Limit class for situation, need to use that.
Repeatedly seeing the following error in getDomain.
16:46:24.087 [pool-10-thread-38] WARN io.fluo.core.worker.WorkTask - Failed to execute observer CollisionFreeMapObserver notification : um:u:4 fluoRecipes cfm:um 153397
java.lang.RuntimeException: java.text.ParseException: Invalid host: whattoexpect.co.au
at io.fluo.webindex.data.fluo.UriMap$UriUpdateObserver.getDomain(UriMap.java:148) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
at io.fluo.webindex.data.fluo.UriMap$UriUpdateObserver.updatingValues(UriMap.java:135) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
at io.fluo.recipes.map.CollisionFreeMap.process(CollisionFreeMap.java:203) ~[fluo-recipes-core-1.0.0-beta-1-SNAPSHOT.jar:1.0.0-beta-1-SNAPSHOT]
at io.fluo.recipes.map.CollisionFreeMapObserver.process(CollisionFreeMapObserver.java:44) ~[fluo-recipes-core-1.0.0-beta-1-SNAPSHOT.jar:1.0.0-beta-1-SNAPSHOT]
at io.fluo.core.worker.WorkTask.run(WorkTask.java:69) ~[fluo-core-1.0.0-beta-2-SNAPSHOT.jar:1.0.0-beta-2-SNAPSHOT]
at io.fluo.core.worker.NotificationProcessor$2.run(NotificationProcessor.java:131) [fluo-core-1.0.0-beta-2-SNAPSHOT.jar:1.0.0-beta-2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
Caused by: java.text.ParseException: Invalid host: whattoexpect.co.au
at io.fluo.webindex.data.util.LinkUtil.createURL(LinkUtil.java:42) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
at io.fluo.webindex.data.util.LinkUtil.getHost(LinkUtil.java:78) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
at io.fluo.webindex.data.util.LinkUtil.hasIP(LinkUtil.java:87) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
at io.fluo.webindex.data.util.LinkUtil.getReverseTopPrivate(LinkUtil.java:98) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
at io.fluo.webindex.data.fluo.UriMap$UriUpdateObserver.getDomain(UriMap.java:146) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
... 8 common frames omitted
I think it would make the code easier to follow if all of the code dealing with the query table were centralized. This would be the code for creating mutations and querying. The spark, Fluo, and web code would all make calls to this code.
@mikewalch and I had a conversation about this when talking about #16 . If this change is made, UriCountExport would be much shorter and could possibly be moved into UriMap.
This command was removed in #49. The command did not work when it was removed, it was probably broken by #16.
The command would rebuild the index table using the Fluo table. Need to consider the implications of outstanding notifications when implementing the command. Can the command use the url and domain maps?
While init.sh
initializes an empty Fluo and Accumulo, load.sh
will add pages to Fluo and combine with previous data. This should be written in Spark to parallelize the loading across several nodes.
It would be nice to create an internal API for the external webindex search table. Fluo and Spark code write to this table. Web app code reads from it. The code for interacting with this external table is spread far and wide. Would be nice to bring that code together into once place and have a simple API for it. This would have made a change like #71 easier to make and more important test. I manually tested the changes for #71. If the webindex search table had its own internal API, then that could have test.
The page of top links could offer the option to jump to a count. For example jump to pages with an inbound link count of 3000.
Create a run
command that scripts the running of webindex applications.
Both are used in code currently to refer to same thing. The term 'uri' makes more sense.
The init script command is failing with the following error :
cp: missing destination file operand after `/home/kturner/fluo-dev/install/fluo-1.0.0-beta-2-SNAPSHOT/apps/webindex/lib'
I think its because $WI_DATA_JAR
is not being set and that is causing the following command to fail in bin/impl/init.sh
.
cp $WI_DATA_JAR $FLUO_APP_LIB
These were created to simplify running WebIndex. With the creation of the dev server (run via webindex dev
), they are no longer needed for several reasons:
webindex dev
While working on apache/fluo#593 I was testing on a 10 node EC2 cluster. I did not want lots of tablets so I lowered that config, however that meant the collision free map and export queue had a few large buckets. Processing these buckets resulted in very large transactions. Would be better to have the option to have a few tablets and lots of buckets.
Currently webindex init
copies needed jars to the Fluo app lib using the Maven command below:
mvn dependency:get -Dartifact=org.apache.fluo:fluo-recipes-core:1.0.0-incubating-SNAPSHOT:jar -Ddest=$FLUO_APP_LIB
It would be better if the version did not have be included using the command below (copied from the change @keith-turner made to stresso):
mvn dependency:copy-dependencies -DincludeArtifactIds=fluo-recipes-core -DoutputDirectory=$FLUO_APP_LIB
The indexes should match what is created by the the Spark Init job.
The test should perform the following steps:
There is code in example that can build the Query table from the Fluo table This code was broken in #16
Code should be added to WebIndex to send metrics to InfluxDB (using dropwizard) and set up dashboard in Grafana to view them.
Below are some possible metrics to send:
These metrics could be sent using both dropwizard Meters (for rates) and Counters (for totals):
The Spark Load job currently parses all pages in the input data set before inserting pages into the Fluo table. It would be better if the loader was modified to insert data as the input is parsed.
Need to update default splits after switching to use collision free map.
PageMap & PageInfo might be easier to understand.
While running webindex on EC2 I have noticed the link parsing done by the load task is very CPU intensive. This is usually the bottleneck for loading data when running one load task per node.
For example on a 20 node m3.xlarge EC2 cluster with 20 load task running, the maximum load rate is around 1000 pages/sec. As load increases on the system from having more data (caused by compactions, etc), this takes more CPU and causes the load rate to drop.
While working on #49 I realized the Fluo code may be computing domain counts incorrectly. I think its computing the number of links to a domain instead of the unique number of URI's seen in a domain.
For the shaded jar, need to ensure that the versions of Fluo, Accumulo, Hadoop, and Spark installed on the system are used. For Spark and Hadoop this can accomplished by excluding those from the shaded jar. For Fluo and Accumulo and fluo version
and accumulo version
commands can be used when building the shaded jar.
Just made changes like this for stress :
Was seeing the following in Accumulo during a long run of webindex. Would be nice to reorganize the index so there are not huge rows.
2016-05-04 15:23:52,429 [tablet.Tablet] WARN : tserver:worker6 Cannot split tablet 2;d:com.blogger;d:com.blogg it contains a big row : d:com.blogger
2016-05-04 15:25:42,570 [tablet.Tablet] WARN : tserver:worker2 Cannot split tablet 2;d:com.twitter;d:com.tu it contains a big row : d:com.twitter
This should increase performance. See http://spark.apache.org/docs/latest/tuning.html
The unit test should verify the output of Spark indexing to Fluo and Accumulo. It should also verify code that reads from Fluo and rebuilds indexes in Accumulo.
Freemarker templates should be editable while running dev server. This would make development easier and faster.
The load spark jobs are failing with Accumulo 1.8.0-SNAPSHOT. The problem seems to be that Accumulo upgraded to thrift 0.9.3 and spark includes an older version of thrift. The older version of thrift in spark gets picked up firsts and this causes Accumulo client code to fail.
The spark code deals with RowColumns and Values. I think it would be simpler if it dealt with POJOs, especially after #16. What the spark job is doing made a lot more sense before #16.
If the code was reworked to use POJOs, the RDDs with POJOs could then be converted into the Fluo and Query tables at the end of processing.
Could make use of Java 8 and reduce amount of code
While running a very long webindex run on EC2 I noticed the load job created 20 task each with 100's of files to load. Each file takes a while to process. The finish times of the task were highly skewed. Many more task with less files per task would better.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.