parsa-epfl / cloudsuite Goto Github PK
View Code? Open in Web Editor NEWA Benchmark Suite for Cloud Services.
Home Page: http://cloudsuite.ch
License: Other
A Benchmark Suite for Cloud Services.
Home Page: http://cloudsuite.ch
License: Other
Hi~!
I want to test my cloud.
So, I try to CloudSuite.
But Data_Analytics isn't working.....
Data_Analytics benchmark show this message.
Mahout: seqwiki
MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
Running on hadoop, using /opt/hadoop-2.7.3/bin/hadoop and HADOOP_CONF_DIR=/opt/hadoop-2.7.3/etc/hadoop
MAHOUT-JOB: /opt/mahout-0.12.2/mahout-examples-0.12.2-job.jar
17/08/07 12:45:51 INFO WikipediaToSequenceFile: Input: /user/root/wiki Out: /user/root/wiki-seq Categories: /root/categories All Files: false
17/08/07 12:45:52 INFO RMProxy: Connecting to ResourceManager at master/10.0.9.2:8032
17/08/07 12:45:53 WARN JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/08/07 12:45:54 INFO FileInputFormat: Total input paths to process : 1
17/08/07 12:45:54 INFO JobSubmitter: number of splits:5
17/08/07 12:45:55 INFO JobSubmitter: Submitting tokens for job: job_1502109896912_0001
17/08/07 12:45:55 INFO YarnClientImpl: Submitted application application_1502109896912_0001
17/08/07 12:45:55 INFO Job: The url to track the job: http://master:8088/proxy/application_1502109896912_0001/
17/08/07 12:45:55 INFO Job: Running job: job_1502109896912_0001
And i can't see next step....
I already complete this step in other computer.
How can i solve this problem?
I want to see next step.... :(
Help me~!~!
Thank you!!
Hello all,
I am trying to run media streaming benchmark by keeping the server in one machine and client in another machine. I have exposed and published the port while running the server as this should allow any incoming request from outside. I am wondering if I am doing anything wrong in port binding or in setting up the container network (I am using the User-defined networks i.e. streaming_network)
I have downloaded all sources from the CloudSuite github repo and followed the steps as below to run a test:
To build dataset*******
hostmachine1:
-/cloudsuite-master/benchmarks/media-streaming/dataset$ sudo docker build -t dataset .
-/cloudsuite-master/benchmarks/media-streaming/dataset$ sudo docker create --name streaming_dataset dataset
create network b/w client and server*******
hostmachine1:
-/cloudsuite-master/benchmarks/media-streaming/dataset$ sudo docker network create streaming_network
To run the server*******
hostmachine1:
-/cloudsuite-master/benchmarks/media-streaming/server$ sudo docker build -t server .
-/cloudsuite-master/benchmarks/media-streaming/server$ sudo docker run -p 80:80 -d --name streaming_server --volumes-from streaming_dataset --net streaming_network server
Now on another machine I have built the client by following the steps as below:
create network b/w client and server*******
hostmachine 2:
-/cloudsuite-master/benchmarks/media-streaming/client$ sudo docker network create streaming_network
To run the client*******
hostmachine2:
-/cloudsuite-master/benchmarks/media-streaming/client$ sudo docker build -t client .
-/cloudsuite-master/benchmarks/media-streaming/client$ sudo docker run -t --name=streaming_client -v /path/to/output:/output --volumes-from streaming_dataset --net streaming_network client streaming_server
This gives me an error msg:
docker: Error response from daemon: No such container: streaming_dataset.
This error is very obvious as I don't have the streaming dataset in host machine 2. I tried to replace the streaming_dataset with the container ID and streaming_server with the host IP
-/cloudsuite-master/benchmarks/media-streaming/client$ sudo docker run -t --name=streaming_client -v /path/to/output:/output --volumes-from 7be24343fb69 --net streaming_network client 143.117.95.33
docker: Error response from daemon: No such container: 7be24343fb69.
Then I removed the dataset and also I have changed the script hostlist.server(from files/run) by replacing streaming_server with my host machine 1's IP address. Then I tried to run the client to see if I can connect to the host machine 1 but once again it failed.
-/cloudsuite-master/benchmarks/media-streaming/client$ sudo docker run -t --name=streaming_client -v /path/to/output:/output --net streaming_network client 143.117.95.33
Total clients = 4
Minimum number of sessions = 25
Maximum number of sessions = 500
Launching 4 clients on localhost
Running command httperf --hog --server 143.117.95.34 --videosesslog=[/videos/logs/cl*],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log --num-sessions=25 --rate=2 2>>/output/bt1.trace
Running command httperf --hog --server 143.117.95.34 --videosesslog=[/videos/logs/cl*],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result2.log --num-sessions=25 --rate=2 2>>/output/bt2.trace
Running command httperf --hog --server 143.117.95.34 --videosesslog=[/videos/logs/cl*],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result3.log --num-sessions=25 --rate=2 2>>/output/bt3.trace
Running command httperf --hog --server 143.117.95.34 --videosesslog=[/videos/logs/cl*],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result4.log --num-sessions=25 --rate=2 2>>/output/bt4.trace
sizeof(fd_set) = 128
sizeof(fd_set) = 128
sizeof(fd_set) = 128
sizeof(fd_set) = 128
peak_hunter/launch_hunt_bin.sh: line 55: 0*100/0: division by 0 (error token is "0")
Benchmark succeeded for 25 sessions
Launching 4 clients on localhost
Running command httperf --hog --server 143.117.95.34 --videosesslog=[/videos/logs/cl*],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log --num-sessions=500 --rate=50 2>>/output/bt1.trace
Running command httperf --hog --server 143.117.95.34 --videosesslog=[/videos/logs/cl*],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result2.log --num-sessions=500 --rate=50 2>>/output/bt2.trace
Running command httperf --hog --server 143.117.95.34 --videosesslog=[/videos/logs/cl*],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result3.log --num-sessions=500 --rate=50 2>>/output/bt3.trace
Running command httperf --hog --server 143.117.95.34 --videosesslog=[/videos/logs/cl*],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result4.log --num-sessions=500 --rate=50 2>>/output/bt4.trace
sizeof(fd_set) = 128
sizeof(fd_set) = 128
sizeof(fd_set) = 128
sizeof(fd_set) = 128
peak_hunter/launch_hunt_bin.sh: line 55: 0*100/0: division by 0 (error token is "0")
Benchmark succeeded for 500 sessions
Maximum limit for number of sessions too low.
Requests: 0
Replies: 0
Runtime error (func=(main), adr=9): Divide by zero
Reply rate:
Runtime error (func=(main), adr=9): Divide by zero
Reply time:
Net I/O: 0
sed: -e expression #1, char 19: unterminated `s' command
sed: -e expression #1, char 19: unterminated `s' command
sed: -e expression #1, char 19: unterminated `s' command
sed: -e expression #1, char 19: unterminated `s' command
........
I was wondering if I need to change anything in the Dockerfile or in any of the given scripts? Can someone please advise me on how to run the streaming server and client containers in two different machines as the benchmark is designed to run both the server and client in the same host?
Many thanks,
Esha Barlaskar
There's no README. Would be useful to know:
elgg_ready_dump.sql file for generating users (generate_users.sh) in the db_server component of the webserving benchmark is missing.
It's hard for me to get larger index file than 12GB index file offered by CloudSuite3.0
How can I get larger index file to run web search benchmark, or.. If you already have bigger index file, could you share for me? I think it would be 20GB~30GB size or something..
Hello, I have a problem with data caching benchmark.
As shown in the guide, I run the following instruction first:
./loader -a ../twitter_dataset/twitter_dataset_30x -s docker_servers.txt -g 0.8 -T 1 -c 200 -w 8
it seem to have been running (for one day..), and I see that the rps is about 470000. So I go on the next instruction:
./loader -a ../twitter_dataset/twitter_dataset_30x -s docker_servers.txt -g 0.8 -T 1 -c 200 -w 8 -e -r rps
where I use 470000*0.9 = 423000 as the rps value. But it still seem to have been running....(more than one day) and never stop.
I also have tried another rps value (200000), but it has the same problem.
I want to know: Do I use the benchmark in a correct way? In my imagination, the running progress will end and report the results, but now it never stops...
Many thanks!
Hi,
Where to download the Twitter dataset used in the data caching benchmark? The tutorial did not provide the download links. Thanks!
The serf package acquired here: https://github.com/ParsaLab/cloudsuite/blob/master/benchmarks/data-analytics/base/Dockerfile#L21
has been moved. Please pick the most suitable replacement from here and update.
HI,
In Data analytic benchmark, in final step which is running ./run.sh
every thing goes forward well,
but some mahout commands in the end of file do not run correctly and encounter some error:
Running on hadoop, using /opt/new_analytic/hadoop-2.7.1/bin/hadoop and HADOOP_CONF_DIR= MAHOUT-JOB: /opt/new_analytic/apache-mahout-distribrtuon-0.11.0/examples/target/mahout-examples-0.11.0-job.jar ERROR MahoutDriver: : Try new vector backed naivebays classifier see examples/bin/classify-20newsgroups.sh
and according to this link and this link it seems the mahout in image file has deprecated files.
Hi all,
This issue is related to the issue #87.
I was trying to run media streaming benchmark from cloudsuite in swarm mode by following the instructions given in http://cloudsuite.ch//2015/12/20/swarm/. I was expecting that with an overlay network in swarm mode I can make containers from different nodes communicate. But every time I use a locally built image or if I use the images from a private repository from my docker hub account, swarm fails to start the containers. I even tried to pull the images from cloudsuite; still it fails. Below are the steps that I have followed so far:
I have run all the commands in swarm master and pulled the images in swarm worker node 1, node 2 and node 3 as below:
To build dataset
swarm_node1:
sudo docker pull cloudsuite/media-streaming:dataset
Swarm_master:
$ sudo docker service create --name streaming_dataset --network swarm-network --constraint "node.id==ffx5v6f1ddnxtb1g8434q27dn" -d cloudsuite/media-streaming:dataset
33612kongezcbodk1o7n4lv8b
$ sudo docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
33612kongezc streaming_dataset replicated 0/1 cloudsuite/media-streaming:dataset
Although the container is created, it fails to start in the worker node 1 or any other worker nodes and I get the following information:
$ sudo docker service ps streaming_dataset --no-trunc
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
rpl6yvtqy7qdcx3pw7pdbf5oz streaming_dataset.1 cloudsuite/media-streaming:dataset@sha256:cd3f09ae1f75ccb91608ec3ac6cf9cba1cf63f83bee4e5c92992bd8d5b144879 swarm1 Running Preparing 5 seconds ago
w456i5k8wotxrzds15tffiiji _ streaming_dataset.1 cloudsuite/media-streaming:dataset@sha256:cd3f09ae1f75ccb91608ec3ac6cf9cba1cf63f83bee4e5c92992bd8d5b144879 swarm1 Shutdown Complete 5 seconds ago
j5wh0m90el0uoay3rf31xt0bs _ streaming_dataset.1 cloudsuite/media-streaming:dataset@sha256:cd3f09ae1f75ccb91608ec3ac6cf9cba1cf63f83bee4e5c92992bd8d5b144879 swarm1 Shutdown Complete about a minute ago
sreqi2kbovb5wu3cq4t0dme3b _ streaming_dataset.1 cloudsuite/media-streaming:dataset@sha256:cd3f09ae1f75ccb91608ec3ac6cf9cba1cf63f83bee4e5c92992bd8d5b144879 swarm1 Shutdown Complete 3 minutes ago
To run the server
Next, I tried to run the streaming server and it was working fine.
Swarm Master:
$ sudo docker service create --name streaming_server --network swarm-network --mount type=volume,source=streaming_dataset,destination=/streaming_dataset --constraint "node.id==ffx5v6f1ddnxtb1g8434q27dn" cloudsuite/media-streaming:server
ic3h22efitlc0khdl3dv6as7p
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
$ sudo docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
ic3h22efitlc streaming_server replicated 1/1 cloudsuite/media-streaming:server
ijov1x016sf5 streaming_dataset replicated 0/1 cloudsuite/media-streaming:dataset
To run the client
As the server was running in swarm node 1 so I tried to run the client in swarm node 3:
Swarm Master:
$ sudo docker service create -t --name=streaming_client --mount type=volume,source=streaming_dataset,destination=/path/to/output:/output --constraint "node.id==55cipngnlvcp1z7le0vnhgfx2" --network swarm-network cloudsuite/media-streaming:client streaming_server
viiu9vfcn99da1nifri8cymdq
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
Although the client is deployed in swarm node 3 but it fails after few seconds:
$ sudo docker service ps streaming_client
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
9yvb1kl81y3v streaming_client.1 cloudsuite/media-streaming:client swarm3 Ready Ready 3 seconds ago
dy9hte7l3xbw _ streaming_client.1 cloudsuite/media-streaming:client swarm3 Shutdown Failed 4 seconds ago "task: non-zero exit (1)"
ubh8u9qy60cy _ streaming_client.1 cloudsuite/media-streaming:client swarm3 Shutdown Failed 11 seconds ago "task: non-zero exit (1)"
jnhd85bpbd0k _ streaming_client.1 cloudsuite/media-streaming:client swarm3 Shutdown Failed 18 seconds ago "task: non-zero exit (1)"
zo1nhf69zppm _ streaming_client.1 cloudsuite/media-streaming:client swarm3 Shutdown Failed 24 seconds ago "task: non-zero exit (1)"
Am I missing something in the process of deploying the media streaming client and dataset in swarm mode? I would appreciate any sort of assistance in this regard.
P.S: other benchmarks(GraphAnalytics, DataCaching) from Cloudsuite are also failing the same way in swarm mode.
Many thanks.
Esha
Apr 15, 2017 9:50:45 AM com.sun.faban.driver.engine.AgentThread logError
[java] WARNING: ElggDriverAgent[0].112.addFriend: Server returned HTTP response code: 502 for URL: http://10.6.9.155:8050/action/friends/add?friend=362&__elgg_ts=1492249412&__elgg_token=f46722e9c46ebaed9b2f4495933a031b
[java] java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.6.9.155:8050/action/friends/add?friend=362&__elgg_ts=1492249412&__elgg_token=f46722e9c46ebaed9b2f4495933a031b
These kinds of error occur and throughput degrades drastically.
Sometimes there is HTTP response code: 503 and sometimes HTTP response code: 504
Is it due to server saturation/overload?
Seems that we need to add the following line to the appropriate dokerfile (Cassandra server)
chmod +x /docker-entrypoint.sh
ustiugov@n133:~$ docker run -it --name cassandra-server --net serving_network cloudsuite/data-serving:server bash
exec: "/docker-entrypoint.sh": permission denied
Error response from daemon: Cannot start container 0d4485fc7227af364ccee95ba814682d903a3476cc04ad0d8f74c44fc17a1f11: [8] System error: exec: "/docker-entrypoint.sh": permission denied
command:
docker inspect --format '{{ .NetworkSettings.Networks.wsnet.IPAddress }}' web_server
result:
<no value>
I am experiencing issues when running the newest version of Cloudsuite’s Web Search benchmark (docker images from dockerhub).
All the client runs start in a similar fashon:
Starting Faban Server
Please point your browser to http://b81b4338b185:9980/
Buildfile: /usr/src/faban/search/build.xml
init:
[mkdir] Created dir: /usr/src/faban/search/build/classes
compile:
[javac] /usr/src/faban/search/build.xml:35: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 1 source file to /usr/src/faban/search/build/classes
[javac]
[javac] WARNING
[javac]
[javac] The -source switch defaults to 1.8 in JDK 1.8.
[javac] If you specify -target 1.5 you now must also specify -source 1.5.
[javac] Ant will implicitly add -source 1.5 for you. Please change your build file.
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.5
[javac] warning: [options] source value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] target value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
[javac] 4 warnings
bench.jar:
[mkdir] Created dir: /usr/src/faban/search/build/lib
[jar] Building jar: /usr/src/faban/search/build/lib/search.jar
deploy.jar:
[jar] Building jar: /usr/src/faban/search/build/search.jar
deploy:
BUILD SUCCESSFUL
Total time: 9 seconds
Print= :1
Jun 06, 2018 1:18:36 PM com.sun.faban.common.RegistryImpl main
INFO: Registry started.
Usage: AgentImpl <driverName> <agentId> <masterMachine>
Jun 06, 2018 1:18:39 PM com.sun.faban.common.RegistryImpl reregister
INFO: Registering Master on 172.19.0.3
Jun 06, 2018 1:18:39 PM com.sun.faban.driver.engine.MasterImpl runBenchmark
INFO: RunID for this run is : 1
Jun 06, 2018 1:18:39 PM com.sun.faban.driver.engine.MasterImpl runBenchmark
INFO: Output directory for this run is : /usr/src/outputFaban/1
Jun 06, 2018 1:18:40 PM com.sun.faban.common.RegistryImpl getServices
INFO: Get services by type: SearchDriverAgent
Jun 06, 2018 1:18:40 PM com.sun.faban.common.RegistryImpl getServices
WARNING: Registry.getServices : Cannot find Service type : SearchDriverAgent
Jun 06, 2018 1:18:40 PM com.sun.faban.driver.engine.MasterImpl configure
WARNING: Cannot find SearchDriverAgents. Not starting SearchDriver.
Jun 06, 2018 1:18:40 PM com.sun.faban.driver.util.Timer idleTimerCheck
INFO: SearchDriverAgent[0]: Performing idle timer check
Jun 06, 2018 1:18:40 PM com.sun.faban.driver.util.Timer idleTimerCheck
INFO: SearchDriverAgent[0]: Idle timer characteristics:
Accuracy=3,
min. invocation cost=32,
med. invocation cost (math)=44.0,
med. invocation cost (phys)=44,
avg. invocation cost=48.354,
max. invocation cost=358,
variance of invocation cost=251.35208400000352.
Jun 06, 2018 1:18:43 PM com.sun.faban.driver.engine.AgentImpl run
INFO: SearchDriverAgent[0]: Successfully started 10 driver threads.
However, later only every third/fourth run of Web Search client is successful, the rest are failing with one of the three errors (pretty randomly):
Jun 06, 2018 4:47:15 PM com.sun.faban.driver.engine.TimeThread doRun
SEVERE: SearchDriverAgent[0].7: Error initializing driver object.
java.lang.NullPointerException
at com.sun.org.apache.xerces.internal.dom.DeferredElementNSImpl.synchronizeData(DeferredElementNSImpl.java:108)
at com.sun.org.apache.xerces.internal.dom.ElementNSImpl.getNamespaceURI(ElementNSImpl.java:250)
at com.sun.org.apache.xml.internal.dtm.ref.dom2dtm.DOM2DTM.addNode(DOM2DTM.java:263)
at com.sun.org.apache.xml.internal.dtm.ref.dom2dtm.DOM2DTM.nextNode(DOM2DTM.java:524)
at com.sun.org.apache.xml.internal.dtm.ref.DTMDefaultBase._nextsib(DTMDefaultBase.java:567)
at com.sun.org.apache.xml.internal.dtm.ref.DTMDefaultBase.getNextSibling(DTMDefaultBase.java:1142)
at com.sun.org.apache.xml.internal.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(DTMDefaultBaseTraversers.java:465)
at com.sun.org.apache.xpath.internal.axes.AxesWalker.getNextNode(AxesWalker.java:337)
at com.sun.org.apache.xpath.internal.axes.AxesWalker.nextNode(AxesWalker.java:365)
at com.sun.org.apache.xpath.internal.axes.WalkingIterator.nextNode(WalkingIterator.java:197)
at com.sun.org.apache.xpath.internal.axes.NodeSequence.nextNode(NodeSequence.java:344)
at com.sun.org.apache.xpath.internal.axes.NodeSequence.item(NodeSequence.java:539)
at com.sun.org.apache.xpath.internal.objects.XNodeSet.str(XNodeSet.java:281)
at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.getResultAsType(XPathImpl.java:309)
at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:274)
at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:363)
at com.sun.faban.driver.engine.DriverContext.getXPathValue(DriverContext.java:412)
at sample.searchdriver.SearchDriver.<init>(SearchDriver.java:86)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at com.sun.faban.driver.engine.TimeThread.doRun(TimeThread.java:73)
at com.sun.faban.driver.engine.AgentThread.run(AgentThread.java:202)
Jun 06, 2018 4:47:16 PM com.sun.faban.driver.engine.AgentImpl kill
WARNING: SearchDriverAgent[0]: Killing benchmark run
Jun 06, 2018 4:47:16 PM com.sun.faban.driver.engine.AgentThread run
SEVERE: SearchDriverAgent[0].6: null
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at com.sun.faban.driver.engine.AgentThread.waitStartTime(AgentThread.java:470)
at com.sun.faban.driver.engine.TimeThread.doRun(TimeThread.java:96)
at com.sun.faban.driver.engine.AgentThread.run(AgentThread.java:202)
Jun 07, 2018 8:26:37 AM com.sun.faban.driver.engine.TimeThread doRun
SEVERE: SearchDriverAgent[0].5: Error initializing driver object.
java.io.FileNotFoundException: (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at sample.searchdriver.SearchDriver.<init>(SearchDriver.java:90)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at com.sun.faban.driver.engine.TimeThread.doRun(TimeThread.java:73)
at com.sun.faban.driver.engine.AgentThread.run(AgentThread.java:202)
Jun 07, 2018 8:26:38 AM com.sun.faban.driver.engine.AgentImpl kill
WARNING: SearchDriverAgent[0]: Killing benchmark run
Jun 07, 2018 8:26:38 AM com.sun.faban.driver.engine.AgentThread run
SEVERE: SearchDriverAgent[0].9: null
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at com.sun.faban.driver.engine.AgentThread.waitStartTime(AgentThread.java:470)
at com.sun.faban.driver.engine.TimeThread.doRun(TimeThread.java:96)
at com.sun.faban.driver.engine.AgentThread.run(AgentThread.java:202)
Jun 07, 2018 11:49:36 AM com.sun.faban.driver.engine.MasterImpl executeRun
INFO: Started all threads; run commences in 2998 ms
Jun 07, 2018 11:49:41 AM sample.searchdriver.SearchDriver doGet
SEVERE: ERROR!
Jun 07, 2018 11:49:41 AM com.sun.faban.driver.engine.AgentThread validateTimeCompletion
SEVERE: SearchDriverAgent[0].4.doGet: Transport incomplete! Please ensure transport exception is thrown from operation.
Jun 07, 2018 11:49:41 AM com.sun.faban.driver.engine.AgentThread run
SEVERE: SearchDriverAgent[0].4: SearchDriverAgent[0].4.doGet: Transport incomplete! Please ensure transport exception is thrown from operation.
com.sun.faban.driver.FatalException: SearchDriverAgent[0].4.doGet: Transport incomplete! Please ensure transport exception is thrown from operation.
at com.sun.faban.driver.engine.AgentThread.validateTimeCompletion(AgentThread.java:532)
at com.sun.faban.driver.engine.TimeThread.doRun(TimeThread.java:173)
at com.sun.faban.driver.engine.AgentThread.run(AgentThread.java:202)
Jun 07, 2018 11:49:42 AM com.sun.faban.driver.engine.AgentImpl kill
WARNING: SearchDriverAgent[0]: Killing benchmark run
Jun 07, 2018 11:49:42 AM com.sun.faban.driver.engine.AgentThread logError
WARNING: SearchDriverAgent[0].14.doGet: java.lang.RuntimeException: Sleep interrupted. Run terminating.
Note: Error not counted in result.
Either transaction start or end time is not within steady state.
java.lang.RuntimeException: java.lang.RuntimeException: Sleep interrupted. Run terminating.
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1506)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3036)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.faban.driver.transport.sunhttp.SunHttpTransport.readURL(SunHttpTransport.java:177)
at com.sun.faban.driver.transport.sunhttp.SunHttpTransport.readURL(SunHttpTransport.java:191)
at com.sun.faban.driver.transport.sunhttp.SunHttpTransport.readURL(SunHttpTransport.java:217)
at sample.searchdriver.SearchDriver.doGet(SearchDriver.java:197)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.faban.driver.engine.TimeThread.doRun(TimeThread.java:169)
at com.sun.faban.driver.engine.AgentThread.run(AgentThread.java:202)
Caused by: java.lang.RuntimeException: Sleep interrupted. Run terminating.
at com.sun.faban.driver.util.Timer.wakeupAt(Timer.java:405)
at com.sun.faban.driver.engine.DriverContext.recordStartTime(DriverContext.java:599)
at com.sun.faban.driver.transport.util.TimedOutputStream.write(TimedOutputStream.java:126)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.PrintStream.flush(PrintStream.java:338)
at sun.net.www.MessageHeader.print(MessageHeader.java:301)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:644)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:655)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:693)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1585)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
... 9 more
If I simply remove the client container and try running it again then, after few attempts, eventually I achieve a successful run, without applying any changes to the setup.
Any ideas how to make the environment more reliable? I was running experiments with an older version of web search benchmark in the past and didn't experience issues like this.
I'm running the client on:
Hi,
I am starting to use qflex and I would like to have the setup in-memory data analytics (spark) on a qcow image. I think the docker doesn't seem feasible on top of the image running with ARM emulation mode. It doesn't look like that the memcached image distributed with qflex runs with the docker. Can you share any documents installing the in-memory analytics used in cloudsuite if exist?
Thanks,
Jeongseob
Create a new task on Pivotal Tracker whenever a new issue is created.
I am trying to build the
docker images locally and in an offline manner. Here is what have done already
1) docker load -i /home/mahmood/cloudsuite_v3/ubuntu.tar
2) mkdir -p data-analytics/dataset
3) vim data-analytics/dataset/Dockerfile
FROM ubuntu:10.04
RUN apt-get update && apt-get install wget -y
RUN mkdir /data
RUN cd /data \
&& wget -nv http://parsa.epfl.ch/cloudsuite/software/enwiki-20100904-pages-articles1.xml.bz2
VOLUME ["/data"]
4) docker build data-analytics/dataset
the last lines of a successful build looks like
Step 5 : VOLUME /data
---> Running in 7d9f2f95c6e8
---> 5d0c5e9188d9
Removing intermediate container 7d9f2f95c6e8
Successfully built 5d0c5e9188d9
Then, according to the guide, the next command is
docker create --name data cloudsuite/data-analytics:dataset
However, am I not able to run that. Instead, I ran
docker build -t data-analytics:dataset ./data-analytics/dataset
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu:14.04
---> 780bd4494e2f
Step 2 : RUN apt-get update && apt-get install wget -y
---> Using cache
---> 4418b70655ff
Step 3 : RUN mkdir /data
---> Using cache
---> 9db3edf4a620
Step 4 : RUN cd /data && wget -nv http://parsa.epfl.ch/cloudsuite/software/enwiki-20100904-pages-articles1.xml.bz2
---> Using cache
---> fd44c4dc38ad
Step 5 : VOLUME /data
---> Using cache
---> 5d0c5e9188d9
Successfully built 5d0c5e9188d9
Now, I have to start the master. What are the steps then? The pull command is used for online activities through docker. I want to build it offline from a local image
The datacaching instructions say to use vim.tiny to edit docker_servers.txt, but I can't seem to launch vim.tiny:
memcache@77d401e39036:/usr/src/memcached/memcached_client$ vim.tiny
bash: vim.tiny: command not found
memcache@77d401e39036:/usr/src/memcached/memcached_client$ vim.tiny dockers_servers.txt
bash: vim.tiny: command not found
It doesn't appear to be installed either:
memcache@77d401e39036:/usr/src/memcached/memcached_client$ ls /usr/bin
[ debconf-set-selections hostnamectl nawk sdiff tee
addpart debconf-show i386 newgrp select-editor test
apt delpart iconv nice sensible-browser tic
apt-cache diff id nl sensible-editor timedatectl
apt-cdrom diff3 infocmp nohup sensible-pager timeout
apt-config dircolors infotocap nproc seq tload
apt-get dirname install nsenter setarch toe
apt-key dpkg ionice numfmt setsid top
apt-mark dpkg-deb ipcmk od setterm touch
arch dpkg-divert ipcrm pager sg tput
awk dpkg-maintscript-helper ipcs partx sha1sum tr
base32 dpkg-query ischroot passwd sha224sum truncate
base64 dpkg-split join paste sha256sum tset
basename dpkg-statoverride last pathchk sha384sum tsort
bashbug dpkg-trigger lastb perl sha512sum tty
bootctl du lastlog perl5.22.1 shred tzselect
busctl env ldd pg shuf unexpand
captoinfo expand line pgrep skill uniq
catchsegv expiry link pinky slabtop unlink
chage expr linux32 pkill snice unshare
chattr factor linux64 pldd sort update-alternatives
chcon faillog locale pmap split uptime
chfn fallocate localectl pr stat users
chrt find localedef printenv stdbuf utmpdump
chsh flock logger printf sum vmstat
cksum fmt logname prlimit systemd-analyze w
clear fold lsattr ptx systemd-cat w.procps
clear_console free lscpu pwdx systemd-cgls wall
cmp getconf lsipc realpath systemd-cgtop watch
comm getent lslocks rename.ul systemd-delta wc
csplit getopt lslogins renice systemd-detect-virt whereis
cut gpasswd lspgpot reset systemd-path which
deb-systemd-helper gpg mawk resizepart systemd-resolve who
deb-systemd-invoke gpg-zip mcookie rev systemd-run whoami
debconf gpgsplit md5sum rgrep systemd-stdio-bridge x86_64
debconf-apt-progress gpgv md5sum.textutils runcon tabs xargs
debconf-communicate groups mesg savelog tac yes
debconf-copydb head mkfifo script tail zdump
debconf-escape hostid namei scriptreplay taskset
-link is legacy
please, refer to https://docs.docker.com/engine/userguide/networking/dockernetworks/
also, in case of data-serving, the proposed instructions do not work.
regards,
Dmitrii
Links work but travis complains
- ./_site/pages/publications/index.html
* External link https://infoscience.epfl.ch/record/176330/export/btex failed: response code 0 means something's wrong.
It's possible libcurl couldn't connect to the server or perhaps the request timed out.
Sometimes, making too many requests at once also breaks things.
Either way, the return message (if any) from the server is: SSL connect error
Hi,
I failed to run the SPARK ALS which is a benchmark of in-memory analytics when following the official documentation due to out of memory. According to the page, it looks like the ALS benchmark with the large dataset is working with 1g driver memory and 1g executor memory, but it wasn't for me. Here is output log file, als.oom.txt
I need to have at least 2g driver memory and 2g executor memory like below. It may not be criticial, but I think it would be better if the documentation can be updated.
$ docker run --rm --volumes-from data cloudsuite/in-memory-analytics /data/ml-latest /data/myratings.csv --driver-memory 2g --executor-memory 4g
Thanks,
Jeongseob
Hi
In web serving benchmark all steps run well, but the last step which wants to run ./run.sh is the problem!
in fact there is no file named run.sh in client container.
The mentioned file run.sh is in which container? server or client?
thank.
According to my search I figure out that cloudsuite only support cassendra,memcached,Spark. Is there anyone else for which cloudsuite benchmark can be used? I have visited your official website as well and find these three systems benchmarking with different workloads. Actually i want to know about the all those NoSQL systems which are supported by cloudsuite. Kindly help me.
This message shows up when creating containers for slaves and master. Pertinent to previous version of Hadoop as well.
The command is in file located at:
/opt/hadoop-2.7.4/sbin/hadoop-daemon.sh
cause: $USER is not defined
The current Dockerfile gives the following error:
W: GPG error: http://www.apache.org 21x InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A278B781FE4B2BDA
WARNING: The following packages cannot be authenticated!
cassandra
E: There are problems and -y was used without --force-yes
Adding the key in
$ apt-key adv --keyserver ha.pool.sks-keyservers.net --recv-keys 514A2AD631A57A16DD0047EC749D6EEC0353B12C
makes it work.
I was trying to run the benchmark by increasing the value of MAX_PM_CHILDREN
from its default value 80 to 200 using the command:
sudo docker run -idt --net=user_nw --name=web_server cloudsuite/web-serving:web_server /etc/boostrap.sh ${DATABASE_SERVER_IP} ${MEMCACHED_SERVER_IP} 200
Here where user_nw
is a user defined network.
Now when I run the faban_client
using
sudo docker run --net=user_nw --name=faban_client cloudsuite/web-serving:faban_client 172.21.0.7
it just does not run and the screen is blank. When I do dmesg
I get the following errors:
[690583.988429] traps: php5-fpm[1623] trap stack segment ip:79d0f4 sp:7ffd5a71cbe0 error:0 in php5-fpm[400000+800000]
[690585.097677] traps: php5-fpm[1630] trap stack segment ip:79d0f4 sp:7ffd5a71cbe0 error:0 in php5-fpm[400000+800000]
[690586.194116] traps: php5-fpm[1637] trap stack segment ip:79d0f4 sp:7ffd5a71cbe0 error:0 in php5-fpm[400000+800000]
[690587.294071] traps: php5-fpm[1644] trap stack segment ip:79d0f4 sp:7ffd5a71cbe0 error:0 in php5-fpm[400000+800000]
Any help in resolving this issue would be appreciable.
cloudsuite/benchmarks/web-serving/faban_client/files/web20_benchmark/src/workload/driver/Web20Driver.java
When I ran faban_client, I got the warning every time a new user logged in:
....
[java] Logging in: ZIFdBLBGXk
[java] Mar 15, 2016 11:52:08 PM com.sun.faban.driver.transport.sunhttp.ThreadCookieHandler$CookieStore select
[java] WARNING: Request http://web_server:8080/ does not contain a hostname or has an invalid hostname.
[java] Mar 15, 2016 11:52:08 PM com.sun.faban.driver.transport.sunhttp.ThreadCookieHandler$CookieStore select
[java] WARNING: Request http://web_server:8080/ does not contain a hostname or has an invalid hostname.
[java] Mar 15, 2016 11:52:09 PM com.sun.faban.driver.transport.sunhttp.ThreadCookieHandler$CookieStore select
[java] WARNING: Request http://web_server:8080/ does not contain a hostname or has an invalid hostname.
[java] hostUrl + LOGIN_URL: http://172.18.0.02:8080/action/login // this line was added by myself to see the login
[java] PRINT: Successfully logged in
....
The requests seemed to have reached the web server Ok which means I had the correct web server ip address for the faban_client.
(If it matters, I am using Linux Kernel 4.4.0-62-generic on x86_64 architecture using a recent version of QEMU (2.11.92)running in KVM mode)
Hello all,
I have been facing an issue when I run the Web Serving benchmark inside of a Virtual Machine in QEMU-KVM. I can start all of the docker images associated with the Web Service benchmark normally.
I use the below commands to start all of the services -
docker run -dt --net=host --name=mysql_server cloudsuite/web-serving:db_server 10.0.2.15
docker run -dt --net=host --name=memcache_server cloudsuite/web-serving:memcached_server
docker run -dt --net=host --name=web_server cloudsuite/web-serving:web_server /etc/bootstrap.sh 127.0.0.1 127.0.0.1 80 (I have tried with lesser children as well)
Finally I run the faban_client with the below command -
docker run --net=host --name=faban_client cloudsuite/web-serving:faban_client 10.0.2.15 7
(I tried using 127.0.0.1 for the Web-server IP address while launching the client as well)
Here 10.0.2.15 is the first IP address that is assigned to QEMU via DHCP. I have tried various combinations but I am still not able to fully run the Web-server benchmark because the client cannot establish connection to the server.
However, using the same set of steps I can establish connection to the server when I do not use a VM (i.e. in a bare metal environment).
I decided to look around the error messages-
Inside of the server's log file, I found error messages like these-
/var/log/nginx/examples_error.log
recv() failed (104 : Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "HEAD / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.2.15:8080"
Moreover, when I try to do a wget
to the server from inside of the VM, I get the below message-
wget http://10.0.2.15:8080/
Connecting to 10.0.2.15:8080... connected
HTTP Request sent, awaiting response... 502 Bad Gateway
So I get a Bad Gateway but the connection gets established.
I cannot seem to figure out what could be the source of this problem. Is there any way to identify and solve the cause of this problem ?
Thank You for your time.
Best Regards,
Arnab
I am trying to run the webserving benchmark (following the instructions here [1]).
When I launch faban in the last step [2] I get the following errors in dmesg:
............................
[47184.696252] traps: php5-fpm[6235] trap stack segment ip:79c8b4 sp:7ffd0a308970 error:0 in php5-fpm[4
00000+7ff000]
[47185.788775] traps: php5-fpm[6246] trap stack segment ip:79c8b4 sp:7ffd0a308970 error:0 in php5-fpm[4
00000+7ff000]
[47186.882077] traps: php5-fpm[6263] trap stack segment ip:79c8b4 sp:7ffd0a308970 error:0 in php5-fpm[4
00000+7ff000]
............................
Any ideas on how to fix this?
[1] http://cloudsuite.ch///pages/benchmarks/webserving/
[2] docker run --net=host --name=faban_client cloudsuite/web-serving:faban_client ${WEB_SERVER_IP} ${LOAD_SCALE}
The data-analytics benchmark is failing to build because the base image (the new Hadoop image) doesn't have bzip2 installed.
benchmarks/spark-base/spark-base uses SPARK_VERSION 1.5.1 and HADOOP_VERSiON 2.6 but the commons folder contains HADOOP 2.7.3 and SPARK 1.5.1 as well as 2.1.0. Hadoop is used in data-analytics and spark in in-memory-analytics and graph analytics. I do not know if they will create compatibility issues elsewhere.
Hello,
I'm using the web serving benchmark and all the components are on the localhost. I start them using the following commands:
docker run --security-opt seccomp:unconfined -dt --net=host --name=mysql_server cloudsuite/web-serving:db_server localhost
docker run --security-opt seccomp:unconfined -dt --net=host --name=memcache_server cloudsuite/web-serving:memcached_server
docker run --cap-add SYS_PTRACE --security-opt seccomp:unconfined -dt --net=host --name=web_server cloudsuite/web-serving:web_server /etc/bootstrap.sh
docker run --security-opt seccomp:unconfined --net=host --name=faban_client cloudsuite/web-serving:faban_client localhost
Then, the benchmark hung and never finish. I login to the web_server
docker and examine the PHP-FPM master process using strace -f -s 2000 -o app2.trc -p 37
and I see
root@kettle:/usr/share/nginx/html# strace -f -s 2000 -o app2.trc -p 37
Process 37 attached
Process 15234 attached
Process 15235 attached
Process 15236 attached
Process 15237 attached
Process 15238 attached
Process 15239 attached
Process 15240 attached
Process 15241 attached
^CProcess 37 detached
Process 15240 detached
Process 15241 detached
That's an indication of processes under PHP-FPM dying quickly. Then, I look into the trace app2.trc
and
I see
15234 write(2, "NOTICE: PHP message: PHP WARNING: 2018-04-21 19:55:45 (UTC): \"mm
ysql_connect(): Unknown MySQL server host 'mysql_server' (2)\" in file /usr/sharr
e/nginx/html/elgg/engine/classes/Elgg/Database.php (line 142)\n", 204) = 204
If I run strace command a little bit longer and I can see other warning messages
16349 connect(6, {sa_family=AF_LOCAL, sun_path="/var/run/mysqld/mysqld.sock"}, 11
10) = -1 ENOENT (No such file or directory)
16349 fcntl(6, F_SETFL, O_RDWR) = 0
16349 shutdown(6, SHUT_RDWR) = 0
16349 close(6) = 0
16349 write(2, "NOTICE: PHP message: PHP WARNING: 2018-04-21 20:17:25 (UTC): \"mm
ysql_real_escape_string(): Can't connect to local MySQL server through socket '//
var/run/mysqld/mysqld.sock' (2)\" in file /usr/share/nginx/html/elgg/engine/class
ses/Elgg/Database.php (line 602)\n", 255) = 255
16349 write(2, "NOTICE: PHP message: PHP WARNING: 2018-04-21 20:17:25 (UTC): \"mm
ysql_real_escape_string(): A link to the server could not be established\" in fii
le /usr/share/nginx/html/elgg/engine/classes/Elgg/Database.php (line 602)\n", 211
6) = 216
I think there might be some problem with my setup but I cannot figure out where.
Thanks for the help!
Since the web server is running inside docker, the absolute URLs will not resolve if you want to access the services from outside the container. For example, when you go to the web server main page I see this:
And I see the following errors in the console:
Notice how both hostname and port number are hard-coded, though I am accessing the server on both a different port and a different hostname. Maybe these URLs are generated by the backend, so it thinks the hostname and port are x and y, whereas they are actually remapped outside the container. To fix, these need to be relative paths instead.
Hello, I'm struggling with runnging the Web-Serving benchmark. I need some help.
I ran below commands.. I created new network named network1
docker network create network1
WEB_SERVER_IP="web_server"
docker run -dt --net=network1 --name=mysql_server cloudsuite/web-serving:db_server $WEB_SERVER_IP
docker run -dt --net=network1 --name=memcache_server cloudsuite/web-serving:memcached_server
docker run -dt --net=network1 --name=web_server cloudsuite/web-serving:web_server /etc/bootstrap.sh
docker run --net=network1 --name=faban_client cloudsuite/web-serving:faban_client $WEB_SERVER_IP
When I ran faban client, I got this warning message:
Starting Faban Server
Please point your browser to http://adcec9fe65da:9980/
Sep 12, 2017 5:02:22 AM com.sun.faban.driver.transport.sunhttp.ThreadCookieHandler$CookieStore select
WARNING: Request http://web_server:8080/action/login does not contain a hostname or has an invalid hostname.
And finally:
[java] variance of invocation cost=607.1004115899992.
[java] Sep 12, 2017 5:02:27 AM com.sun.faban.driver.engine.TimeThreadWithBackground doRun
[java] SEVERE: ElggDriverAgent[0].0: Error initializing driver object.
[java] java.lang.ArrayIndexOutOfBoundsException: 2
[java] at workload.driver.Web20Driver.<init>(Web20Driver.java:233)
[java] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[java] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
[java] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[java] Sep 12, 2017 5:02:27 AM com.sun.faban.driver.engine.AgentImpl run
[java] WARNING: ElggDriverAgent[0]: Run aborted before starting 7 driver threads.
[java] 1 threads were started.
[java] Sep 12, 2017 5:02:28 AM com.sun.faban.driver.engine.AgentImpl kill
[java] WARNING: ElggDriverAgent[0]: Killing benchmark run
[java] Sep 12, 2017 5:02:37 AM com.sun.faban.driver.engine.MasterImpl main
[java] SEVERE: Master terminated with errors.
[java] com.sun.faban.driver.FatalException: Run Aborted.
[java] at com.sun.faban.driver.engine.MasterImpl.executeRun(MasterImpl.java:677)
[java] at com.sun.faban.driver.engine.MasterImpl.runBenchmark(MasterImpl.java:276)
[java] at com.sun.faban.driver.engine.MasterImpl.main(MasterImpl.java:1566)
[java]
BUILD FAILED
/web20_benchmark/build.xml:81: Java returned: 1
Total time: 14 seconds
cat: /faban/output/*/summary.xml: No such file or directory
Docker version: 1.12.5
Did I miss something in command line?
Thank you
I have two docker hosts.
Setup:
Firstly, I instantiate a docker swarm on host 1, and join host 2 to it. After that, I create an overlay network.
Memcache Server
Then I created 6 Memcache servers but only used four for testing (configured in the Memcache client).
When I ran the Docker Network inspect command on host 2, it revealed :
`
[
{
"Name": "overnet",
"Id": "n9zmjfum9hgndf0yybeq18dyi",
"Created": "2017-12-21T13:51:48.671584108-08:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"4572e9f31d463a208e50691a10a9f793b10dd8c6ad2188fdaf51478b7760d7f6": {
"Name": "dc-server3",
"EndpointID": "b5e1653115889ceda4669f31cd8e49c0346e73f11d982889ae58b07ece638ae5",
"MacAddress": "02:42:0a:00:00:04",
"IPv4Address": "10.0.0.4/24",
"IPv6Address": ""
},
"a50d0277256099dea2fc43ed174b65538af772f07f1c3b9112d14e9da0d41377": {
"Name": "dc-server1",
"EndpointID": "9358a4ca9f37e1b55d037537792d00428a0dd994cdbe01195702c28f839e1137",
"MacAddress": "02:42:0a:00:00:02",
"IPv4Address": "10.0.0.2/24",
"IPv6Address": ""
},
"c595bd30495edd9433608f075690da57f6f6e656631383fb6e857cb7067cbc94": {
"Name": "dc-server2",
"EndpointID": "697f93b5bb3c26b9b07e70825cfb8335d0f1ac9c9637b2dc4f5dcefd91244676",
"MacAddress": "02:42:0a:00:00:03",
"IPv4Address": "10.0.0.3/24",
"IPv6Address": ""
},
"f7b24a1968695b2ff4ff91f8fd71299ccc587613047d7a248665e01f6ec41dde": {
"Name": "dc-server4",
"EndpointID": "4bf37b7296c20e3ef65400750e1c45dc3b3008aaa0aa020c941a2455e6cee280",
"MacAddress": "02:42:0a:00:00:05",
"IPv4Address": "10.0.0.5/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
`
The command Docker network inspect on host 1 reveals:
`
[
{
"Name": "overnet",
"Id": "n9zmjfum9hgndf0yybeq18dyi",
"Created": "2017-12-21T14:03:09.506768468-08:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"2d8ef847095009c86d278fbf00c1ad89e9c707e92628b63b9d2fd3e53f9bf44e": {
"Name": "dc-server5",
"EndpointID": "c211723f7fc3bc7b4cc3ba3e910e032b8f8754d7c658652f60bd3b29649d03c4",
"MacAddress": "02:42:0a:00:00:06",
"IPv4Address": "10.0.0.6/24",
"IPv6Address": ""
},
"6499147bdecc06e7fdd4aec4cf64e4f77cdc1901220727ad779d9136e427b245": {
"Name": "dc-server6",
"EndpointID": "d77e845e0b359f0d6ede6f1766d33d8272fc5ee192a63959a14bef940b686f83",
"MacAddress": "02:42:0a:00:00:07",
"IPv4Address": "10.0.0.7/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": {},
`
Memcache Client
I instantiated the client on Host 1, and configured the client to use four servers (2 on each docker host) by ammending docker_servers.txt in /usr/src/memcached/memcached_client/.
Finally, when scaling up the Twitter dataset, I encounter a connection error, despite it connecting to the containers in the other host.
`
memcache@2f8c477d044e:/usr/src/memcached/memcached_client$ ./loader -a ../twitter_dataset/twitter_dataset_unscaled -o ../twitter_dataset/twitter_dataset_30x -s docker_servers.txt -w 4 -S 30 -D 4096 -j -T 1
stats_time = 1
Configuration:
nProcessors on system: 64
nWorkers: 4
runtime: -1
Get fraction: 0.900000
Naggle's algorithm: False
host: dc-server3
address: 10.0.0.4
host: dc-server4
address: 10.0.0.5
host: dc-server5
address: 10.0.0.6
host: dc-server6
address: 10.0.0.7
Loading key value file...Average Size = 1057.34758
Keys to Preload = 3557357
created uniform distribution 1000
rps -1 cpus 4
Overridge n_connections_total because < n_workers
num_worker_connections 1
Connection error
`
Lastly, I would also like to ask if "ping" can be installed on to the docker?
Thanks!
Is there a way to deploy CloudSuite in a distributed fashion?
Thanks.
Hi. I've recently faced & fixed a bug in the web serving benchmark. I'm not sure if it reproduces in the other environments -- I use Ubuntu 16.04/4.4.0-96 kernel, Docker version 18.02.0-ce, build fc4de44, two servers (loadgen and server) connected with docker swarm + overlay network.
Bug description
When the load generator tries logging in to the Elgg webserver, it finds a string "You have been logged in" from the http response to decide whether the login was successful (public void doLogin() / Web20Driver.java). The current logic fetches the URL only once (with postRequest) and the fetched page does not contain the substring even if the login was successful (I assume). This causes the benchmark to fail.
Solution
In doLogin(), fetch the URL "your_web_server_address/activity" after sending the login request -- it should have the string "You have been logged in" in the response.
Sorry for the incorrect report -- there was a bug in my local repo. Closing the issue.
I want to analyze the memory performance (Bandwidth, memory usage..) of the CloudSuite3.0
Especially I run the 'data-caching'
Experiment environment is written below
- 2 Socket, 32 cores (16cores/socket)
- Memory: 2133MHz, 32GB * 16
- Network throughput: 1gb/s
- I don't use the 'docker'
- 'data-caching': memcached-1.4.24 for memcached server
twitter_dataset and memcached_client offered by CloudSuite for clients
In case of 'data-caching':
In this experiment, I expected that the 'data-caching' uses memory bandwidth pretty much, but it doesn't and it is bounded by the CPU. I'm confused that the below configuration is correct.
Server network is too slow to test the 'data-caching', so I run memcached server and client in the physically one server, and make them use different socket respectively. (16cores/socket)
- 4 server processes with 16 threads for each process
- 40GB server main memory size
- 100 client with 16 client threads for each server process (total 100*4 = 400clients)
- 10GB twitter data set (Scaling factor of 30)
In this case, average memory bandwidth utilization is about 6GB/s and 10GB/s for server and client,
average memory usage is 5GB and 2GB for each server process and client process,
(We measured resident set size as a memory usage)
and the CPU utilization is 1560% and 1200% for server processes and client processes.
I want to know that above configuration is reasonable and is there any other configuration to burden memory.
--link is legacy
please, refer to https://docs.docker.com/engine/userguide/networking/dockernetworks/
regards,
Dmitrii
Cloudsuite supports which NoSQL system? Please tell me the names of those systems.
I was trying to run cloudsuite using the default host network using the following commands:
docker run -dt --net=host --name=web_server cloudsuite/web-serving:web_server /etc/bootstrap.sh 127.0.0.1 127.0.0.1 7
docker run -dt --net=host --name=memcache_server cloudsuite/web-serving:memcached_server
docker run -dt --net=host --name=mysql_server cloudsuite/web-serving:db_server 10.7.189.146
docker run --net=host --name=faban_client cloudsuite/web-serving:faban_client 10.7.189.146 7
When I run the faban_client
using the above command, I don't see anything happening the screen is just blank.
On the other hand when I created a separate user defined network it worked! I wanted to know if I am missing anything in the commands. Any help would be appreciated.
I get the following error messages through dmesg:
[693700.974440] traps: php5-fpm[13036] trap stack segment ip:79d0f4 sp:7fff831b3330 error:0 in php5-fpm[400000+800000]
[693702.076074] traps: php5-fpm[13200] trap stack segment ip:79d0f4 sp:7fff831b3330 error:0 in php5-fpm[400000+800000]
[693703.180881] traps: php5-fpm[13210] trap stack segment ip:79d0f4 sp:7fff831b3330 error:0 in php5-fpm[400000+800000]
[693704.282564] traps: php5-fpm[13220] trap stack segment ip:79d0f4 sp:7fff831b3330 error:0 in php5-fpm[400000+800000]
[693705.389770] traps: php5-fpm[13230] trap stack segment ip:79d0f4 sp:7fff831b3330 error:0 in php5-fpm[400000+800000]
PS: 10.7.189.146
is the IP address of the machine I am currently running on.
Docker Version: Docker version 17.03.0-ce
, build 3a232c8
Hadoop 2.7.3 tar file is not available at the following source anymore, and it causes the built to fail on Travis-CI:
http://mirrors.gigenet.com/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
We need to point to another source.
We may also need to update our Hadoop version (add another dockerfile with a newer version of Hadoop)
Travis-CI log file:
https://travis-ci.org/parsa-epfl/cloudsuite/jobs/340623146
[Problem] The terminal shows "./entrypoint.sh: line 42: curl: command not found" when executing the client container.
$ sudo docker run -it --name client --net [network-name] cloudsuite/web-search:client [server-ip] 50 90 60 60
Starting Faban Server
Please point your browser to http://ip-x.x.x.x:9980/
Buildfile: /usr/src/faban/search/build.xml
init:
[mkdir] Created dir: /usr/src/faban/search/build/classes
compile:
[javac] /usr/src/faban/search/build.xml:35: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 1 source file to /usr/src/faban/search/build/classes
[javac]
[javac] WARNING
[javac]
[javac] The -source switch defaults to 1.8 in JDK 1.8.
[javac] If you specify -target 1.5 you now must also specify -source 1.5.
[javac] Ant will implicitly add -source 1.5 for you. Please change your build file.
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.5
[javac] warning: [options] source value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] target value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
[javac] 4 warnings
bench.jar:
[mkdir] Created dir: /usr/src/faban/search/build/lib
[jar] Building jar: /usr/src/faban/search/build/lib/search.jar
deploy.jar:
[jar] Building jar: /usr/src/faban/search/build/search.jar
deploy:
BUILD SUCCESSFUL
Total time: 6 seconds
Print= :1
/entrypoint.sh: line 42: curl: command not found
./entrypoint.sh: line 42: curl: command not found
./entrypoint.sh: line 42: curl: command not found
./entrypoint.sh: line 42: curl: command not found
./entrypoint.sh: line 42: curl: command not found
./entrypoint.sh: line 42: curl: command not found
./entrypoint.sh: line 42: curl: command not found
./entrypoint.sh: line 42: curl: command not found
[Solution] add the following in the client Dockerfile:
https://github.com/parsa-epfl/cloudsuite/blob/master/benchmarks/web-search/client/Dockerfile
apt-get install -y curl
I was hoping I might use curl with the Faban component of the web search module for testing what's available/possible through the API. The documentation available from here: http://faban.org/1.3/docs/api/index.html is a bit terse, so I was hoping that posting here might yield a better explanation as to how (as a start) I might duplicate the return of the same .json output I currently get with: docker run -it --name client --net search_network cloudsuite/web-search:client server_address 50 90 60 60 . Although this may seem like a trivial task, I have not been able to accomplish this. I'm hopeful someone may be gracious enough to assist me in minimally interacting with this api :)
Thanks!
I run in-memory-analytics without docker. Experiment environment is written below
I complete the execution of this with 'ml-latest-small' input set. And I want to run with more bigger input set 'ml-latest'. But if I run it with --driver-memory 1g --executor-memory 1g option as described on the tutorial page, I got an error message : java.lang.OutOfMemoryError: Java heap space
I checked that executor memory is set to 2GB in the SparkConf() MovieLensALS.scala.
.set("spark.executor.memory", "2g")
I want to remove that line of the code and set the 'executor-memory' size explicitly in run-time.
I'm struggling with the re-build MovieLensALS.scala..
How can I build it ?
And.. do you have any recommendation for proper executor memory size to run ml-latest input ?
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.