Comments (5)
Hi Esha,
You can use Swarm to build an overlay network over which Docker containers running on different machines can communicate. Please take a look at these instructions and let me know if you face any problems: http://cloudsuite.ch//2015/12/20/swarm/
Cheers,
Alex
from cloudsuite.
Hi Alex,
Thanks for replying.
I am doing bandwidth experiments for client-server interaction as part of my research in cloud. So for this I need the client to be outside the cloud setup. I have my media streaming server running in one of the VMs in OpenStack and the client is installed in another machine outside the OpenStack servers. I want to create a client-server scenario which can represent real world media streaming service where multiple clients from different machines can stream media services from the media streaming server. Do you think this is doable without using swarm as I do not prefer the client-server to be running in the same overlay network?
Many thanks.
Esha.
from cloudsuite.
Swarm creates an overlay network to create a "virtual" cluster and hence facilitate managing all the participating nodes. While I haven't specifically tried your deployment scenario, I don't think its use is limited within a single physical cluster. Swarm is not functionally required, but it's useful once the number of nodes you use starts growing.
Using directly the IPs should work, but you should make sure the client and server machines can reach each other and the used ports are open (in this case, port 80). As a first step, can you try repeating the process you posted above using a client and a server machine within the same OpenStack cluster?
from cloudsuite.
Hi Alex,
As per your advice I have created a swarm and also an overlay network. Swarm nodes are working fine and I can create any docker service using the official images from docker hub, like redis, nginx, etc, it just works without any issues. But if I try to use the images from your cloudsuite or my private repo, it fails to start.
Swarm issues with images from private registry:
In my case I am building an image on the nodes in the cluster and also I have tried to use the images from my docker hub registry. I can pull the images from my hub registry in each of the swarm nodes but it doesn't work with docker service create command. Thus, it seems that I can only use images from official docker hub and not if I build the image and push/pull from my docker hub registry. I have tried to debug this issue and attempted different possible solutions but nothing worked for me.
These are my start commands:
Note: The commands given in http://cloudsuite.ch//2015/12/20/swarm/ for running a service doesn't work in my swarm mode as those are for older version of swarm. Hence I have used 'docker service create' commands found in https://docs.docker.com/engine/reference/commandline/service_create/
ubuntu@swarmmasternew:/$ sudo docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username : mydockerID
Password:
Login Succeeded
ubuntu@swarmmasternew:/$ sudo docker service create --name streaming_dataset --network swarm-network --constraint "node.id==ffx5v6f1ddnxtb1g8434q27dn" --with-registry-auth mydockerID/repository:dataset
l3emhilu51c00ivijkk4hag3z
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
But with service ls I see that the service fails to start:
ubuntu@swarmmasternew:/$ sudo docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
81ddszzjwjl9 streaming_dataset replicated 0/1 mydockerID/repository-dataset
ubuntu@swarmmasternew:/$ sudo docker service ps streaming_dataset
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ohxmvg2vel6e streaming_dataset.1 mydockerID/repository:dataset swarm1 Running Pending 29 seconds ago
And this keeps showing pending even after 3-4 hours.
CloudSuite related issue in swarm mode:
Then I tried to run dc-server from cloudsuite and this time I didn't pull the image from my registry. These are the commands that I used:
ubuntu@swarmmasternew:/$ sudo docker service create --name dc-server --network swarm-network --constraint "node.id==ffx5v6f1ddnxtb1g8434q27dn" -d cloudsuite/data-caching:server -t 4 -m 4096 -n 550
satyv06nlrxh9bwzv1kwo7opk
ubuntu@swarmmasternew:/$ sudo docker service ps dc-server
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
i2iq7yd0feq2 dc-server.1 cloudsuite/data-caching:server swarm1 Running Preparing 2 seconds ago
After an hour or so when I executed the service ps command I found that now its running.
ubuntu@swarmmasternew:/$ sudo docker service ps dc-server
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
i2iq7yd0feq2 dc-server.1 cloudsuite/data-caching:server swarm1 Running Running about an hour ago
However, now the problem is shown in dc-client which can't connect to the dc-server and also if I run the dc-client directly on the swarm node it fails to attach to the swarm overlay network as we have to deploy it via the swarm manager.
ubuntu@swarm2:~$ sudo docker run -it --net swarm-network --name dc-client cloudsuite/data-caching:client bash
docker: Error response from daemon: Could not attach to network swarm-network: rpc error: code = 7 desc = network swarm-network not manually attachable.
Then I tried to run it by removing the --net flag and although it was created it couldn't connect to the server.
Below are the steps which I followed:
sudo docker run -it --name dc-client cloudsuite/data-caching:client bash
- '[' bash = -rps ']'
- exec bash
memcache@2036dbdf7c8a:/$ cd /usr/src/memcached/memcached_client/
memcache@2036dbdf7c8a:/usr/src/memcached/memcached_client$ vi docker_servers.txt
In the docker_servers.txt file I have entered the IP and port of the swarm node in which the dc-server is running. Then I tried the following which failed with "Connection error" message:
memcache@2036dbdf7c8a:/usr/src/memcached/memcached_client$ ./loader -a ../twitter_dataset/twitter_dataset_unscaled -o ../twitter_dataset/twitter_dataset_30x -s docker_servers.txt -w 4 -S 30 -D 4096 -j -T 1
stats_time = 1
Configuration:
nProcessors on system: 4
nWorkers: 4
runtime: -1
Get fraction: 0.900000
Naggle's algorithm: False
host: 10.0.0.103
address: 10.0.0.103
Loading key value file...Average Size = 1057.34758
Keys to Preload = 3557357
created uniform distribution 1000
rps -1 cpus 4
Overridge n_connections_total because < n_workers
num_worker_connections 1
Connection error
Next, I have tried to create the dc-client from the manager node; it starts but shut-downs quickly - as it obviously cannot get into bash from the manager to some other node as expected. Below are the commands that I have tried for this:
ubuntu@swarmmasternew:/$ sudo docker service create --name dc-client --network swarm-network --constraint "node.id==rcx96urvzsyrvfy9cwttn152n" cloudsuite/data-caching:client bash
dllovu3wu05lm9tc86mi2nu76
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
ubuntu@swarmmasternew:/$ sudo docker service ps dc-client
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
6vc7h24yko26 dc-client.1 cloudsuite/data-caching:client swarm2 Ready Ready 2 seconds ago
h1nc1jzbm0mu _ dc-client.1 cloudsuite/data-caching:client swarm2 Shutdown Complete 2 seconds ago
42l1ekylfiyg _ dc-client.1 cloudsuite/data-caching:client swarm2 Shutdown Complete 8 seconds ago
ubuntu@swarmmasternew:/$ sudo docker service ps dc-client
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
qh25ulxcsjjt dc-client.1 cloudsuite/data-caching:client swarm2 Running Starting 1 second ago
h6833dl7x93h _ dc-client.1 cloudsuite/data-caching:client swarm2 Shutdown Complete 6 seconds ago
km3v3pupcnp2 _ dc-client.1 cloudsuite/data-caching:client swarm2 Shutdown Complete 12 seconds ago
n19r9a1jn0ov _ dc-client.1 cloudsuite/data-caching:client swarm2 Shutdown Complete 18 seconds ago
shozekxvhaf2 _ dc-client.1 cloudsuite/data-caching:client swarm2 Shutdown Complete 25 seconds ago
I would really appreciate if you could help me with the above issues.
Many thanks,
Esha
from cloudsuite.
Please reopen if this problem still remains.
from cloudsuite.
Related Issues (20)
- HHVM and Django left overs
- Better documentation how to run the workloads
- Add to the configuration parameters of the docker, the number of vcpus that will be allocated to Spark and Hadoop
- Media Streaming Should Have High-Resolution Video (1080p, 4K)
- In-Memory doc
- Data Analytics implementation questions HOT 10
- Media Streaming Multi-node HOT 4
- Ramp up workload intensity for Data-Caching and Data-Serving Benchmarks HOT 1
- media streaming dataset: Issue with running ffmpeg - timestamp out of range HOT 1
- Can I run cloudsuite benchmark on arm64 architecture directly by offical code? HOT 13
- Sysbench Meets Error When Running on ARM64 Platform
- In-memory analytics benchmark run question HOT 8
- media-streaming dataset error HOT 7
- Investigate CloudSuite web serving workload in QEMU
- Instructions for compiling for RISC-V ISA (Data-caching) HOT 3
- Starting the web-server for the the web-serving benchmark fails as rootless
- data-analytics: Native library missing for arm64 HOT 1
- data-analytics: failure to run benchmark on different hosts without proper DNS resolution of nodes
- data-caching: cache not warmed up?
- Data caching DNS Error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudsuite.