Shiny Proxy Can't Access Container On Host about shinyproxy

[fmichielssen]

Hi @dragoonis ,

The 'Container unresponsive' message is caused by a test which is performed right after starting the container.
A HTTP GET request is sent to the target URL (http://172.18.0.1:20001). If no 200 (OK) response is received, the warning message is logged and a retry is performed.

If the HTTP server inside the container takes some time to get started, one or two of these warnings may appear. By default, 10 tries will be attempted with 2 second intervals.

[PlasmaPower]

If you try docker exec shiny_proxy_1 curl http://172.18.0.1:20001 while the shiny container is starting up, does it work? I suspect that it will not, whereas running curl http://localhost:20001/ on the host might. This would indicate a networking issue.

To ensure that the container is starting up properly, I'd recommend using the openanalytics/shinyproxy-demo image for testing.

[dragoonis]

Hi @PlasmaPower @fmichielssen

Indeed, your shiny_proxy container can't talk to the host.

We have no specific docker networking rules set up, so the docker-compose is running on its own default network and the extra containers shiny_proxy has spawned are on the host.

Note: our setup has worked for some time and continues to work until few days ago. Here is our YML file:

version: '2.2'

services:

    shiny_proxy:
        image: coreoasis/shiny_proxy:%RELEASE_TAG%
        ports:
        - 8080
        environment:
        # todo - shift this to a .env file
        - FLAMINGO_ENVIRONMENT=Integration
        - FLAMINGO_API_PORT=80
        - FLAMINGO_DB_IP=10.10.0.97
        - FLAMINGO_DB_PORT=1433
        - FLAMINGO_DB_NAME=Flamingo_Integration
        - FLAMINGO_DB_USERNAME=Integration
        - FLAMINGO_DB_PASSWORD=Integration
        - FLAMINGO_PORT_RANGE_START=20000
        - FLAMINGO_DOCKER_IMAGE=coreoasis\/flamingo_shiny:%RELEASE_TAG%
        - FLAMINGO_SHARE_FILEPATH_ON_HOST=%FLAMINGO_DIR%/Files
        volumes:
        - /var/run/docker.sock:/var/run/docker.sock

    flamingo_server:
        image: coreoasis/flamingo_server:%RELEASE_TAG%
        volumes:
        - %FLAMINGO_DIR%/Files:/var/www/oasis/Files
        environment:
        # todo - shift this to a .env file
        - FLAMINGO_ENVIRONMENT=Integration
        - FLAMINGO_DB_IP=10.10.0.97
        - FLAMINGO_DB_PORT=1433
        - FLAMINGO_DB_NAME=Flamingo_Integration
        - FLAMINGO_DB_USERNAME=Integration
        - FLAMINGO_DB_PASSWORD=Integration
        - OASIS_LOG_LEVEL=DEBUG
        - IS_WINDOWS_HOST=True
        volumes:
        - ../flamingo_workspace/tests:/var/www/oasis/tests

[PlasmaPower]

The simplest way to configure this is setting network_mode: host for shiny_proxy, then changing the docker url (and host if it's set) in application.yml from 172.18.0.1 to 127.0.0.1.

[dragoonis]

@PlasmaPower how is this looking now? I replaced %DOCKER_HOST_IP% with 127.0.0.1

shiny:
  proxy:
    title: Oasis Flamingo
    #logo-url: http://www.openanalytics.eu/sites/www.openanalytics.eu/themes/oa/logo.png
    #logo-url: https://www.boatoasis.com/uploads/9/2/9/3/92934216/published/boatlogo-nobg.png?1483977780
    landing-page: /
    heartbeat-rate: 10000
    heartbeat-timeout: 60000
    port: 8080
    network_mode: host
    authentication: none
    docker:
      cert-path: /home/none
      url: http://127.0.0.1:2375
      host: 127.0.0.1
      port-range-start: %FLAMINGO_PORT_RANGE_START%
  apps:
  - name: BFE_RShiny
    display-name: Start Flamingo
    docker-cmd: ["R", "-e", "flamingo::runFlamingo()"]
    docker-volumes: [ "%FLAMINGO_SHARE_FILEPATH_ON_HOST%:/var/www/oasis/Files" ]
    docker-image: %FLAMINGO_DOCKER_IMAGE%
    docker-env-file: flamingo.env

logging:
  file:
    shinyproxy.log

I have modified the docker section and changed %DOCKER_HOST_IP% to 127.0.0.1

shiny:
  proxy:
    title: Oasis Flamingo
    #logo-url: http://www.openanalytics.eu/sites/www.openanalytics.eu/themes/oa/logo.png
    #logo-url: https://www.boatoasis.com/uploads/9/2/9/3/92934216/published/boatlogo-nobg.png?1483977780
    landing-page: /
    heartbeat-rate: 10000
    heartbeat-timeout: 60000
    port: 8080
    network_mode: host
    authentication: none
    docker:
      cert-path: /home/none
      url: http://127.0.0.1:2375
      host: 127.0.0.1
      port-range-start: %FLAMINGO_PORT_RANGE_START%
  apps:
  - name: BFE_RShiny
    display-name: Start Flamingo
    docker-cmd: ["R", "-e", "flamingo::runFlamingo()"]
    docker-volumes: [ "%FLAMINGO_SHARE_FILEPATH_ON_HOST%:/var/www/oasis/Files" ]
    docker-image: %FLAMINGO_DOCKER_IMAGE%
    docker-env-file: flamingo.env

logging:
  file:
    shinyproxy.log

[PlasmaPower]

Oh, I meant setting network_mode: host in the docker-compose file for shinyproxy, not in the shinyproxy application.yml. Other than that your changes look good.

[dragoonis]

@PlasmaPower the issue here with this solution, is that it doesn't work from a ports perspective.

Before switching to network_mode: host we were doing ports: - 8080 in the compose YML, and it would allocate a random port for us, and forward to 8080.

With host networking it attempts to bind on port 8080 every time due to shinyproxy/application.yml having port: 8080 setting on it, and it's getting a conflict after the first run on that host.

We have multiple shiny proxy instances running on this machine, so I'm wondering, given how you've designed shiny proxy, how it is intended to work with random port allocation and your host networking recommendation (to allow tcp://127.0.0.1 access)

Thanks!

[PlasmaPower]

In that case, I'd use my "in docker" feature: #33

I'm not sure if it made it into the latest version of shinyproxy, so you might have to build from git.

[dragoonis]

@PlasmaPower we have shinyproxy-0.9.3.jar in our git repository. What version should be targetted to achieve internal-networking ?

[PlasmaPower]

The latest Shinyproxy release doesn't have that, so you'll have to build from Git. Luckily it's not that hard. Clone or download the repository, run "mvn install" in it, and it'll build a jar in a new "target" folder.

[dragoonis]

Hey @PlasmaPower I have made the changes and built latest master to give me v1.0.2

Should I be using master or a different branch to get the changes you're referring to ?

Upon booting it up I'm getting this message - can you help ?

https://gist.githubusercontent.com/dragoonis/1a2e5250e9732207f6ce301469ec0c4f/raw/ed446a2461bd403551aaa5c831125ce13a1b718e/gistfile1.txt

[PlasmaPower]

Oh, looks like you'll have to build from the develop branch instead of the master branch. The latter is now only for released versions.

[dragoonis]

Rebuilt from develop. Generated flamingo.1.0.3-SNAPSHOT.jar

Error:
2018-01-24 14:44:41.165 INFO 17 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec : I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: Permission denied

shinyproxy config:
https://gist.github.com/dragoonis/21caf09c43408db8a17434882727de01

I'm wondering if the DOCKER_HOST option is something we should take out, now that I'm switching to using docker sockets (instead of TCP)

Note:

• This isn't on the host, this is in the container properly
• I'm mounting the docker socket into shiny_proxy compose yml

[PlasmaPower]

Since you're mounting the docker socket, yes, remove the DOCKER_HOST.

[dragoonis]

Removed DOCKER_HOST.

Shiny Proxy UI now loads - but the HTML output is this

Error
Status code: 500

Message: Failed to create container

Stack Trace: 

My docker config is this:

# The max number of open files for the daemon itself, and all
# running containers.  The default value of 1048576 mirrors the value
# used by the systemd service unit.
DAEMON_MAXFILES=1048576

# Additional startup options for the Docker daemon, for example:
# OPTIONS="--ip-forward=true --iptables=true"
# By default we limit the number of open files per container
OPTIONS="  -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 --default-ulimit nofile=1024:4096 "

# How many seconds the sysvinit script waits for the pidfile to appear
# when starting the daemon.
DAEMON_PIDFILE_TIMEOUT=10

[PlasmaPower]

What are the shinyproxy logs? Is the docker socket available in the shinyproxy container?

[dragoonis]

@PlasmaPower

2018-02-07 10:56:04.687  INFO 17 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec   : I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: Permission denied
2018-02-07 10:56:04.688  INFO 17 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec   : Retrying request to {}->unix://localhost:80
2018-02-07 10:56:04.688  INFO 17 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec   : I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: Permission denied
2018-02-07 10:56:04.689  INFO 17 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec   : Retrying request to {}->unix://localhost:80
2018-02-07 10:56:04.689  INFO 17 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec   : I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: Permission denied

The file perms on the host for the .sock file are:

[ec2-user@ip-10-10-0-15 oasis_build]$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Jan 23 12:17 /var/run/docker.sock

In the container we're running as jenkins user, so it isn't allowed to touch a root:docker owned file.

[PlasmaPower]

The container should be running as root. After all, anything that can start up docker containers is root equivalent.

[dragoonis]

Got it working.

…

On 7 Feb 2018 2:14 pm, "Lee Bousfield" ***@***.***> wrote: The container should be running as root. After all, anything that can start up docker containers is root equivalent. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#47 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAI7kTRqMp5rWk-92SXI7VlAD6G2PSRDks5tSa_cgaJpZM4RWS5w> .