Openshift instance issue about multi-juicer

[sashibhusan-u]

in values.yaml I see this - juiceShop: # juiceShop.maxInstances -- Specifies how many JuiceShop instances MultiJuicer should start at max. Set to -1 to remove the max Juice Shop instance cap maxInstances: -1 # Juice Shop Image to use image: bkimminich/juice-shop tag: v11.1.0

…

On Wed, 2 Sep 2020 at 12:04, Jannik Hollenbach ***@***.***> wrote: Mh might be another OpenShift permission issue with JuiceShop, thats kinda weird I dont think this has changed recently... What Juice Shop version are you using? @sashibhusan-u <https://github.com/sashibhusan-u> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADHJMYNDWJNVM7G7ZXDCQJDSDYKDFANCNFSM4M2L2M4A> .

-- Best, Sashi Upadhyaya [phone]

[sashibhusan-u]

For me also it works now, I am able to create and run 30 instances concurrently.

…

On Thu, 3 Sep 2020 at 09:48, Jannik Hollenbach ***@***.***> wrote: Mh doesn't really look like a race condition to me. I've tried this a couple of times and v11.1.0 is always failing, while v11.1.3 always works... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADHJMYL4PW6FP6SOHQMFXULSD5C6BANCNFSM4M2L2M4A> .

-- Best, Sashi Upadhyaya [phone]

[J12934]

Hi 👋
Can you take a look at the failing request in the browsers network tab and also take a look at the logs of the juice-balancer pod, which is likely the causes of the error. Would also help if you can post both the OpenShift and the underlying kubernetes version.

https://iteratec.github.io/multi-juicer/ is actually working correctly, it's just not a real website, but only a helm repository from which helm downloads the charts. See https://iteratec.github.io/multi-juicer/index.yaml for the repository definition 😀

[sashibhusan-u]

Hi,
Thanks a lot for your prompt response, I need your further cooperation in this regard.
After your response, the helm repo works now, till yesterday it was saying 'repo not valid'.

Following are the details you requested for:

Openshift version -
OpenShift Master: v3.11.188
Kubernetes Master: v1.11.0+d4cacc0
OpenShift Web Console: v3.11.135

Pod log -

Browser log -

[J12934]

Ah ok, i've actaully seen this before. I think this is happening because of the pretty old kubernetes version not handling ownerReferences like the newer versions. We added the owner references so that the Juice Shop get automatically deleted when you uninstall MultiJuicer.

I'll try to add a config value so that you can skip these references, which should make it work.

[J12934]

I've added this option.
Its not released yet, it would be great if you could confirm if this fixes your issues before it publish a release.

You can install the fix by installing MultiJuicer like the following:

# Clone the repo
git clone [email protected]:iteratec/multi-juicer.git

helm install multi-juicer ./helm/multi-juicer/ --set="balancer.skipOwnerReference=true" --set="balancer.tag=skipOwnerReference"

[sashibhusan-u]

I reinstalled the instance with the above command-line options and it seems to have worked. I am able to create teams and login to the team instance(s). Thank you so much for your support. Could you also briefly explain(or share some url) about the functionality of the progress-watchdog ?

[J12934]

The ProgressWatchdog is basically responsible for the backup and auto apply challenge progress in case of Juice Shop container restarts feature. Its backs up the continue code as an annotation on the Deployment of the Juice Shop instance.

[J12934]

I've released the option with Release v3.3.0.

Will close the issue now. But if you have more questions feel free to ask them here or the OWASP Slack 🙏

[sashibhusan-u]

When I install v3.4 without the commandline options, it doesn't work, I get the same issue. However it works when I install with the commandline options. I thought the issue was addressed with some permanent config fix, but it seems every time we need to redeploy this way. Is this understanding correct ?

[J12934]

@sashibhusan-u yeah you will have to pass this option every time when using older kubernetes versions.

MultiJuicer should work on newer kubernetes versions without the option. I'm not really sure which kubernetes version is fixing this, but 1.15 and later should definitely work.

[sashibhusan-u]

Hi, We are trying to install the multi-juicer by increasing the max instances of juice-shop to 30 (in values.yaml & config.json). But looks like the custom deployment doesn't work, it's unable to spin up juice-shop instances at all, it's timing out. Kindly help. Install command - helm install -f C:\Users\s.upadhyaya\Git\multi-juicer\helm\multi-juicer\values.yaml -multi-juicer C:\Users\s.upadhyaya\Git\multi-juicer\helm\multi-juicer\ --set="balancer.skipOwnerReference=true" --set="balancer.tag=skipOwnerReference" Pod logs - time="2020-09-02T09:23:16.733Z" level="info" msg="JuiceBalancer listening on port 3000!" time="2020-09-02T09:31:26.232Z" level="info" msg="Team kn01 doesn't have a JuiceShop deployment yet" time="2020-09-02T09:31:27.211Z" level="info" msg="Creating JuiceShop Deployment for team \"kn01"" time="2020-09-02T09:31:27.236Z" level="info" msg="Created JuiceShop Deployment for team \"kn01"" time="2020-09-02T09:31:27.321Z" level="info" msg="Awaiting readiness of JuiceShop Deployment for team \"kn01"" time="2020-09-02T09:31:58.466Z" level="info" msg="Awaiting readiness of JuiceShop Deployment for team \"kn01"" time="2020-09-02T09:32:29.541Z" level="info" msg="Awaiting readiness of JuiceShop Deployment for team \"kn01"" time="2020-09-02T09:33:00.744Z" level="info" msg="Awaiting readiness of JuiceShop Deployment for team \"kn01"" time="2020-09-02T09:33:31.808Z" level="info" msg="Awaiting readiness of JuiceShop Deployment for team \"kn01"" time="2020-09-02T09:34:02.869Z" level="info" msg="Awaiting readiness of JuiceShop Deployment for team \"kn01"" time="2020-09-02T09:34:28.396Z" level="error" msg="Waiting for deployment of team \"kn01" timed out" time="2020-09-02T09:34:59.579Z" level="error" msg="Waiting for deployment of team \"kn01" timed out"

…

On Tue, 19 May 2020 at 16:14, Jannik Hollenbach ***@***.***> wrote: @sashibhusan-u <https://github.com/sashibhusan-u> yeah you will have to pass this option every time when using older kubernetes versions. MultiJuicer should work on newer kubernetes versions without the option. I'm not really sure which kubernetes version is fixing this, but 1.15 and later should definitely work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADHJMYPQJLTZPFXG6WTOZ4LRSKH3XANCNFSM4M2L2M4A> .

-- Best, Sashi Upadhyaya [phone]

[J12934]

Hi again @sashibhusan-u 👋

Can you take a look at the Juice Shop Deployment and its logs for the team kn01?
Might be a problem with the juice shop configuration that is preventing the pod from starting up. Or something wrong with the cluster.

[sashibhusan-u]

Juice-shop instance log below, looks like some npm issue - info: All dependencies in ./package.json are satisfied (OK) (node:19) UnhandledPromiseRejectionWarning: Error: EACCES: permission denied, copyfile '/juice-shop/data/static/JuiceShopJingle.vtt' -> '/juice-shop/frontend/dist/frontend/assets/public/videos/JuiceShopJingle.vtt' at Object.copyFileSync (fs.js:1912:3) at restoreOverwrittenFilesWithOriginals (/juice-shop/lib/startup/restoreOverwrittenFilesWithOriginals.js:13:6) at Object.<anonymous> (/juice-shop/server.js:97:62) at Module._compile (internal/modules/cjs/loader.js:1200:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1220:10) at Module.load (internal/modules/cjs/loader.js:1049:32) at Function.Module._load (internal/modules/cjs/loader.js:937:14) at Module.require (internal/modules/cjs/loader.js:1089:19) at require (internal/modules/cjs/helpers.js:73:18) at /juice-shop/app.js:7:18 at processTicksAndRejections (internal/process/task_queues.js:97:5) (Use `node --trace-warnings ...` to show where the warning was created) (node:19) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode ). (rejection id: 1) (node:19) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

…

On Wed, 2 Sep 2020 at 11:47, Jannik Hollenbach ***@***.***> wrote: Hi again @sashibhusan-u <https://github.com/sashibhusan-u> 👋 Can you take a look at the Juice Shop Deployment for the team kn01? Might be a problem with the juice shop configuration that is preventing the pod from starting up. Or something wrong with the cluster. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADHJMYOK2ZZCKE4GTL3PPVTSDYIDPANCNFSM4M2L2M4A> .

-- Best, Sashi Upadhyaya [phone]

[J12934]

Mh might be another OpenShift permission issue with JuiceShop, thats kinda weird I dont think this has changed recently...
What Juice Shop version are you using? @sashibhusan-u

[J12934]

Okay this looks like a general Juice Shop on OpenShift Issue, I'm getting the same error on our OpenShift cluster 😟
Will try to look into this this evening and fix it in Juice Shop.

[J12934]

@sashibhusan-u I tried it again using bkimminich/juice-shop:v11.1.3 as the version and this seems to be working on our OpenShift version. Would recommend to update to this version.

Thought I'm not really why this version bump fixes this, maybe this was fixed by the downgrade to Node.js from 14 to 12. @bkimminich any other ideas?

I'll update the default Juice Shop version in MultiJuicer to v11.1.3 to avoid this issue for other users.

[bkimminich]

This file copy error seems to be some race condition during server launch scripts, that was reported only occasionally by Linux users in the past. Can't imagine it has something to do with OpenShift in particular.

[J12934]

Mh doesn't really look like a race condition to me.
I've tried this a couple of times and v11.1.0 is always failing, while v11.1.3 always works...

[J12934]

Nice 👍
Will close the issue then again. Feel free to reopne if something else goes wrong.