Comments (7)
"JIRA applications were shut down incorrectly, or failed to shut down" is one of the reasons for the lock file getting created. Pods may start and stop during rolling patching of servers and this should be able to handle this activity on its own without preventing itself from starting.
from data-center-helm-charts.
@bordenit thanks for the suggestion. Can you perhaps share the script and elaborate on how exactly Jira pods occasionally refused to start (stacktraces)? As far as I understand you had corrupt caches from time to time, and it'd be great to have an automated way to flush them? Please, correct me if I am wrong :)
from data-center-helm-charts.
I can include a stack trace the next time it happens, but it's basically this error:
https://community.atlassian.com/t5/Jira-questions/jira-startup-failed/qaq-p/1215226
So, I delete the .jira-home.lock file and flush the plugin caches in .bundled-plugins and .osgi-plug-ins in both the shared-home and home directories (as best can be deleted when these files are in use by the pods). Typically, the app comes back up after those steps. However, since the app is mounted to the pvc and the app is using the files you can't fully delete the plugin caches when you exec into the pod. So, I think that's a second reason for an init container so that these caches can be flushed fully in the pvcs before jira pods start. In non-kubernetes environment, I think the remediation is to fully shut down jira, then delete those caches. So, kubernetes deployment should ideally be able to do something similar. Hopefully, proactively rather than reactively.
from data-center-helm-charts.
thanks @bordenit indeed, a valid issue. I am not sure an init container is the best solution though - this init pod will flush cashes every time the container restarts, until the pod spec is updated to remove the init container. What you can do now is to define your init container in values.yaml and we'll pick it up and add to pod spec.
from data-center-helm-charts.
I believe the issue is a resource issue where Kubernetes itself does not load balance properly. If the pods or plugins fail to start, we force it to start on a different node to resolve the issue. The plug-in startup is very resource intensive and kicks the cpu on servers well over 100%. Hopefully, plug-in startup can be made less resource intensive, but you can close this for now, as we have a work around (even if not a very good one). Thanks.
from data-center-helm-charts.
Kubernetes upgraded to v1.19.9 and same issues. 100% failure rate in first startup attempt. Flush cache, remove lock file, delete pod, wait, repeat is the workaround. We disabled McAfee on access scanner, increased JVM memory, catalina_opts startup timeout. Ticket is opened with Atlassian support, but itβs looking like moving to a VM or Ec2 instance might be the best idea. Tracking one more lead for plugin status in database. We have 4 environments that all fail hard at the plug-in startup requiring manual effort.
from data-center-helm-charts.
This is the fix for this. https://confluence.atlassian.com/jirakb/jira-startup-fails-with-message-that-required-plugins-are-not-started-254738702.html. I did that an restarted pods 10 times and didn't experience the issues again. This can be closed.
from data-center-helm-charts.
Related Issues (20)
- [Suggestion] - Make securityContext in fetch-jmx-exporter configurable HOT 3
- [Suggestion] - Add resource requests and limits for init-containers HOT 1
- [Breaking - Changes] - podAnnotations don't support Datadog labels anymore HOT 5
- [Suggestion] - Handling database and/or storage recovery HOT 1
- [Suggestion] - Flag to remove setup ingress in Confluence helm chart HOT 3
- [Suggestion] - monitoring.jmxExporterCustomConfig does not work if exposeJmxMetrics is set to true HOT 1
- [Suggestion] - Security Vulnerabilities on v8.5.2 and Helm Chart Doesn't Support v8.5.3 HOT 3
- [Suggestion] - Support Kubernetes StatefulSetStartOrdinal feature gate .spec.ordinals.start HOT 5
- [Suggestion] - Upgrade to Fluentbit HOT 2
- [Suggestion] - BlockList additions to Confluence JMX Exporter config HOT 8
- [Suggestion] - Don't assign pod annotations to test pods HOT 4
- [Suggestion] - internalProxies line should be removed when there is no proxyInternalIps value in tomcatConfig HOT 9
- [Suggestion] - Make examples in the Bitbucket DC Chart more clear HOT 3
- [Suggestion] - Remove confluence shared home from synchrony volumes when creating a dedicated synchrony volume HOT 1
- [Suggestion] - Use different labels for jmx-exporter service HOT 15
- Installation problem after license HOT 3
- [Suggestion] - Use image from values.yaml for all tests containers images HOT 4
- [Suggestion] - Shared home should not be removed from synchrony [revert request for #745] HOT 2
- [Suggestion] - Add chmod to import-cert init-container script when running as non-root HOT 2
- [Suggestion] Enable HTTPS connection to the Bamboo server HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from data-center-helm-charts.