jv-conseil / ucloud

The missing repo to start GitHubbing on UCloud with Ubuntu Terminal

JV conseil 👨‍💻

DevOps • Full Stack Developer • Web App Architect

Python 🐍 Django • PostgreSQL 🐘 JavaScript • Node.js • Azure Cloud ☁️ NLP (Natural Language Processing) • ETL Developer (Extract, Transform, Load)

Greater Paris Metropolitan Region, France 🇫🇷

ucloud's Issues

Bug Report 🐞kubernetes - upstream connect error or disconnect/reset before headers. reset reason: connection failure

kubernetes - upstream connect error or disconnect/reset before headers. reset reason: connection failure 👀

I solved it. In my case the yaml file was wrong. I reviewed it and the problem now is solved. Thank you – stackoverflow.com

Describe the bug 🐛

UCloud public link does not connect with a running Django app 🐍

When visiting the page https://app-githubbing.cloud.sdu.dk we got an error message 🙅‍♂️

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Configuration ⚙️

nginx
Python 3.11.0
PostgreSQL Server 14.5
Django version 4.1.2

Expected behavior

Display a 🚀 when visiting https://app-githubbing.cloud.sdu.dk

Documentation 📚

UCloud utilizes Kubernetes for Container orchestration. This is used both for the deployment of UCloud and scheduling of user jobs. — UCloud 3rd party dependencies

Diagnosis attempts 🩺

curl request

curl "https://app-githubbing.cloud.sdu.dk" --verbose👇

Connection state changed (MAX_CONCURRENT_STREAMS == 128)!

*   Trying 130.225.164.100:443...
* Connected to app-githubbing.cloud.sdu.dk (130.225.164.100) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=DK; ST=Syddanmark; O=Syddansk Universitet; CN=*.cloud.sdu.dk
*  start date: Aug  8 00:00:00 2022 GMT
*  expire date: Aug  8 23:59:59 2023 GMT
*  subjectAltName: host "app-githubbing.cloud.sdu.dk" matched cert "*.cloud.sdu.dk"
*  issuer: C=NL; O=GEANT Vereniging; CN=GEANT OV RSA CA 4
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: app-githubbing.cloud.sdu.dk]
* h2h3 [user-agent: curl/7.85.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x558727b789b0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: app-githubbing.cloud.sdu.dk
> user-agent: curl/7.85.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)! 👈 👀 
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 404 
< server: nginx
< date: Fri, 24 Feb 2023 00:35:55 GMT
< content-length: 0
< vary: Origin
< x-envoy-upstream-service-time: 0
< referrer-policy: same-origin
< strict-transport-security: max-age=15768000; includeSubDomains
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< job-id: 8139b0e2ada0597194f6965a8b90bf64
< 
* Connection #0 to host app-githubbing.cloud.sdu.dk left intact

Next testing Ubuntu (Virtual Machine) 22.04

virtual machine are credits restricted

Bug Report 🐞 (disposable) Public Links work only once 1️⃣

• I have activated the browser Private mode to evaluate if the error still happens.
• I have turn-off any extension on my browser like adblock that may interfere with the app behaviour.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I execute it again in debug mode (DEBUF = True in settings).

Describe the bug 🐛

Status 503 Service Unavailable

Public Links run only once when connected to a Django job, then fail when launch a second time displaying the error message 👉 upstream connect error or disconnect reset before headers reset reason connection failure

To Reproduce 🔂

1. Create a Public Links.
2. Waiting for Ingress status to be Ingress is now ready.
3. Connect the newly created Public Link to a Django job.
4. Click on Submit to launch the Django job.
5. Click on Open interface does open a browser window displaying the newly created Public Link in the navigation bar.
6. Click on Stop application to terminate the running Django job.
7. Click on Run application again to launch the Django job with the same parameters.
8. Click on Submit to launch the Django job.
9. Click on Open interface does open a browser window displaying the same Public Link as first run in the navigation bar, this time displaying the error message: upstream connect error or disconnect/reset before headers. reset reason: connection termination.

Configuration ⚙️

• Python 3.11.0
• PostgreSQL Server 14.5
• Django version 4.1.2

Expected behavior 🚀

app-mission-ocean.cloud.sdu.dk running in DEBUG=False with cache and collectstatic.

Additional context 🌍

When conducting searches on "kubernetes" "ingress" "upstream connect error or disconnect/reset before headers. reset reason: connection failure" on google.com, we note a lot of chatters around that issue, notably that one 👇

For some time, our users reported seeing upstream connect errors and 503s like ”upstream connect error or disconnect/reset before headers. reset reason: connection termination“. This issue goes away if we refresh our browser page. However, it was very difficult to reproduce this error. -- Puzzling 503s and Istio

Notes 📝

• Kubernetes Documentation > Ingress Controllers.
• "kubernetes" "ingress" "upstream connect error or disconnect/reset before headers. reset reason: connection failure" search on google.com.
• #1
• UCloud ticket 🎫 https://jira.cloud.sdu.dk/servicedesk/customer/portal/43/TICKET-1847

Public Links  🔗

Public Links	Created	Worked Last	Status
app-9a7f8023b8b09392140f3ff9f12c91f2.cloud.sdu.dk	19/03/2023	19/03/2023	🚫
app-githubbing.cloud.sdu.dk (deleted and created again)	18/03/2023		🚫
app-health-check.cloud.sdu.dk	18/03/2023	18/03/2023	✅
app-health-status.cloud.sdu.dk	18/03/2023	23/03/2023	✅
app-mission-ocean.cloud.sdu.dk	19/02/2023		🚫
app-research-funding.cloud.sdu.dk	11/03/2023		🚫
app-thalassa.cloud.sdu.dk	11/03/2023	23/03/2023	✅
app-yerun.cloud.sdu.dk	17/03/2023		🚫

UCloud Web Qualification

Validate UCloud as a stable cloud service to serve Python Django v4.1.2 web app coupled with a PostgreSQL Server
Provided UCloud is up and running 👉 https://status.cloud.sdu.dk/

app-thalassa.cloud.sdu.dk running in DEBUG=False with cache activated and collectstatic.

• PostgreSQL Server v14.5 initialization.
• PostgreSQL Server v14.5 New Database and User creation.
• PostgreSQL Server v14.5 SSL set up pg_hba.conf
• Django v4.1.2 access environment variables through os.environ.get("DBHOST")
• Django v4.1.2 DEBUG=True
• Django v4.1.2 SECURE_SSL_REDIRECT = True fails 👉 should not be declared in settings.py 🚫
• Django v4.1.2 SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https") ✅
• Django v4.1.2 <> PostgreSQL Server v14.5 connection django.db.backends.postgresql
• Django v4.1.2 <> PostgreSQL Server v14.5 SSL transactions SSLMODE=require
• Django v4.1.2 makemigrations & migrate
• Django v4.1.2 create_superuser
• Django v4.1.2 CSRF_COOKIE_SECURE = True and CSRF_USE_SESSIONS = True
• Django v4.1.2 DEBUG=False
• Django v4.1.2 ALLOWED_HOSTS set to
  ['localhost', 'app-627236-0.cloud.sdu.dk', 'app-githubbing.cloud.sdu.dk']
• Django v4.1.2 collectstatic
• Django v4.1.2 serves static files through whitenoise
• Django v4.1.2’s cache framework > Filesystem caching django.core.cache.backends.filebased.FileBasedCache
• Django v4.1.2 <> PostgreSQL Server v14.5 ETL python manage.py xloader.
• Django v4.1.2 run in pseudo UCloud localhost mode (see below 👀 ) https://app-{job-id}-0.cloud.sdu.dk.
• Django v4.1.2 run with a Public link https://app-githubbing.cloud.sdu.dk.
• UCloud app run over 24 hours ⏳
• PostgreSQL Server v15.2 initialization.
• PostgreSQL Server v15.2 New Database and User creation.
• PostgreSQL Server v15.2 SSL set up pg_hba.conf
• Django v4.1.2 <> PostgreSQL Server v15.2 connection django.db.backends.postgresql
• Django v4.1.2 <> PostgreSQL Server v15.2 SSL transactions SSLMODE=require
• Django v4.1.2 <> PostgreSQL Server v15.2 ETL python manage.py xloader

Issues

• Public Link running only once then error 503 upstream connect error or disconnect/reset before headers. reset reason: connection termination 🙅 (Solved on 2023, March 24th).
• Outdated Python 🐍 versions in containerized app.

End of March, 2023 👀

• PostgreSQL server v14.5 version bump to v15.2
• Django v4.1.2 version bump to v4.1.7
• Python version bump to v3.11.2

TODO

• Python 3.11.2 - Always run latest stable release of Python 🐍 in containerized app (at least Django).
• PgBouncer - 1.18.0 Dec 12, 2022. Install, setup with PostgreSQL Server v15.2.
• Custom Domain Name (DNS) instead of Public Links.
• PenTest
• ...

Pseudo UCloud localhost mode

Pseudo UCloud localhost mode = when Django app is running on https://app-{job-id}-0.cloud.sdu.dk interface with no Public link connected.

1. Pseudo UCloud localhost execution is cookie restricted to the current browser session 👉 cannot be shared with another browser session; an error 403 is raised: same browser in Private mode will fail, another browser on the same computer will fail.
2. Public link and Pseudo UCloud localhost mode are exclusive to one another.
3. Public link resources cannot be shared among users 👉 one user has to launch a job and attach a Public link he "owns", he cannot attach a Public link created by another user.
4. As an extra-check/validation, UCloud does check your Public link availability on creation for a similar one already existing created by someone else.

Containers overview

March 2022 with u1-standard-8 machine.

Environment	set -eufo pipefail	${USER}	Python 3.11.0	apt update	Creation	Age
Terminal Debian 0.9.0	❌	✅	3.6.8 ❌		28/12/2022	75 days
Terminal Ubuntu 0.20.0	❌	✅	3.10.7 ❌	960 sec	28/12/2022	75 days
Terminal Ubuntu 0.21.0 (Linux version 5.4.228.el8 Ubuntu 22.10)	❌	✅	3.10.7 ❌	233 sec	21/03/2023	0 days
Terminal CentOS 0.8.0	❌	✅	3.6.8 ❌		14/06/2022	272 days
PostgreSQL Server v14.5 (Linux v5.4.228.el8 Debian)	✅	❌	❌	218 sec	12/10/2022	152 days
PostgreSQL Server v15.2 (Linux v5.4.228.el8 Debian 6.1.2023)	✅	❌	3.9.2 ❌	24 sec	16/03/2023	4 days
Django v4.1.2 4.1.2 (Linux v5.4.228.el8 Debian)	✅	❌	✅	962 sec	08/11/2022	125 days

Incident review

Health Page Status 👉 https://status.cloud.sdu.dk/

Year	Incidents (nb)
2023	9
2022	18

Service Level Objective (SLO) for UCloud Compute >= 85% for Monthly Uptime Percentage.

Notes 📝

• Aalborg (AAU) presentation of UCloud 🎓