Git checkout takes too long in VM about cortx-re

[kpricepc]

I was able to get a hold of GitHub support on the phone and they had a similar report from another Colorado customer and they brought the IP address of their load balancer we were also having issue with out of the loop. The issue seems to be resolved from our end and I can do quick clones now. Looks like the Jenkins jobs are also able to checkout now.

This issue should be resolved so let the Lab team know otherwise.

[shailesh-vaidya]

@mukul-seagate11 We face this issue almost daily in Jenkins build process. Can we raise ticket with GitHub for this.

[mukul-seagate11]

@mukul-seagate11 We face this issue almost daily in Jenkins build process. Can we raise ticket with GitHub for this.

Sure, we can take this forward with GitHub call

[stale]

This issue/pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shailesh-vaidya]

@mukul-seagate11 As updated by Dhruv in today's call, Can you please raise support ticket for this with GitHub team.

[mukul-seagate11]

@mukul-seagate11 As updated by Dhruv in today's call, Can you please raise support ticket for this with GitHub team.

OK sure

[mukul-seagate11]

[gowthamchinna]

Absorbing random failures in pipeline

http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/NFS/53/

http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/S3server/155/console

[gowthamchinna]

+1

http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/Hare/160/console

[gowthamchinna]

+1
http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/Motr/115//console

This issue occurred between 25 th Aug 11 PM IST to 26 th Aug 2.20 AM IST

[gowthamchinna]

Update : The checkout is working on the above mentioned time frame, But still getting random failure on checkout

[mukul-seagate11]

Casey Mokricky (GitHub Support)
Aug 25, 2020, 4:16 PM UTC

Hi Mukul,

Thank you for contacting GitHub Enterprise Support.

Just to make sure I'm understanding correctly, it sounds like your git client is having performance issues on a CloudForm VM instance. Is that correct?

If so, please prefix your git command with the GIT_TRACE_PERFORMANCE environment variable and respond with any output. For example:

GIT_TRACE_PERFORMANCE=1 git clone https://github.com/my/repo.git
Feel free to include any additional information as well.

Thank you,
Casey

[shailesh-vaidya]

@mukul-seagate11 - We are observing this issue quite frequently now. @gowthamchinna has collected debug logs as below. Git checkout is taking around 14 mins which is causing build failures.

[root@4be3b7e549a9 /]# time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-csm.git
02:32:13.129992 git.c:371               trace: built-in: git 'clone' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
Cloning into 'cortx-csm'...
02:32:13.133696 run-command.c:369       trace: run_command: 'git-remote-https' 'origin' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
* Couldn't find host github.com in the .netrc file; using defaults
* About to connect() to github.com port 443 (#0)
*   Trying 140.82.114.4...
* Connected to github.com (140.82.114.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US
*       start date: May 05 00:00:00 2020 GMT
*       expire date: May 10 12:00:00 2022 GMT
*       common name: github.com
*       issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-GitHub-Request-Id: 2A98:2BC3:3987949:54AED2F:5F39EC2D
< X-Frame-Options: DENY
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:39879CA:54AED5C:5F39EC2D
< X-Frame-Options: DENY
<
* Connection #0 to host github.com left intact
02:32:15.583453 run-command.c:369       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.584471 exec_cmd.c:118          trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.586117 git.c:371               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6214

* upload completely sent off: 6214 out of 6214 bytes
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:3987A2E:54AEE1E:5F39EC2E
< X-Frame-Options: DENY
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
02:32:16.435083 run-command.c:369       trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.436121 exec_cmd.c:118          trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.437700 git.c:371               trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
* Connection #0 to host github.com left intactiB | 74.00 KiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722
Receiving objects: 100% (56013/56013), 41.69 MiB | 68.00 KiB/s, done.
Resolving deltas: 100% (25949/25949), done.
02:46:30.154135 run-command.c:369       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.154769 exec_cmd.c:118          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.156481 git.c:371               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'

real    14m17.110s
user    0m3.559s
sys     0m1.760s

[shailesh-vaidya]

@mukul-seagate11 @gowthamchinna - Currently all Jenkins jobs are failing at git checkout stage. I suspect this SSC VM specific issue. I have raised https://jts.seagate.com/browse/LRL-391 for the same.
CC: @amolkongre @priyankdal

[kpricepc]

This appears to be an issue site wide at LCO and not just lab specific. We are starting a Sev1 with IT Networking at Seagate.

[kpricepc]

The issue seems to be with Github load balancers. If we use a different load balancer, we see expected results.

[509589@ssc-vm-0001 ~]$ git clone [email protected]:Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
The authenticity of host 'lb-192-30-255-113-sea.github.com (192.30.255.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lb-192-30-255-113-sea.github.com,192.30.255.113' (RSA) to the list of known hosts.
remote: Enumerating objects: 372, done.
remote: Counting objects: 100% (372/372), done.
remote: Compressing objects: 100% (285/285), done.
remote: Total 57547 (delta 160), reused 146 (delta 77), pack-reused 57175
Receiving objects: 100% (57547/57547), 42.16 MiB | 20.98 MiB/s, done.
Resolving deltas: 100% (27082/27082), done.
[509589@ssc-vm-0001 ~]$ git clone [email protected]:Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
remote: Enumerating objects: 372, done.
remote: Counting objects: 100% (372/372), done.
remote: Compressing objects: 100% (285/285), done.
Receiving objects:  17% (9783/57547), 3.25 MiB | 38.00 KiB/s

[kpricepc]

I've submitted a ticket here with GitHub support...

https://enterprise.githubsupport.com/hc/en-us/requests/814338

I've also tried to call them and haven't been able to reach anyone.
I am still working with IT as they could possibly route traffic a different direction if needed. However this is more risky as it affects all of the LCO site.

[mukul-seagate11]

Issue resolved

[gowthamchinna]

@mukul-seagate11 Issue happening again

http://eos-jenkins.colo.seagate.com/job/Release_Engineering/job/github-work/job/GitHub%20Multibranch%20Jobs/job/motr-multibranch/job/PR-253/2/console

[stale]

This issue/pull request has been automatically marked as stale because it has not had recent activity. Tagging @shailesh-vaidya

[shailesh-vaidya]

@gowthamchinna I feel we are not observing this issue again. Could you please confirm and if yes mark this one as closed.

[stale]

This issue/pull request has been automatically marked as stale because it has not had recent activity. Tagging @shailesh-vaidya

[gowthamchinna]

No issues found for last 2 weeks closing this.