rancher / git-mirror

Mirror git repositories

git-mirror

Mirror git repositories.

Architecture

flowchart LR
  rancher

  git-clone  -. create .-> persistent-volume
  git-porter -. read   .-> persistent-volume
  git-scrub  -. delete .-> persistent-volume
  git-mirror -. update .-> persistent-volume

  rancher -. read .-> git-porter

  subgraph chart
    git-scrub
    git-clone
    git-porter
    git-mirror
    persistent-volume
  end

git-mirror's Issues

Replace git-sync with CronJob

After finally getting git-sync to play nicely with git-porter, I realized that it is not actually setup for use on the server-side... It's basically just doing a shallow clone, but we need a --bare clone and then synchronization on top of that.

Reference(s):

• https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server
• https://github.com/influxdata/helm-charts/blob/master/charts/influxdb/templates/backup-retention-cronjob.yaml
• kubernetes/git-sync#184

Add debug parameter for git-sync deployment to chart values

We'll need to add this to the deployment template:

spec:
  containers:
    - args:
      - -v 9
      command:
      - /git-sync

Reference(s):

• https://github.com/kubernetes/git-sync#primary-flags

git-sync pod in crashloop backoff for partner-charts repository

INFO: detected pid 1, running init handler
I0616 18:01:40.203668      12 main.go:473] "level"=0 "msg"="starting up" "pid"=12 "args"=["/git-sync"]
I0616 18:01:40.203713      12 main.go:923] "level"=0 "msg"="cloning repo" "origin"="https://github.com/rancher/partner-charts" "path"="/tmp/git"
E0616 18:01:40.576644      12 main.go:525] "msg"="too many failures, aborting" "error"="Run(git clone -v --no-checkout -b master https://github.com/rancher/partner-charts /tmp/git): exit status 128: { stdout: "", stderr: "Cloning into '/tmp/git'...\nPOST git-upload-pack (353 bytes)\nfatal: Remote branch master not found in upstream origin\n" }" "failCount"=0

Fix git-clone init container when scaling deployments up and down

Scaling down the deployment, and then scaling back up will result in this error:

fatal: destination path 'community-catalog.git' already exists and is not an empty directory. 

We need to add additional logic to the git-clone init container to check if the repository has already been cloned.

Originally posted by @oxr463 in #23 (comment)

Custom ingress annotations for chart values.yml

e.g., for ALB's in AWS

Implement Horizontal Pod Autoscaler

policy should adjust up or down to as request demands change could be node, pod or both

Investigate alternatives for running nginx and fcgiwrap with tini for git-porter image

There are two potential solutions to look into:

• A) Rewriting git-porter to use ingress-nginx FCGI services and annotations
• B) Decomposing git-porter into separate containers: fcgiwrap, and nginx (using volumes to mount the shared socket, and updating the nginx config to point to the fcgiwrap service via cluster-local DNS)

Reference(s):

• https://github.com/gitorious/git-http-multi-backend
• https://stackoverflow.com/q/58437697/8507637
• https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services
• https://github.com/Team-Fruit/docker-nginx-fcgiwrap/blob/master/1.17/docker-entrypoint.sh

fcgiwrap.sock connect() failed no such file or directory for git-porter container

Here is the error from the container log:

2022/06/22 19:22:27 [crit] 12#12: *4 connect() to unix:/var/run/fcgiwrap.socket failed (2: No such file or directory) while connecting to upstream, client: 10.42.0.181, server: git-mirror.ftp.sh, request: "GET /rke2-charts.git/info/refs?service=git-upload-pack HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "git-mirror.ftp.sh"

Here are the permissions:

cd /var/run
ls -la
total 20
drwxr-xr-x    1 root     root          4096 Jun 22 13:44 .
drwxr-xr-x    1 root     root          4096 Jun 22 13:44 ..
drwxrwsr-x    2 fcgiwrap www-data      4096 Jun 22 13:44 fcgiwrap
-rw-r--r--    1 root     root             3 Jun 22 13:44 nginx.pid
drwxr-xr-x    3 root     root          4096 Jun 22 13:44 secrets

cd /var/run/fcgiwrap
ls -la
total 8
drwxrwsr-x    2 fcgiwrap www-data      4096 Jun 22 13:44 .
drwxr-xr-x    1 root     root          4096 Jun 22 13:44 ..

Reference(s):

• https://unix.stackexchange.com/a/636270/247886
• https://stackoverflow.com/a/30716240/8507637

Permission denied for '/root/.config/git/attributes'

git clone https://git-mirror.ftp.sh/rke2-charts.git
Cloning into 'rke2-charts'...
remote: warning: unable to access '/root/.config/git/attributes': Permission denied
remote: Enumerating objects: 7363, done.
remote: Counting objects: 100% (7363/7363), done.
remote: Compressing objects: 100% (2679/2679), done.
remote: Total 7363 (delta 4221), reused 7363 (delta 4221), pack-reused 0
Receiving objects: 100% (7363/7363), 6.61 MiB | 2.01 MiB/s, done.
Resolving deltas: 100% (4221/4221), done.

Update NOTES template for helm chart

• We need to display all of the repository URLs, instead of just the one... which is currently broken.
• We need to update the URL for the source repository

Release "git-mirror" has been upgraded. Happy Helming!
NAME: git-mirror
LAST DEPLOYED: Thu Jun 16 17:59:25 2022
NAMESPACE: git-mirror-chart-testing
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  https://map[host:git.rancher.io]

Source can be found here: https://github.com/rancher/git-mirror-docker

git-sync pod in crashloop backoff for rke2-charts repository

INFO: detected pid 1, running init handler
I0616 18:01:41.176330      13 main.go:473] "level"=0 "msg"="starting up" "pid"=13 "args"=["/git-sync"]
I0616 18:01:41.176372      13 main.go:923] "level"=0 "msg"="cloning repo" "origin"="https://github.com/rancher/rke2-charts" "path"="/tmp/git"
E0616 18:01:41.559356      13 main.go:525] "msg"="too many failures, aborting" "error"="Run(git clone -v --no-checkout -b master https://github.com/rancher/rke2-charts /tmp/git): exit status 128: { stdout: "", stderr: "Cloning into '/tmp/git'...\nPOST git-upload-pack (353 bytes)\nfatal: Remote branch master not found in upstream origin\n" }" "failCount"=0

info/refs no such file or directory

2022/06/28 18:28:40 [error] 12#12: *1 open() "/var/git/rke2-charts/info/refs" failed (2: No such file or directory), client: 10.42.0.181, server: git-mirror.ftp.sh, request: "GET /rke2-charts/info/refs?service=git-upload-pack HTTP/1.1", host: "git-mirror.ftp.sh" 

Originally posted by @oxr463 in #24 (comment)