flatcar / flatcar-release-mirror

Mirror script for *.release.flatcar-linux.net

Mirror script for *.release.flatcar-linux.net

This script mirrors the Flatcar release server to the current directory. It creates a subfolder per release channel and updates its contents when rerun.

The folder for stable.release.flatcar-linux.net will be called stable, and the same applies to alpha, beta, and edge.

Files are only created and never deleted. Files per release channel are downloaded sequential, but all release channels are downloaded in parallel.

It supports flags to filter limit the downloading to new releases or to exclude certain image files by pattern. A detailed log can be enabled.

Usage:
--above-version VERSION    Skip folders which have versions that are lower than VERSION (e.g., 2000)
--not-files FILTER         Skip files/folders with certain patterns by running
                           grep -v 'FILTER' on the list of './NAME' entries
                           (e.g., 'vmware\|virtualbox')
--only-files FILTER        Mirror files only with certain patterns by running
                           grep 'FILTER' on the list of './NAME' entries
                           (e.g., 'vmware\|virtualbox')
  NOTE: Only one of two options, --not-files and --only-files, should be used.
--logfile FILE             Write detailed log to FILE (can also be /dev/stderr)
--channels CHANNELS        Coma-separated list of channels to mirror (e.g. stable,beta).
                           By default all channels are mirrored.
--arch ARCH                Filter the only architecture to download (either amd64 or arm64).
--help                     Show flags

flatcar-release-mirror's Issues

Use flock to mange locks

Flock is a command to manage locks from shell script, present in util-linux (so, all distributions probably have it installed by default).

We should probably use that, instead of the manual touch we are doing that has race conditions.

Not urgent, of course

@pothos @dongsupark What do you think?

Allow filtering by arch

Similar to #5, for environments which do not need ARM images, it would be nice to be able to filter them out as well.

Consider using curl --etag flags

curl/curl@18e5cb7

Please not, that for example recently released Ubuntu 20.04 still don't have new enough version of curl to benefit from it, so this should perhaps be optional.

Images are updated on every script run

It seems there is some logic to skip updates when image did not change, but it doesn't seem to work for me. Logs from download_file:

+ local url=https://stable.release.flatcar-linux.net/./amd64-usr/./2345.3.0/./flatcar_production_image.bin.bz2
+ local file=./flatcar_production_image.bin.bz2
+ local log=
+ [[ -n flatcar_production_image ]]
++ echo ./flatcar_production_image.bin.bz2
++ grep flatcar_production_image
+ [[ -z ./flatcar_production_image.bin.bz2 ]]
+ [[ -f ./flatcar_production_image.bin.bz2 ]]
++ caddy_etag ./flatcar_production_image.bin.bz2
++ local file=./flatcar_production_image.bin.bz2
+++ stat -c %Y ./flatcar_production_image.bin.bz2
++ local modtime=1589454620
+++ stat -c %s ./flatcar_production_image.bin.bz2
++ local size=20574208
+++ base36enc 1589454620
+++ local input=1589454620
+++ base36=($(echo {0..9} {a..z}))
++++ echo 0 1 2 3 4 5 6 7 8 9 a b c d e f g h i j k l m n o p q r s t u v w x y z
++++ bc
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n q
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n a
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n b
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n j
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n 1
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n 8
+++ echo
++ local modtime36=qabj18
+++ base36enc 20574208
+++ local input=20574208
+++ base36=($(echo {0..9} {a..z}))
++++ echo 0 1 2 3 4 5 6 7 8 9 a b c d e f g h i j k l m n o p q r s t u v w x y z
++++ bc
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n c
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n 8
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n z
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n 5
+++ for i in $(bc <<< "obase=36; $input")
+++ echo -n s
+++ echo
++ local size36=c8z5s
++ printf '"%s%s"\n' qabj18 c8z5s
+ local 'etag="qabj18c8z5s"'
++ curl https://stable.release.flatcar-linux.net/./amd64-usr/./2345.3.0/./flatcar_production_image.bin.bz2 -I -H 'If-None-Match: "qabj18c8z5s"' --location --retry 5 --silent -f -S
+ log='HTTP/2 200
server: nginx/1.17.4
date: Thu, 14 May 2020 11:11:30 GMT
content-type: application/octet-stream
content-length: 489265127
etag: "q6hc7a83andz"
last-modified: Sat, 29 Feb 2020 20:13:58 GMT
accept-ranges: bytes
'
+ [[ 0 != \0 ]]
++ echo 'HTTP/2 200
server: nginx/1.17.4
date: Thu, 14 May 2020 11:11:30 GMT
content-type: application/octet-stream
content-length: 489265127
etag: "q6hc7a83andz"
last-modified: Sat, 29 Feb 2020 20:13:58 GMT
accept-ranges: bytes
'
++ grep '304 Not Modified'
+ local notmodified=
+ [[ ! -z '' ]]
+ echo -n +
++ echo 'Updating https://stable.release.flatcar-linux.net/./amd64-usr/./2345.3.0/./flatcar_production_image.bin.bz2'
Updating https://stable.release.flatcar-linux.net/./amd64-usr/./2345.3.0/./flatcar_production_image.bin.bz2
++ curl https://stable.release.flatcar-linux.net/./amd64-usr/./2345.3.0/./flatcar_production_image.bin.bz2 -R -o ./flatcar_production_image.bin.bz2 --location --retry 5 --silent -f -S
^C++ rm /tmp/mirror-lock
++ trap - SIGTERM
++ kill -- -25007

Let user specify what the final output directory should be

Current situation
I am building a private basic docker image with the command options that I need, and then setting up docker-compose to allow us to quickly stand up a mirror. I run into an issue with bind mounts/volumes though because the script downloads files into the same directory it is in. I would like to setup my files like:

/app/flatcar-release-mirror.sh
/data/stable
/data/alpha

with separate volumes for /app and /data

Impact
This would change how the files are output.

Ideal future situation
The flatcar-release-mirror.sh script should take a --output-dir option to allow me to specify a relative or absolute path that the final files will be written to.

Implementation options
Let me know your thoughts on this, I have a fork and can submit a PR

Allow filtering by channel

To minimize the amount of downloads for minimal mirror, it would be nice to be able to mirror only specific channel. With combination with above version, this should allow downloading only one/few images.

Download images to temporary file

If update script gets interrupted or the image is updated and used at the same time, this may lead to serving inconsistent data, which may lead to provisioning failure.

I think new images should be downloaded into temporary file and then moved into final destination.