Coder Social home page Coder Social logo

vegardit / fast-apt-mirror.sh Goto Github PK

View Code? Open in Web Editor NEW
21.0 21.0 2.0 59 KB

fast-apt-mirror.sh is a self-contained Bash script that helps you to easily and quickly determine and configure a fast APT repository mirror on Debian, Ubuntu and Pop!_OS systems.

Home Page: https://buymeacoffee.com/vegardit

License: Apache License 2.0

Shell 98.75% Ruby 1.25%
apt apt-get apt-repo-action apt-repository apt-source bash bash-script command-line-tool debian mirrors ubuntu

fast-apt-mirror.sh's People

Contributors

cloudenum avatar sebthom avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

dbpaul 424778940z

fast-apt-mirror.sh's Issues

cannot determine current apt mirror

Not sure what happened or what could have even caused this, it might just be actions being weird, seems to be pretty recent.

Seen here: https://github.com/tinygrad/tinygrad/actions/runs/5881949748/job/15951433949?pr=1558
here: https://github.com/tinygrad/tinygrad/actions/runs/5881949748/job/15951433775?pr=1558
here: https://github.com/tinygrad/tinygrad/actions/runs/5881931452/job/15951377952?pr=1537
here: https://github.com/tinygrad/tinygrad/actions/runs/5875504570/job/15931888916?pr=1076
here: https://github.com/tinygrad/tinygrad/actions/runs/5875504570/job/15931889162?pr=1076

  APT::Acquire::Retries "3";
  + curl -sSfL --retry 3 --max-time 6 https://raw.githubusercontent.com/vegardit/fast-apt-mirror.sh/v1/fast-apt-mirror.sh -o /home/runner/fast-apt-mirror.sh
  Current mirror: unknown
  Randomly selecting 20 mirrors...done
  Checking health status of 20 mirrors using '/dists/focal-security/Contents-amd64.gz'....................done
   -> outdated   (last modified: 2023-08-15 08:47:44 UTC) http://mirror.umd.edu/ubuntu/
   -> outdated   (last modified: 2023-08-15 08:47:44 UTC) http://mirror.umd.edu/ubuntu/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) http://mirrors.rit.edu/ubuntu/
   -> outdated   (last modified: 2023-08-15 08:47:44 UTC) http://mirror.mrjester.net/ubuntu/archive/
   -> outdated   (last modified: 2023-08-15 08:47:44 UTC) http://mirror.mrjester.net/ubuntu/archive/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) http://mirror.steadfastnet.com/ubuntu/
   -> outdated   (last modified: 2023-08-15 08:47:44 UTC) http://plug-mirror.rcac.purdue.edu/ubuntu/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) http://mirror.pit.teraswitch.com/ubuntu/
   -> outdated   (last modified: 2023-08-01 05:51:46 UTC) https://mirror.mia.velocihost.net/ubuntu/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) http://repos.forethought.net/ubuntu/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) http://www.gtlib.gatech.edu/pub/ubuntu/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) http://mirror.us-sc.kamatera.com/ubuntu/
   -> outdated   (last modified: 2023-08-15 08:47:44 UTC) https://mirrors.wikimedia.org/ubuntu/
   -> outdated   (last modified: 2023-08-15 08:47:44 UTC) http://mirror.uoregon.edu/ubuntu/
   ->                         n/a                          https://mirror.enzu.com/ubuntu/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) http://mirrors.syringanetworks.net/ubuntu-archive/
   ->                         n/a                          http://ftp.ussg.iu.edu/linux/ubuntu/
   -> outdated   (last modified: 2020-08-19 04:19:51 UTC) http://mirrors.arpnetworks.com/Ubuntu/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) https://mirror.lstn.net/ubuntu/
   -> UP-TO-DATE (last modified: 2023-08-16 06:07:16 UTC) https://repo.ialab.dsu.edu/ubuntu/
   => 9 mirrors are reachable and up-to-date
  Speed testing 9 of the available 9 mirrors (sample download size: 1024KB).........done
   -> http://mirrors.syringanetworks.net/ubuntu-archive/ (782KB/s)
   -> http://repos.forethought.net/ubuntu/ (1.1MB/s)
   -> http://mirror.us-sc.kamatera.com/ubuntu/ (2.0MB/s)
   -> https://repo.ialab.dsu.edu/ubuntu/ (2.7MB/s)
   -> https://mirror.lstn.net/ubuntu/ (2.8MB/s)
   -> http://mirror.pit.teraswitch.com/ubuntu/ (3.5MB/s)
   -> http://mirror.steadfastnet.com/ubuntu/ (5.2MB/s)
   -> http://www.gtlib.gatech.edu/pub/ubuntu/ (5.7MB/s)
  Current mirror: unknown
  ERROR: Cannot set APT mirror: cannot determine current APT mirror.

Doesn't update multiple repos in the same deb822 file (Ubuntu 24.04)

The fast-apt-mirror.sh find --apply feature doesn't update the repo URI if there are multiple repositories in the same deb822-formated file, as is the case by default in Ubuntu 24.04. It appears to only update the first one.

Default Ubuntu 22.04 /etc/apt/sources.list.d/ubuntu.sources file (relevant lines):


Types: deb
URIs: http://archive.ubuntu.com/ubuntu
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb
URIs: http://security.ubuntu.com/ubuntu
Suites: noble-security
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Ubuntu 22.04 /etc/apt/sources.list.d/ubuntu.sources file after running fast-apt-mirror.sh find --apply (relevant lines):


Types: deb
URIs: http://fast.mirror.example.com/ubuntu
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb
URIs: http://security.ubuntu.com/ubuntu
Suites: noble-security
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Note that only the URI of the first repository is changed even though both are "main" repos.

The desired behavior would be that both URIs are changed.


Types: deb
URIs: http://fast.mirror.example.com/ubuntu
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb
URIs: http://fast.mirror.example.com/ubuntu
Suites: noble-security
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Mirror not changed (or not fully tested)

CI build still fails as mirror only tested for downloading for benchmark.

Today's issue is a bit different, which is not covered by the script.
Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)

Also maybe an option to blacklist repos would be helpful, or at least an option to ban all azure repos

Linux: Configure fast APT mirror
  APT::Acquire::Retries "3";
  + curl -sSfL --retry 3 --max-time 6 https://raw.githubusercontent.com/vegardit/fast-apt-mirror.sh/v1/fast-apt-mirror.sh -o /home/runner/fast-apt-mirror.sh
  Current mirror: http://azure.archive.ubuntu.com/ubuntu/ (/etc/apt/apt-mirrors.txt)
  Randomly selecting 20 mirrors...done
  Checking health status of 20 mirrors using '/dists/jammy-security/Contents-amd64.gz'....................done
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://archive.ubuntu.com/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://azure.archive.ubuntu.com/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirror.cogentco.com/pub/linux/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirror.math.princeton.edu/pub/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirror.metrocast.net/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirror.team-cymru.org/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirror.us-ny2.kamatera.com/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirror.us-tx.kamatera.com/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirrors.liquidweb.com/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirrors.ocf.berkeley.edu/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://mirrors.usinternet.com/ubuntu/archive/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://ubuntu.phoenixnap.com/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) http://www.gtlib.gatech.edu/pub/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) https://atl.mirrors.clouvider.net/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) https://la.mirrors.clouvider.net/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) https://mirrors.iu13.net/ubuntu/
   -> UP-TO-DATE (last modified: 2024-04-23 01:29:38 UTC) https://mirrors.tscak.com/ubuntu/
   -> outdated   (last modified: 2024-04-20 01:29:58 UTC) http://mirrors.advancedhosters.com/ubuntu/
   -> outdated   (last modified: 2023-01-26 05:30:47 UTC) https://mirror.enzu.com/ubuntu/
   ->                         n/a                          http://mirrors.arpnetworks.com/Ubuntu/
   => 19 mirrors are reachable
  Speed testing 10 of the available 19 mirrors (sample download size: 1024KB)..........done
   -> http://mirror.team-cymru.org/ubuntu/ (174KB/s)
   -> http://archive.ubuntu.com/ubuntu/ (1.5MB/s)
   -> http://mirrors.ocf.berkeley.edu/ubuntu/ (1.6MB/s)
   -> http://mirror.us-tx.kamatera.com/ubuntu/ (2.6MB/s)
   -> http://mirrors.liquidweb.com/ubuntu/ (3.6MB/s)
   -> http://mirror.metrocast.net/ubuntu/ (4.2MB/s)
   -> http://mirror.math.princeton.edu/pub/ubuntu/ (5.6MB/s)
   -> http://mirror.us-ny2.kamatera.com/ubuntu/ (14MB/s)
   -> http://mirror.cogentco.com/pub/linux/ubuntu/ (21MB/s)
   => http://azure.archive.ubuntu.com/ubuntu/ (82MB/s) determined as fastest mirror within 5 seconds
  Current mirror: http://azure.archive.ubuntu.com/ubuntu/ (/etc/apt/apt-mirrors.txt)
  Nothing to do, already using: http://azure.archive.ubuntu.com/ubuntu/
  http://azure.archive.ubuntu.com/ubuntu/
  + sudo apt-get update
  Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [142 B]
  Get:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
  Err:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
    Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
  Hit:2 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
  Get:3 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
  Get:4 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
  Get:5 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
  Get:7 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1610 kB]
  Get:8 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB]
  Get:9 http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1830 kB]
  Hit:13 https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu jammy InRelease
  Get:10 http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB]
  Get:11 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1070 kB]
  Get:12 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB]
  Get:14 http://azure.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB]
  Get:15 http://azure.archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB]
  Get:16 http://azure.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1[39](https://github.com/OneKeyHQ/bluetooth-firmware-pro/actions/runs/8814823025/job/24195516436#step:2:41)2 kB]
  Get:17 http://azure.archive.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB]
  Get:18 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1768 kB]
  Get:19 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB]
  Get:20 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB]
  Get:21 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB]
  Reading package lists...
  E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
  E: The repository 'https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease' is no longer signed.
  Error: Process completed with exit code 100.

Add the option to select country for ubuntu-based distro

As of now the mirrors.txt will always return

http://archive.ubuntu.com/ubuntu/

Which render this script become useless, because it is essentially just speed testing the default mirror (which again already being used in most cases).

By adding the country options this issue can be solved, another way is using automatic country detection using IP.

Sometimes failing with Broken Pipe in Github Actions

Noticed here: https://github.com/tinygrad/tinygrad/actions/runs/5790180583/job/15692747572

Relevant log section:

APT::Acquire::Retries "3";
+ curl -sSfL --retry 3 --max-time 6 https://raw.githubusercontent.com/vegardit/fast-apt-mirror.sh/v1/fast-apt-mirror.sh -o /home/runner/fast-apt-mirror.sh
Current mirror: http://azure.archive.ubuntu.com/ubuntu/ (/etc/apt/sources.list)
Randomly selecting 20 mirrors...done
Checking health status of 20 mirrors....................done
 -> 15 mirrors are reachable and up-to-date
http://azure.archive.ubuntu.com/ubuntu/
https://mirrors.xtom.com/ubuntu/
https://mirrors.wikimedia.org/ubuntu/
http://www.club.cc.cmu.edu/pub/ubuntu/
http://ubuntu.phoenixnap.com/ubuntu/
http://ubuntu.mirror.frontiernet.net/ubuntu/
http://repos.forethought.net/ubuntu/
http://plug-mirror.rcac.purdue.edu/ubuntu/
http://mirrors.tripadvisor.com/ubuntu/
http://mirrors.syringanetworks.net/ubuntu-archive/
Speed testing 10 of the available 15 mirrors (sample download size: 1024KB)..........done
/home/runner/fast-apt-mirror.sh: line 337: echo: write error: Broken pipe
21:42:48 Error - exited with status 1 in /home/runner/fast-apt-mirror.sh at line 337:
21:42:48 Error - exited with status 1 in /home/runner/fast-apt-mirror.sh at line 337:
   334	    return $RC_MISC_ERROR
   335	  fi
   336	  fastest_mirror=$(echo "$mirrors_with_speed" | head -1 | cut -d" " -f2)
   337	  fastest_mirror_speed=$(echo "$mirrors_with_speed" | head -1 | cut -d" " -f1 | numfmt --to=iec --suffix=B/s)
   338	  >&2 echo " -> $fastest_mirror ($fastest_mirror_speed) determined as fastest mirror within $(( $(date +%s) - start_at )) seconds"
   339	  if [[ ${verbosity:-} -gt 0 ]]; then
   340	    echo "$mirrors_with_speed" | tail -n +2 | while IFS= read -r mirror; do

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.