coreos / afterburn Goto Github PK
View Code? Open in Web Editor NEWA one-shot cloud provider agent
Home Page: https://coreos.github.io/afterburn/
License: Apache License 2.0
A one-shot cloud provider agent
Home Page: https://coreos.github.io/afterburn/
License: Apache License 2.0
If an Azure instance is created with "Boot diagnostics: off" and then later on while booting the setting is turned to "on", Afterburn boot check-in will fail with an output like this:
Jul 21 13:21:25.449 INFO Fetching http://168.63.129.16/?comp=versions: Attempt #1
Jul 21 13:21:25.455 INFO Fetch successful
Jul 21 13:21:25.462 INFO Fetching http://168.63.129.16/machine/?comp=goalstate: Attempt #1
Jul 21 13:21:25.468 INFO Fetch successful
Jul 21 13:21:25.506 INFO Posting http://168.63.129.16/machine/?comp=health: Attempt #1
Jul 21 13:21:26.520 INFO Posting http://168.63.129.16/machine/?comp=health: Attempt #2
...
Jul 21 13:22:02.653 INFO Posting http://168.63.129.16/machine/?comp=health: Attempt #10
Error: failed to run
Caused by: checking-in instance boot to cloud provider
Caused by: maximum number of retries (10) reached
Caused by: POST failed: 410 Gone
This seems unrelated to #459, i.e. it isn't a timeout issue.
I played a guessing game on the root cause for this, and via empirical experimentation it seems to be due to a mismatch in GoalStateIncarnation
.
In particular, Azure documentation tells us to hardcode <GoalStateIncarnation>1</GoalStateIncarnation>
in requests.
However, flipping the setting seems to increment the machine incarnation, and querying the goalstate endpoint after that returns an XML document with <Incarnation>2</Incarnation>
.
Manually tweaking the request to have a matching Incarnation <-> GoalStateIncarnation
seems to fix the failure, even after further tweaking settings (manually confirmed with incarnations up to 4).
Any
At present, coreos-metadata --ssh-keys
implicitly assumes that update-ssh-keys is distributed as part of the OS. That is, it assumes that SSH keys should be written into ~/.ssh/authorized_keys.d/<file>
, and then all files in authorized_keys.d
should be merged into authorized_keys
. On distros without update-ssh-keys, this model doesn't immediately cause problems: coreos-metadata writes ~/.ssh/authorized_keys.d/coreos-metadata
and then performs the sync via an embedded copy of the update-ssh-keys library. However, if the user updates authorized_keys
directly and reboots, and coreos-metadata --ssh-keys
runs again, it will clobber changes to authorized_keys
. Instead, the user is expected to put their keys in e.g. ~/.ssh/authorized_keys.d/my-keys
, but then doesn't have a tool to perform the synchronization.
Ultimately, update-ssh-keys is a hack for supporting authorized_keys.d
directories without any cooperation from sshd, and should be dropped. It changes the workflow of updating SSH keys and isn't used on other distros; Fedora CoreOS and Red Hat CoreOS currently do not intend to ship it. Instead, in the short term, Fedora CoreOS will have special-case support for authorized_keys.d
files written by individual tools, and in the medium term it will add generic support for the key directory (coreos/fedora-coreos-tracker#139 (comment)).
Modify coreos-metadata as follows:
authorized_keys.d
fragment file, don't merge the fragment file into authorized_keys
except on CL. Probably gate this on a compile-time flag.coreos-metadata --ssh-keys
will only work on distros that include .ssh/authorized_keys.d/coreos-metadata
in sshd's AuthorizedKeysFile
, or that otherwise arrange to read all keyfiles in .ssh/authorized_keys.d
.This project uses cargo-release in order to prepare new releases, tag and sign the relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
RELEASE_VER=x.y.z
UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git checkout v${RELEASE_VER}
grep "^version = \"${RELEASE_VER}\"$" Cargo.toml
produces outputgit push ${UPSTREAM_REMOTE} v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
Create a daemon mode for afterburn for clouds where having a reactive/scheduled pattern makes sense instead of oneshot runs.
TBD
Allow specific providers to have a daemon mode which runs consistently and reacts to specific changes.
@darkmuggle has done some initial plumbing work.
Cloud providers may have specific quirks and restrictions which could result in surprising behaviors of coreos-metadata
or which could require workarounds in the codebase. Those should be documented in the repository, so that we have an explicit note to reference.
The first example of this has been reported at coreos/bugs#2468 (comment).
Azure may (or may not, depending on agent version and settings) prevent non-root users to reach the metadata endpoint for security reasons. This affects how the coreos-metadata
service can be run.
Followup to coreos/coreos-assembler#1119 (comment).
We decided to go in a different direction and move Classic logic to a dedicated platform ID ibmcloud-classic
.
30.20191014.1 (CoreOS preview)
(happened in previous versions as well)
4.1.3
aws
no failing systemd unit after boot
● [email protected] - Afterburn (SSH Keys)
Loaded: loaded (/usr/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2019-11-28 10:25:13 UTC; 8min ago
Process: 934 ExecStart=/usr/bin/afterburn ${AFTERBURN_OPT_PROVIDER} --ssh-keys=core (code=exited, status=1/FAILURE)
Main PID: 934 (code=exited, status=1/FAILURE)
Nov 28 10:25:12 ip-10-100-20-105.eu-west-1.compute.internal systemd[1]: Starting Afterburn (SSH Keys)...
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Nov 28 10:25:13.314 INFO Fetching http://169.254.169.254/2009-04-04/meta-data/public-keys: Attempt #1
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Nov 28 10:25:13.327 INFO Fetch failed with 404: resource not found
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Error: writing ssh keys
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Caused by: failed to open "/var/home/core/.ssh/authorized_keys.d" for syncing
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Caused by: No such file or directory (os error 2)
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal systemd[1]: [email protected]: Failed with result 'exit-code'.
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal systemd[1]: Failed to start Afterburn (SSH Keys).
There has roughly been the answer to life, the universe, and everything
commits since 4.1.0
was released. Recently #224 merged and we'd like to start utilizing it. Could we get a new release cut for afterburn
?
This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
export RELEASE_VER=x.y.z
export UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
git checkout v${RELEASE_VER}
git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
unset RELEASE_VER
unset UPSTREAM_REMOTE
I booted a FCOS VM in Azure with just an Ignition config and --authentication-type password
to disable Azure's requirement for SSH keys.
[root@walters-fcos ~]# systemctl status afterburn-sshkeys@core
● [email protected] - Afterburn (SSH Keys)
Loaded: loaded (/usr/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-04-28 15:26:00 UTC; 48s ago
Process: 903 ExecStart=/usr/bin/afterburn ${AFTERBURN_OPT_PROVIDER} --ssh-keys=core (code=exited, status=1/FAILURE)
Main PID: 903 (code=exited, status=1/FAILURE)
Apr 28 15:26:00 walters-fcos afterburn[903]: Apr 28 15:26:00.092 INFO Fetch successful
Apr 28 15:26:00 walters-fcos afterburn[903]: Error: failed to run
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: writing ssh keys
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: failed to get certs
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: failed to get certificates
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: failed to parse uri
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: relative URL without a base
Apr 28 15:26:00 walters-fcos systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Apr 28 15:26:00 walters-fcos systemd[1]: [email protected]: Failed with result 'exit-code'.
Apr 28 15:26:00 walters-fcos systemd[1]: Failed to start Afterburn (SSH Keys).
Azure, Packet
Consider adding a command-line option or mode which reports a successful boot to the cloud platform and then exits. On platforms that don't require health reporting, the command would silently succeed.
The idea is that the command would run as a systemd oneshot after greenboot is happy.
See also coreos/fedora-coreos-tracker#69.
Mounting a config-drive requires CAP_SYS_ADMIN
in the thread invoking the mount
syscall. This is typically true when running coreos-metadata
as the root user, but it would be better to make this explicit and only requiring the specific capabilities it needs.
This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
export RELEASE_VER=x.y.z
export UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
git checkout v${RELEASE_VER}
git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
unset RELEASE_VER
unset UPSTREAM_REMOTE
This is related to #94. If we experience a network failure on Azure while trying to reach the metadata endpoint and the process is not running as root, we should log a more informative message about possible Azure-specific local firewalling,
Instances on ibmcloud
platform (i.e. IBM cloud VPC gen2)
The afterburn-sshkeys
service should work on this platform too, grabbing SSH keys from instance metadata and forwarding them to authorized keys.
On Azure, Afterburn currently sends requests to the metadata/wireserver endpoint with some mildly aggressive retrying logic, which on failures results in sending about 10 requests in ~1 minute.
According to Azure/azure-sdk-for-go#1848, the WireServer routinely goes down for maintenance with a p90 downtime up to 3 minutes (!) for each restart event.
As such, in order to be more lenient toward transient platform issues, we should relax our deadline for retries as to cover extended downtime up to 4 or 5 minutes.
Fedora CoreOS is targeting Alibaba Cloud as a new platform: coreos/fedora-coreos-tracker#276.
Afterburn should also support that, with the platform ID aliyun
.
This cloud platform has a metadata endpoint, documented at https://www.alibabacloud.com/help/doc-detail/49122.htm.
This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
export RELEASE_VER=x.y.z
export UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
git checkout v${RELEASE_VER}
git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
unset RELEASE_VER
unset UPSTREAM_REMOTE
Tagging coreos/fedora-coreos-tracker#279 issue for more visibility here.
In order to solve the above issue, there's one bullet point that talks about displaying information related to ssh_authorized_keys that were added by the afterburn
process. After having some discussion around this topic(coreos/fedora-coreos-config#344 (review)), it appears that sending a journal entry makes more sense here. Ideally, we need to use https://github.com/lucab/libsystemd-rs library to support this feature.
But it looks like (lucab/libsystemd-rs#34) isn't present in the latest release. It would be really helpful if we cut a new release of that library to support this feature.
coreos-metadata.service
does not start on Core OS in Azure.
> cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1800.5.0
VERSION_ID=1800.5.0
BUILD_ID=2018-07-28-2250
PRETTY_NAME="Container Linux by CoreOS 1800.5.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
Azure VM
coreos-metadata.service
starts
coreos-metadata.service
fails
> sudo journalctl -u coreos-metadata.service
-- Logs begin at Thu 2018-08-02 14:43:17 UTC, end at Thu 2018-08-02 15:03:11 UTC. --
Aug 02 14:47:20 teamcity systemd[1]: Starting CoreOS Metadata Agent...
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.742 INFO Fetching http://168.63.129.16/?comp=versions: Attempt #1
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.765 INFO Fetch successful
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.765 INFO Fetching http://168.63.129.16/machine/?comp=goalstate: Attempt #1
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.769 INFO Fetch successful
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.971 INFO Fetching http://168.63.129.16/machine/aa786c20-80a9-4a09-8936-2104c27e6e5c/7f691c1e%2D37bf%2D4b9a%2D8c90%2D010bcc7c4277.%5Fteamcity?comp=certificates&incarnation=2>
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.979 INFO Fetch successful
Aug 02 14:47:20 teamcity coreos-metadata[1503]: thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /build/amd64-usr/var/tmp/portage/dev-libs/rustlib-1.26.0/work/rustc-1.26.0-src/src/libcore/option.rs:335:21
Aug 02 14:47:21 teamcity systemd[1]: coreos-metadata.service: Main process exited, code=exited, status=101/n/a
Aug 02 14:47:21 teamcity systemd[1]: coreos-metadata.service: Failed with result 'exit-code'.
Aug 02 14:47:21 teamcity systemd[1]: Failed to start CoreOS Metadata Agent.
> cat /usr/lib/systemd/system/coreos-metadata.service
[Unit]
Description=CoreOS Metadata Agent
[Service]
Type=oneshot
Environment=COREOS_METADATA_OPT_PROVIDER=--cmdline
ExecStart=/usr/bin/coreos-metadata ${COREOS_METADATA_OPT_PROVIDER} --attributes=/run/metadata/coreos
[Install]
RequiredBy=metadata.target
AliCloud havn't DHCP for set hostname when firstboot, could you add condition for aliyun in afterburn-hostname.service ?
@lucab
Afterburn usually follows Ignition platform naming convention, in order to auto-detect the platform.
However there are several platforms where the metadata component is only optional. On those platforms, in case of failures, Afterburn cannot distinguish a known-good state (i.e. this infra lacks the metadata component) from a failed state (i.e. network or logic error).
As such, Afterburn knows about the following platform which are more specific than the Ignition ones, cannot be auto-detected, and require user-provided hints (as cmdline flags):
cloudstack-metadata
cloudstack-configdrive
openstack-metadata
vagrant-virtualbox
RHCOS/FCOS uses NetworkManager presently. So the logic to look for option 245 in the systemd-networkd lease file would need to be adapted to read from those lease files (/var/lib/NetworkManager
on RHEL7, /var/lib/dhclient
on FCOS and likely RHEL8). Might be worth investigating if NM's D-Bus API exposes this so we don't have to learn dhclient's syntax.
Though note that this is not a blocker for getting those platforms on Azure if we're comfortable relying exclusively on the fallback endpoint (#145) until this is implemented.
Related: #130
This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
export RELEASE_VER=x.y.z
export UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
git checkout v${RELEASE_VER}
git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
unset RELEASE_VER
unset UPSTREAM_REMOTE
This ticket is a work item for the following TODO note:
afterburn/src/providers/ibmcloud/mod.rs
Lines 23 to 24 in dba86a4
This is only relevant for ibmcloud
Classic users, resulting in a unnecessary delay of a few seconds.
This project uses cargo-release in order to prepare new releases, tag and sign the relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
RELEASE_VER=x.y.z
UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git checkout v${RELEASE_VER}
grep "^version = \"${RELEASE_VER}\"$" Cargo.toml
produces outputgit push ${UPSTREAM_REMOTE} v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
Afterburn with --cmdline argument does not work on OpenStack
fedora-coreos-31.20200210.3.0
Afterburn 4.1.3
OpenStack
/usr/bin/afterburn --cmdline--attributes=/run/metadata/afterburn
Mar 07 23:20:33.942 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #1
Mar 07 23:20:34.412 INFO Fetch successful
Mar 07 23:20:34.412 INFO Fetching http://169.254.169.254/latest/meta-data/instance-id: Attempt #1
Mar 07 23:20:34.503 INFO Fetch successful
Mar 07 23:20:34.504 INFO Fetching http://169.254.169.254/latest/meta-data/local-ipv4: Attempt #1
Mar 07 23:20:34.592 INFO Fetch successful
Mar 07 23:20:34.592 INFO Fetching http://169.254.169.254/latest/meta-data/public-ipv4: Attempt #1
Mar 07 23:20:34.676 INFO Fetch successful
/usr/bin/afterburn --cmdline
Error: fetching metadata from provider
Caused by: unknown provider 'openstack'
cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt1)/ostree/fedora-coreos-19190477fad0e60d605a623b86e06bb92aa318b6b79f78696b06f68f262ad5d6/vmlinuz-5.4.17-200.fc31.x86_64 mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=ttyS0,115200n8 ignition.firstboot rd.neednet=1 ip=dhcp ostree=/ostree/boot.1/fedora-coreos/19190477fad0e60d605a623b86e06bb92aa318b6b79f78696b06f68f262ad5d6/0 ignition.platform.id=openstack
cloud provider reported by cmdline is openstack. Not openstack-metadata
When building coreos-3.0.1, rpmlint reports call to setgroups
before setuid
.
This appears to be in crate users-0.7.0
, a dependency of direct dependency crate update-ssh-keys-0.3.0
. Some changes in this area have been integrated in users-0.8.0, currently researching whether those eliminate the warning.
The warning may not indicate an actual problem, but it would be helpful to eliminate the warning if a fix is available through a newer version of a dependency.
OpenSUSE Tumbleweed 20181018-0 x86_64
coreos-metadata-3.0.1
Open Build Service (OBS) osc build
local.
Local RPM lint warning at build time, no error running executable. Not running on any cloud provider.
Build executable that passes standard RPM lint checks.
RPMLINT report:
===============
coreos-metadata.x86_64:
W: missing-call-to-setgroups-before-setuid /usr/bin/coreos-metadata
This executable is calling setuid and setgid without setgroups or initgroups.
There is a high probability this means it didn't relinquish all groups, and
this would be a potential security issue to be fixed. Seek POS36-C on the web
for details about the problem.
Occurrences of setgroups
are all in crate libc-0.2.43
.
Occurrences of setuid
are in crate libc-0.2.43
and crate users-0.7.0
.
The rpmlint warning is not specific to a particular call site. Dependency crate users-0.7.0
has potentially relevant code here: https://github.com/ogham/rust-users/blob/v0.7.0/src/switch.rs#L25
Commits between crate users-0.7.0
(pinned) and crate users-0.8.0
(available) suggest some work may have been done in this area: ogham/rust-users@v0.7.0...master
I will test a build with crate users-0.8.0
and report back here.
Afterburn does not set the hostname on Azure.
RHCOS
master
Azure
/etc/hostname
contains the machine name defined on Azure
no file
curl http://169.254.169.254/metadata/instance/compute/name?api-version=2017-08-01&format=text
gives you the hostname
file location should be /etc/hostname
Currently Azure provider returns None
:
afterburn/src/providers/azure/mod.rs
Lines 413 to 415 in 7db2aff
Due to a bug in the underlying library, some virtual CPU in vmware may be missed by the detection logic, resulting in boot failure when trying to use custom networking cmdline arguments. Bumping vmw_backdoor
to 0.1.3 should fix this.
On each release we publish a .crate
file to crates.io, which is then used by distribution to package coreos-metadata. Its SHA256 is recorded at publish time in the index, eg
rust-lang/crates.io-index@e0bc5cf.
In order to improve the integrity chain and to make manual checks easier, we should also record the SHA256 of the .crate
file on local filesystem (before upload) and publish it in the release notes.
As per discussion with @ashcrow.
This project uses cargo-release in order to prepare new releases, tag and sign the relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
RELEASE_VER=x.y.z
UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git checkout v${RELEASE_VER}
grep "^version = \"${RELEASE_VER}\"$" Cargo.toml
produces outputgit push ${UPSTREAM_REMOTE} v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
When I looked at the providers, there are cloudstack
, openstack
, azurestack
(in the near future) confusing me in the beginning. Would help with brief intro or website links for platforms if anyone feels the same way.
Currently if FCOS fails in very early boot (e.g. in the bootloader or kernel, before switching to the initramfs), it's...hard to detect consistently. For example, we have a test for Secure Boot, but it turns out that if the kernel fails to verify then...coreos-assembler today hangs for a really long time.
We can scrape the serial console in most cases, but we configure the Live ISO not to log to a serial console...so we'd end up having instead to do something like OpenQA and do image recognition on the graphics console 😢
Now we debated this somewhat in
coreos/ignition-dracut#170
and I argued strongly that the most important thing was to cover the "failure in initramfs" case, and we could support the "stream journal in general" by injecting Ignition.
In retrospect...I think I was wrong. It would be extremely useful for us to stream the journal starting from the initramfs at least by default on qemu.
In particular, what we really want is some sort of message from the VM that it has entered the initramfs, but before we start processing Ignition. If we're doing things like reprovisioning the rootfs, it becomes difficult to define a precise "timeout bound". But I think we can e.g. reliably time out after something quite low (like 10 seconds) if we haven't seen the "entered initramfs" message.
So here's my proposal:
Tagging coreos/fedora-coreos-tracker#279 issue for more visibility here.
In order to solve the above issue, there's one bullet point that talks about displaying information related to ssh_authorized_keys that were added by the afterburn
process. After having some discussion around this topic(coreos/fedora-coreos-config#344 (review)), it appears that sending a journal entry makes more sense here. Ideally, we need to use https://github.com/lucab/libsystemd-rs library to support this feature.
But it looks like (lucab/libsystemd-rs#34) isn't present in the latest release. It would be really helpful if we cut a new release of that library to support this feature.
Afterburn fails completely due to 500 errors on the metadata source. With coreos/ignition@0c0ec63 I was able to boot on AzureStack.
A complete different issue:
s)...[ 32.768570] NetworkManager[568]: <info> [1593115470.6525] dhcp4 (eth0): option private_245 => 'a8:3f:81:10'
And then:
[ 64.908820] afterburn[658]: Jun 25 19:57:52.985 WARN Failed to get fabric address from DHCP: maximum number of retries (60) reached
[ 64.988395] afterburn[658]: Jun 25 19:57:52.986 INFO Using fallback address
[ 65.033307] afterburn[658]: Jun 25 19:57:52.986 INFO Fetching http://168.63.129.16/?comp=versions: Attempt #1
^M[ *] A start job is running for Afterburn Hostname (52s / no limit)
[ 65.566088] afterburn[658]: Jun 25 19:57:53.643 INFO Fetch successful
[ 65.621959] afterburn[658]: Jun 25 19:57:53.643 INFO Fetching http://168.63.129.16/machine/?comp=goalstate: Attempt #1
[ 65.698749] afterburn[658]: Jun 25 19:57:53.651 INFO Fetch successful
[ 65.747770] afterburn[658]: Jun 25 19:57:53.659 INFO Fetching http://169.254.169.254/metadata/instance/compute/name?api-version=2017-08-01&format=text: Attempt #1
[ 65.942651] afterburn[658]: Jun 25 19:57:53.674 INFO Failed to fetch: 500 Internal Server Error
And ending with:
Displaying logs from failed units: afterburn-hostname.service
-- Logs begin at Thu 2020-06-25 20:04:16 UTC, end at Thu 2020-06-25 20:05:59 UTC. --
Jun 25 20:05:51 afterburn[655]: Jun 25 20:05:51.338 INFO Failed to fetch: 500 Internal Server Error
Jun 25 20:05:51 afterburn[655]: Error: failed to run
Jun 25 20:05:51 afterburn[655]: Caused by: writing hostname
Jun 25 20:05:51 afterburn[655]: Caused by: failed to get hostname
Jun 25 20:05:51 afterburn[655]: Caused by: maximum number of retries (10) reached
Jun 25 20:05:51 afterburn[655]: Caused by: failed to fetch: 500 Internal Server Error
Jun 25 20:05:51 systemd[1]: afterburn-hostname.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 20:05:51 systemd[1]: afterburn-hostname.service: Failed with result 'exit-code'.
Jun 25 20:05:51 systemd[1]: Failed to start Afterburn Hostname.
AWS recently announced a new generation of their metadata endpoint, called IMDSv2.
It is a session-based service, meant to plug a few holes in the v1 design, most notably SSRF.
See the announcement at https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/.
For the moment both v1 and v2 are active by default, and instances can be configured to disable the v1 endpoint.
In the medium term, Afterburn should move to v2 logic.
This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
export RELEASE_VER=x.y.z
export UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
git checkout v${RELEASE_VER}
git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
unset RELEASE_VER
unset UPSTREAM_REMOTE
AWS
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html describes IMDSv2, which is a 2-step process to retrieving instance metadata. Enforcing IMDSv2 is beneficial for preventing attacks that trick an application into making requests to the IMDS and returning data such as the instance IAM credentials.
A node can be launched to only support IMDSv2, so I suggest making afterburn compliant with this to enable the v2-only switch to be activated.
More details about IMDSv2:
All.
Building release binaries with panic=abort
reduces their size by a considerable amount, which would make including coreos-metadata
in the initramfs more palatable. However, it might interfere with some cleanup behavior we are currently relying on with panic=unwind
.
See #105 for the original discussion.
Any
Consider renaming coreos-metadata. It seems that the project is drifting toward becoming a minimal, generic cloud agent, in which case the name is not very apt. In particular:
coreos
isn't great.metadata
isn't great either.COREOS_
is not great in output variable names. Changing it would be very user-visible and may not be feasible. We could output both old and new names for compatibility, but it's not clear that we could ever stop outputting the old names. We could drop the old names on non-CL distros, but users coming from CL will presumably have unit files and scripts that depend on the old names, and conversion tools (coreos/fedora-coreos-tracker#48) may not be able to catch every use of them.For a new project name, we could go the descriptive route with "platform-agent" or the like, but that doesn't seem very inspired. Maybe an arbitrary name would be better.
What hardware/cloud provider/hypervisor is being used to run Afterburn?
Packet, specifically the n2 servers
At
afterburn/src/providers/packet/mod.rs
Line 248 in d56ad64
Noted by Kinvolk in the course of an integration.
Since in fedora-coreos-pinger
, we need to fetch the instance type of cloud providers and I am trying to avoid too much duplicate code between fedora-coreos-pinger, Afterburn and Zincati, it would be great if we could fetch cloud instance type here.
Apply to all cloud providers.
Fetch cloud instance type
for all providers.
Link to related fedora-coreos-pinger issue: coreos/fedora-coreos-pinger#30
AWS
After SSH keys have been set, don't continue to set them on every boot. With the current behavior, keys that are deleted by the user can come back later, which is surprising in the Ignition world.
Alternatively, document how to disable the service.
See coreos/bugs#1933 (comment) for context.
AWS
afterburn/src/providers/aws/mod.rs
Line 65 in d8f74d5
This is a feature request to add support for a new cloud platform, Vultr.
Their metadata API is described at https://www.vultr.com/metadata/. We are mostly interested in attributes and SSH pubkeys at this point.
Overall platform discussion for Fedora Coreos is ongoing at coreos/fedora-coreos-tracker#355 (which covers ID naming too).
This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.
In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).
This guide requires:
git
cargo
(suggested: latest stable toolchain from rustup)cargo-release
(suggested: cargo install -f cargo-release
)These steps show how to release version x.y.z
on the origin
remote (this can be checked via git remote -av
).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.
cargo test
cargo clean
git clean -fd
export RELEASE_VER=x.y.z
export UPSTREAM_REMOTE=origin
UPSTREAM_REMOTE
should reference the locally configured remote that points to the upstream git repository.
create release commits on a dedicated branch and tag it:
git checkout -b release-${RELEASE_VER}
cargo release
(and confirm the version when prompted)open a PR for this release:
git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
get the PR reviewed, approved and merged
publish the artifacts (tag and crate):
git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
git checkout v${RELEASE_VER}
git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
cargo publish
assemble vendor archive:
cargo vendor
tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor
publish this release on GitHub:
target/afterburn-${RELEASE_VER}-vendor.tar.gz
sha256sum target/package/afterburn-${RELEASE_VER}.crate
sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
clean up:
cargo clean
rm -rf vendor
git checkout master
git pull ${UPSTREAM_REMOTE} master
git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
git branch -d release-${RELEASE_VER}
unset RELEASE_VER
unset UPSTREAM_REMOTE
Afterburn is currently a one-short binary, entirely configured via command-line flags. This is on purpose to keep the complexity (both internally and externally) low.
Our service units are generally written to be easy to override via drop-ins, such as:
[Service]
Environment=AFTERBURN_OPT_PROVIDER="--provider foo"
However, if we start increasing the design complexity (e.g. with a daemon mode, #228), we should consider sourcing file-based configurations.
This would be ideally done via overlayed directories with dropins, plus a simple and standard format (e.g. TOML).
What hardware/cloud provider/hypervisor is being used to run Afterburn?
libvirt running Fedora CoreOS when this was brought up - applies to all platforms though.
Add a target afterburn-sshkeys.target
, and have the [email protected]
template specify RequiredBy=afterburn-sshkeys.target
in the [Install]
section. This would allow other units to synchronize around any instance of afterburn-sshkeys@
, not only for a given user (e.g. [email protected]
, see coreos/fedora-coreos-config#344 (comment)).
First need to test/investigate manually if this approach works - filing as an initial suggestion, and potentially can investigate other ideas for the synchronization here if this does not work.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.