coreos / afterburn

A one-shot cloud provider agent

Afterburn

Afterburn is a one-shot agent for cloud-like platforms which interacts with provider-specific metadata endpoints. It is typically used in conjunction with Ignition.

Features

It comprises several modules which may run at different times during the lifecycle of an instance.

Depending on the specific platform, the following services may run in the initramfs on first boot:

• setting local hostname
• injecting network command-line arguments

The following features are conditionally available on some platforms as systemd service units:

• installing public SSH keys for local system users
• retrieving attributes from instance metadata
• checking in to the provider in order to report a successful boot or instance provisioning

Supported platforms

See Supported platforms.

afterburn's Issues

azure: fails with ignition+password auth

I booted a FCOS VM in Azure with just an Ignition config and --authentication-type password to disable Azure's requirement for SSH keys.

[root@walters-fcos ~]# systemctl status afterburn-sshkeys@core
● [email protected] - Afterburn (SSH Keys)
   Loaded: loaded (/usr/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2020-04-28 15:26:00 UTC; 48s ago
  Process: 903 ExecStart=/usr/bin/afterburn ${AFTERBURN_OPT_PROVIDER} --ssh-keys=core (code=exited, status=1/FAILURE)
 Main PID: 903 (code=exited, status=1/FAILURE)

Apr 28 15:26:00 walters-fcos afterburn[903]: Apr 28 15:26:00.092 INFO Fetch successful
Apr 28 15:26:00 walters-fcos afterburn[903]: Error: failed to run
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: writing ssh keys
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: failed to get certs
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: failed to get certificates
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: failed to parse uri
Apr 28 15:26:00 walters-fcos afterburn[903]: Caused by: relative URL without a base
Apr 28 15:26:00 walters-fcos systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Apr 28 15:26:00 walters-fcos systemd[1]: [email protected]: Failed with result 'exit-code'.
Apr 28 15:26:00 walters-fcos systemd[1]: Failed to start Afterburn (SSH Keys).

Release 4.3.1

Release process

This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • export RELEASE_VER=x.y.z
  • export UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • make sure the upstream tag matches the local tag: git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
  • git checkout v${RELEASE_VER}
  • make sure the tag is what you intend to release; if so this will show an empty output: git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}
  • unset RELEASE_VER
  • unset UPSTREAM_REMOTE

Add support for sending ssh_authorized_keys info into the journald log

Tagging coreos/fedora-coreos-tracker#279 issue for more visibility here.

In order to solve the above issue, there's one bullet point that talks about displaying information related to ssh_authorized_keys that were added by the afterburn process. After having some discussion around this topic(coreos/fedora-coreos-config#344 (review)), it appears that sending a journal entry makes more sense here. Ideally, we need to use https://github.com/lucab/libsystemd-rs library to support this feature.

But it looks like (lucab/libsystemd-rs#34) isn't present in the latest release. It would be really helpful if we cut a new release of that library to support this feature.

Feature request: systemd target to synchronize instances of [email protected]

Feature Request

Environment

What hardware/cloud provider/hypervisor is being used to run Afterburn?

libvirt running Fedora CoreOS when this was brought up - applies to all platforms though.

Desired Feature

Add a target afterburn-sshkeys.target, and have the [email protected] template specify RequiredBy=afterburn-sshkeys.target in the [Install] section. This would allow other units to synchronize around any instance of afterburn-sshkeys@, not only for a given user (e.g. [email protected], see coreos/fedora-coreos-config#344 (comment)).

First need to test/investigate manually if this approach works - filing as an initial suggestion, and potentially can investigate other ideas for the synchronization here if this does not work.

Drop merging of authorized_keys.d into authorized_keys

Feature Request

Environment

Any

Desired Feature

At present, coreos-metadata --ssh-keys implicitly assumes that update-ssh-keys is distributed as part of the OS. That is, it assumes that SSH keys should be written into ~/.ssh/authorized_keys.d/<file>, and then all files in authorized_keys.d should be merged into authorized_keys. On distros without update-ssh-keys, this model doesn't immediately cause problems: coreos-metadata writes ~/.ssh/authorized_keys.d/coreos-metadata and then performs the sync via an embedded copy of the update-ssh-keys library. However, if the user updates authorized_keys directly and reboots, and coreos-metadata --ssh-keys runs again, it will clobber changes to authorized_keys. Instead, the user is expected to put their keys in e.g. ~/.ssh/authorized_keys.d/my-keys, but then doesn't have a tool to perform the synchronization.

Ultimately, update-ssh-keys is a hack for supporting authorized_keys.d directories without any cooperation from sshd, and should be dropped. It changes the workflow of updating SSH keys and isn't used on other distros; Fedora CoreOS and Red Hat CoreOS currently do not intend to ship it. Instead, in the short term, Fedora CoreOS will have special-case support for authorized_keys.d files written by individual tools, and in the medium term it will add generic support for the key directory (coreos/fedora-coreos-tracker#139 (comment)).

Modify coreos-metadata as follows:

• After writing the authorized_keys.d fragment file, don't merge the fragment file into authorized_keys except on CL. Probably gate this on a compile-time flag.
• Document that coreos-metadata --ssh-keys will only work on distros that include .ssh/authorized_keys.d/coreos-metadata in sshd's AuthorizedKeysFile, or that otherwise arrange to read all keyfiles in .ssh/authorized_keys.d.

Release 4.3.0

Release process

This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • export RELEASE_VER=x.y.z
  • export UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • make sure the upstream tag matches the local tag: git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
  • git checkout v${RELEASE_VER}
  • make sure the tag is what you intend to release; if so this will show an empty output: git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}
  • unset RELEASE_VER
  • unset UPSTREAM_REMOTE

Release 4.4.1

Release process

This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • export RELEASE_VER=x.y.z
  • export UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • make sure the upstream tag matches the local tag: git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
  • git checkout v${RELEASE_VER}
  • make sure the tag is what you intend to release; if so this will show an empty output: git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}
  • unset RELEASE_VER
  • unset UPSTREAM_REMOTE

ibmcloud: add support for SSH keys

Feature Request

Environment

Instances on ibmcloud platform (i.e. IBM cloud VPC gen2)

Desired Feature

The afterburn-sshkeys service should work on this platform too, grabbing SSH keys from instance metadata and forwarding them to authorized keys.

Add Daemon Mode

Feature Request

Create a daemon mode for afterburn for clouds where having a reactive/scheduled pattern makes sense instead of oneshot runs.

Environment

TBD

Desired Feature

Allow specific providers to have a daemon mode which runs consistently and reacts to specific changes.

Other Information

@darkmuggle has done some initial plumbing work.

config: consider sourcing options from files

Afterburn is currently a one-short binary, entirely configured via command-line flags. This is on purpose to keep the complexity (both internally and externally) low.

Our service units are generally written to be easy to override via drop-ins, such as:

[Service]
Environment=AFTERBURN_OPT_PROVIDER="--provider foo"

However, if we start increasing the design complexity (e.g. with a daemon mode, #228), we should consider sourcing file-based configurations.

This would be ideally done via overlayed directories with dropins, plus a simple and standard format (e.g. TOML).

aws: add support for metadata endpoint v2 (IMDSv2)

AWS recently announced a new generation of their metadata endpoint, called IMDSv2.

It is a session-based service, meant to plug a few holes in the v1 design, most notably SSRF.
See the announcement at https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/.

For the moment both v1 and v2 are active by default, and instances can be configured to disable the v1 endpoint.

In the medium term, Afterburn should move to v2 logic.

Release 4.5.0

Release process

This project uses cargo-release in order to prepare new releases, tag and sign the relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • RELEASE_VER=x.y.z
  • UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git checkout v${RELEASE_VER}
  • verify that grep "^version = \"${RELEASE_VER}\"$" Cargo.toml produces output
  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}

Azure: doesn't set hostname

Bug

Afterburn does not set the hostname on Azure.

Operating System Version

RHCOS

Afterburn Version

master

Environment

Azure

Expected Behavior

/etc/hostname contains the machine name defined on Azure

Actual Behavior

no file

Other Information

curl http://169.254.169.254/metadata/instance/compute/name?api-version=2017-08-01&format=text gives you the hostname

file location should be /etc/hostname

Currently Azure provider returns None:

afterburn/src/providers/azure/mod.rs

Lines 413 to 415 in 7db2aff

	fn hostname(&self) -> Result<Option<String>> {
	Ok(None)
	}

docs: mention platform naming skew

Afterburn usually follows Ignition platform naming convention, in order to auto-detect the platform.

However there are several platforms where the metadata component is only optional. On those platforms, in case of failures, Afterburn cannot distinguish a known-good state (i.e. this infra lacks the metadata component) from a failed state (i.e. network or logic error).

As such, Afterburn knows about the following platform which are more specific than the Ignition ones, cannot be auto-detected, and require user-provided hints (as cmdline flags):

• cloudstack-metadata
• cloudstack-configdrive
• openstack-metadata
• vagrant-virtualbox

Azure: Failed to start CoreOS Metadata Agent

coreos-metadata.service does not start on Core OS in Azure.

Operating System Version

> cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1800.5.0
VERSION_ID=1800.5.0
BUILD_ID=2018-07-28-2250
PRETTY_NAME="Container Linux by CoreOS 1800.5.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

coreos-metadata Version

Environment

Azure VM

Expected Behavior

coreos-metadata.service starts

Actual Behavior

coreos-metadata.service fails

Other Information

> sudo journalctl -u coreos-metadata.service
-- Logs begin at Thu 2018-08-02 14:43:17 UTC, end at Thu 2018-08-02 15:03:11 UTC. --
Aug 02 14:47:20 teamcity systemd[1]: Starting CoreOS Metadata Agent...
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.742 INFO Fetching http://168.63.129.16/?comp=versions: Attempt #1
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.765 INFO Fetch successful
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.765 INFO Fetching http://168.63.129.16/machine/?comp=goalstate: Attempt #1
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.769 INFO Fetch successful
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.971 INFO Fetching http://168.63.129.16/machine/aa786c20-80a9-4a09-8936-2104c27e6e5c/7f691c1e%2D37bf%2D4b9a%2D8c90%2D010bcc7c4277.%5Fteamcity?comp=certificates&incarnation=2>
Aug 02 14:47:20 teamcity coreos-metadata[1503]: Aug 02 14:47:20.979 INFO Fetch successful
Aug 02 14:47:20 teamcity coreos-metadata[1503]: thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /build/amd64-usr/var/tmp/portage/dev-libs/rustlib-1.26.0/work/rustc-1.26.0-src/src/libcore/option.rs:335:21
Aug 02 14:47:21 teamcity systemd[1]: coreos-metadata.service: Main process exited, code=exited, status=101/n/a
Aug 02 14:47:21 teamcity systemd[1]: coreos-metadata.service: Failed with result 'exit-code'.
Aug 02 14:47:21 teamcity systemd[1]: Failed to start CoreOS Metadata Agent.

> cat /usr/lib/systemd/system/coreos-metadata.service
[Unit]
Description=CoreOS Metadata Agent

[Service]
Type=oneshot
Environment=COREOS_METADATA_OPT_PROVIDER=--cmdline
ExecStart=/usr/bin/coreos-metadata ${COREOS_METADATA_OPT_PROVIDER} --attributes=/run/metadata/coreos

[Install]
RequiredBy=metadata.target

dracut: run hostname service on aliyun

AliCloud havn't DHCP for set hostname when firstboot, could you add condition for aliyun in afterburn-hostname.service ?
@lucab

ibmcloud: split Classic logic to a dedicated platform

Followup to coreos/coreos-assembler#1119 (comment).

We decided to go in a different direction and move Classic logic to a dedicated platform ID ibmcloud-classic.

openstack: check and set CAP_SYS_ADMIN before using a config-drive

Mounting a config-drive requires CAP_SYS_ADMIN in the thread invoking the mount syscall. This is typically true when running coreos-metadata as the root user, but it would be better to make this explicit and only requiring the specific capabilities it needs.

cargo: consider setting panic=abort in the release profile

Feature Request

Environment

All.

Desired Feature

Building release binaries with panic=abort reduces their size by a considerable amount, which would make including coreos-metadata in the initramfs more palatable. However, it might interfere with some cleanup behavior we are currently relying on with panic=unwind.

Other Information

See #105 for the original discussion.

Wrong openstack provider

Bug

Afterburn with --cmdline argument does not work on OpenStack

Operating System Version

fedora-coreos-31.20200210.3.0

Afterburn Version

Afterburn 4.1.3

Environment

OpenStack

Expected Behavior

/usr/bin/afterburn --cmdline--attributes=/run/metadata/afterburn
Mar 07 23:20:33.942 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #1
Mar 07 23:20:34.412 INFO Fetch successful
Mar 07 23:20:34.412 INFO Fetching http://169.254.169.254/latest/meta-data/instance-id: Attempt #1
Mar 07 23:20:34.503 INFO Fetch successful
Mar 07 23:20:34.504 INFO Fetching http://169.254.169.254/latest/meta-data/local-ipv4: Attempt #1
Mar 07 23:20:34.592 INFO Fetch successful
Mar 07 23:20:34.592 INFO Fetching http://169.254.169.254/latest/meta-data/public-ipv4: Attempt #1
Mar 07 23:20:34.676 INFO Fetch successful

Actual Behavior

/usr/bin/afterburn --cmdline
Error: fetching metadata from provider
Caused by: unknown provider 'openstack'

Reproduction Steps

cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt1)/ostree/fedora-coreos-19190477fad0e60d605a623b86e06bb92aa318b6b79f78696b06f68f262ad5d6/vmlinuz-5.4.17-200.fc31.x86_64 mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=ttyS0,115200n8 ignition.firstboot rd.neednet=1 ip=dhcp ostree=/ostree/boot.1/fedora-coreos/19190477fad0e60d605a623b86e06bb92aa318b6b79f78696b06f68f262ad5d6/0 ignition.platform.id=openstack

cloud provider reported by cmdline is openstack. Not openstack-metadata

Other Information

docs: record cloud-specific quirks

Cloud providers may have specific quirks and restrictions which could result in surprising behaviors of coreos-metadata or which could require workarounds in the codebase. Those should be documented in the repository, so that we have an explicit note to reference.

The first example of this has been reported at coreos/bugs#2468 (comment).
Azure may (or may not, depending on agent version and settings) prevent non-root users to reach the metadata endpoint for security reasons. This affects how the coreos-metadata service can be run.

Add support for sending ssh_authorized_keys info into the journald log

Tagging coreos/fedora-coreos-tracker#279 issue for more visibility here.

In order to solve the above issue, there's one bullet point that talks about displaying information related to ssh_authorized_keys that were added by the afterburn process. After having some discussion around this topic(coreos/fedora-coreos-config#344 (review)), it appears that sending a journal entry makes more sense here. Ideally, we need to use https://github.com/lucab/libsystemd-rs library to support this feature.

But it looks like (lucab/libsystemd-rs#34) isn't present in the latest release. It would be really helpful if we cut a new release of that library to support this feature.

platform: support quad-NIC servers on Packet (n2)

Feature Request

Environment

What hardware/cloud provider/hypervisor is being used to run Afterburn?

Packet, specifically the n2 servers

Desired Feature

At

afterburn/src/providers/packet/mod.rs

Line 248 in d56ad64

// According to the folks from packet, all the addresses given to us in the

the Packet provider properly handles dual-NIC systems, but does not currently support the quad-NIC n2 system type.

Other Information

Noted by Kinvolk in the course of an integration.

providers: add a new "azurestack" platform (client logic for AzureStackHub)

Afterburn fails completely due to 500 errors on the metadata source. With coreos/ignition@0c0ec63 I was able to boot on AzureStack.

A complete different issue:

s)...[   32.768570] NetworkManager[568]: <info>  [1593115470.6525] dhcp4 (eth0): option private_245          => 'a8:3f:81:10'

And then:

[   64.908820] afterburn[658]: Jun 25 19:57:52.985 WARN Failed to get fabric address from DHCP: maximum number of retries (60) reached
[   64.988395] afterburn[658]: Jun 25 19:57:52.986 INFO Using fallback address
[   65.033307] afterburn[658]: Jun 25 19:57:52.986 INFO Fetching http://168.63.129.16/?comp=versions: Attempt #1
^M[     *] A start job is running for Afterburn Hostname (52s / no limit)
[   65.566088] afterburn[658]: Jun 25 19:57:53.643 INFO Fetch successful
[   65.621959] afterburn[658]: Jun 25 19:57:53.643 INFO Fetching http://168.63.129.16/machine/?comp=goalstate: Attempt #1
[   65.698749] afterburn[658]: Jun 25 19:57:53.651 INFO Fetch successful
[   65.747770] afterburn[658]: Jun 25 19:57:53.659 INFO Fetching http://169.254.169.254/metadata/instance/compute/name?api-version=2017-08-01&format=text: Attempt #1
[   65.942651] afterburn[658]: Jun 25 19:57:53.674 INFO Failed to fetch: 500 Internal Server Error

And ending with:

Displaying logs from failed units: afterburn-hostname.service
-- Logs begin at Thu 2020-06-25 20:04:16 UTC, end at Thu 2020-06-25 20:05:59 UTC. --
Jun 25 20:05:51 afterburn[655]: Jun 25 20:05:51.338 INFO Failed to fetch: 500 Internal Server Error
Jun 25 20:05:51 afterburn[655]: Error: failed to run
Jun 25 20:05:51 afterburn[655]: Caused by: writing hostname
Jun 25 20:05:51 afterburn[655]: Caused by: failed to get hostname
Jun 25 20:05:51 afterburn[655]: Caused by: maximum number of retries (10) reached
Jun 25 20:05:51 afterburn[655]: Caused by: failed to fetch: 500 Internal Server Error
Jun 25 20:05:51 systemd[1]: afterburn-hostname.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 20:05:51 systemd[1]: afterburn-hostname.service: Failed with result 'exit-code'.
Jun 25 20:05:51 systemd[1]: Failed to start Afterburn Hostname.

Release 4.4.0

Release process

This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • export RELEASE_VER=x.y.z
  • export UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • make sure the upstream tag matches the local tag: git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
  • git checkout v${RELEASE_VER}
  • make sure the tag is what you intend to release; if so this will show an empty output: git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}
  • unset RELEASE_VER
  • unset UPSTREAM_REMOTE

Update AWS metadata version

Feature Request

Environment

AWS

Desired Feature

afterburn/src/providers/aws/mod.rs

Line 65 in d8f74d5

const URL: &str = "http://169.254.169.254/2009-04-04";

references a fairly old AWS metadata version. To avoid confusion during support cases, consider using 2016-09-02 which is the most recent version specified at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html.

Release 4.4.2

Release process

This project uses cargo-release in order to prepare new releases, tag and sign the relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • RELEASE_VER=x.y.z
  • UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git checkout v${RELEASE_VER}
  • verify that grep "^version = \"${RELEASE_VER}\"$" Cargo.toml produces output
  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}

docs/release-guide: move to checklist ticket template

On each release we publish a .crate file to crates.io, which is then used by distribution to package coreos-metadata. Its SHA256 is recorded at publish time in the index, eg
rust-lang/crates.io-index@e0bc5cf.

In order to improve the integrity chain and to make manual checks easier, we should also record the SHA256 of the .crate file on local filesystem (before upload) and publish it in the release notes.

As per discussion with @ashcrow.

Read lease files from dhclient and NetworkManager on Azure and CloudStack

RHCOS/FCOS uses NetworkManager presently. So the logic to look for option 245 in the systemd-networkd lease file would need to be adapted to read from those lease files (/var/lib/NetworkManager on RHEL7, /var/lib/dhclient on FCOS and likely RHEL8). Might be worth investigating if NM's D-Bus API exposes this so we don't have to learn dhclient's syntax.

Though note that this is not a blocker for getting those platforms on Azure if we're comfortable relying exclusively on the fallback endpoint (#145) until this is implemented.

Related: #130

vmware: virtual CPU is misdetected in some cases

Due to a bug in the underlying library, some virtual CPU in vmware may be missed by the detection logic, resulting in boot failure when trying to use custom networking cmdline arguments. Bumping vmw_backdoor to 0.1.3 should fix this.

Ref: https://github.com/lucab/vmw_backdoor-rs/issues/3

azure: boot reporting fails if boot diagnostic is enabled after instance creation

If an Azure instance is created with "Boot diagnostics: off" and then later on while booting the setting is turned to "on", Afterburn boot check-in will fail with an output like this:

Jul 21 13:21:25.449 INFO Fetching http://168.63.129.16/?comp=versions: Attempt #1
Jul 21 13:21:25.455 INFO Fetch successful
Jul 21 13:21:25.462 INFO Fetching http://168.63.129.16/machine/?comp=goalstate: Attempt #1
Jul 21 13:21:25.468 INFO Fetch successful
Jul 21 13:21:25.506 INFO Posting http://168.63.129.16/machine/?comp=health: Attempt #1
Jul 21 13:21:26.520 INFO Posting http://168.63.129.16/machine/?comp=health: Attempt #2
...
Jul 21 13:22:02.653 INFO Posting http://168.63.129.16/machine/?comp=health: Attempt #10
Error: failed to run
Caused by: checking-in instance boot to cloud provider
Caused by: maximum number of retries (10) reached
Caused by: POST failed: 410 Gone

This seems unrelated to #459, i.e. it isn't a timeout issue.

I played a guessing game on the root cause for this, and via empirical experimentation it seems to be due to a mismatch in GoalStateIncarnation.

In particular, Azure documentation tells us to hardcode <GoalStateIncarnation>1</GoalStateIncarnation> in requests.
However, flipping the setting seems to increment the machine incarnation, and querying the goalstate endpoint after that returns an XML document with <Incarnation>2</Incarnation>.
Manually tweaking the request to have a matching Incarnation <-> GoalStateIncarnation seems to fix the failure, even after further tweaking settings (manually confirmed with incarnations up to 4).

ibmcloud: make auto-detection logic concurrent

This ticket is a work item for the following TODO note:

afterburn/src/providers/ibmcloud/mod.rs

Lines 23 to 24 in dba86a4

	// TODO(lucab): make auto-detection parallel (and cancellable), otherwise
	// Classic may suffer some noticeable delay.

This is only relevant for ibmcloud Classic users, resulting in a unnecessary delay of a few seconds.

Cut a new release

There has roughly been the answer to life, the universe, and everything commits since 4.1.0 was released. Recently #224 merged and we'd like to start utilizing it. Could we get a new release cut for afterburn?

Support AWS IMDSv2

Feature Request

Environment

AWS

Desired Feature

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html describes IMDSv2, which is a 2-step process to retrieving instance metadata. Enforcing IMDSv2 is beneficial for preventing attacks that trick an application into making requests to the IMDS and returning data such as the instance IAM credentials.

A node can be launched to only support IMDSv2, so I suggest making afterburn compliant with this to enable the v2-only switch to be activated.

Other Information

More details about IMDSv2:

https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Remove index accesses from Azure provider

Bug

coreos-metadata Version

1.0.0+

Environment

Azure

Expected Behavior

The Azure provider should not use index accesses, as they can cause panics.

Actual Behavior

The Azure provider uses index accesses here and here.

docs: add brief intro for platforms

When I looked at the providers, there are cloudstack, openstack, azurestack(in the near future) confusing me in the beginning. Would help with brief intro or website links for platforms if anyone feels the same way.

Consider making [email protected] ConditionFirstBoot=true

Issue Report

Feature Request

Environment

AWS

Desired Feature

After SSH keys have been set, don't continue to set them on every boot. With the current behavior, keys that are deleted by the user can come back later, which is surprising in the Ignition world.

Alternatively, document how to disable the service.

Other Information

See coreos/bugs#1933 (comment) for context.

Release 4.3.2

Release process

This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • export RELEASE_VER=x.y.z
  • export UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • make sure the upstream tag matches the local tag: git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
  • git checkout v${RELEASE_VER}
  • make sure the tag is what you intend to release; if so this will show an empty output: git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}
  • unset RELEASE_VER
  • unset UPSTREAM_REMOTE

Consider renaming coreos-metadata

Feature Request

Environment

Any

Desired Feature

Consider renaming coreos-metadata. It seems that the project is drifting toward becoming a minimal, generic cloud agent, in which case the name is not very apt. In particular:

• The project will (hopefully!) be useful on other distros, so coreos isn't great.
• It does more than collect metadata, so metadata isn't great either.
• Given the above, COREOS_ is not great in output variable names. Changing it would be very user-visible and may not be feasible. We could output both old and new names for compatibility, but it's not clear that we could ever stop outputting the old names. We could drop the old names on non-CL distros, but users coming from CL will presumably have unit files and scripts that depend on the old names, and conversion tools (coreos/fedora-coreos-tracker#48) may not be able to catch every use of them.

For a new project name, we could go the descriptive route with "platform-agent" or the like, but that doesn't seem very inspired. Maybe an arbitrary name would be better.

azure: log a warning on network failure if non-root

This is related to #94. If we experience a network failure on Azure while trying to reach the metadata endpoint and the process is not running as root, we should log a more informative message about possible Azure-specific local firewalling,

Release 4.3.3

Release process

This project uses cargo-release in order to prepare new releases, tag and sign relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • export RELEASE_VER=x.y.z
  • export UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • make sure the upstream tag matches the local tag: git fetch --tags --verbose ${UPSTREAM_REMOTE} 2>&1 | grep ${RELEASE_VER}
  • git checkout v${RELEASE_VER}
  • make sure the tag is what you intend to release; if so this will show an empty output: git diff release-${RELEASE_VER}~1 v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}
  • unset RELEASE_VER
  • unset UPSTREAM_REMOTE

platform: add support for vultr

This is a feature request to add support for a new cloud platform, Vultr.

Their metadata API is described at https://www.vultr.com/metadata/. We are mostly interested in attributes and SSH pubkeys at this point.

Overall platform discussion for Fedora Coreos is ongoing at coreos/fedora-coreos-tracker#355 (which covers ID naming too).

RPM lint warning in dependency {users-0.7.0 via update-ssh-keys-0.3.0): call to setgroups before setuid

Bug

When building coreos-3.0.1, rpmlint reports call to setgroups before setuid.

This appears to be in crate users-0.7.0, a dependency of direct dependency crate update-ssh-keys-0.3.0. Some changes in this area have been integrated in users-0.8.0, currently researching whether those eliminate the warning.

The warning may not indicate an actual problem, but it would be helpful to eliminate the warning if a fix is available through a newer version of a dependency.

Operating System Version

OpenSUSE Tumbleweed 20181018-0 x86_64

coreos-metadata Version

coreos-metadata-3.0.1

Environment

Open Build Service (OBS) osc build local.

Local RPM lint warning at build time, no error running executable. Not running on any cloud provider.

Expected Behavior

Build executable that passes standard RPM lint checks.

Actual Behavior

RPMLINT report:
===============
coreos-metadata.x86_64:
W: missing-call-to-setgroups-before-setuid /usr/bin/coreos-metadata
This executable is calling setuid and setgid without setgroups or initgroups.
There is a high probability this means it didn't relinquish all groups, and
this would be a potential security issue to be fixed. Seek POS36-C on the web
for details about the problem.

Reproduction Steps

1. Routine osc build to build local package
2. RPM lint warning above displayed.

Other Information

Occurrences of setgroups are all in crate libc-0.2.43.
Occurrences of setuid are in crate libc-0.2.43 and crate users-0.7.0.

The rpmlint warning is not specific to a particular call site. Dependency crate users-0.7.0 has potentially relevant code here: https://github.com/ogham/rust-users/blob/v0.7.0/src/switch.rs#L25

Commits between crate users-0.7.0 (pinned) and crate users-0.8.0 (available) suggest some work may have been done in this area: ogham/rust-users@v0.7.0...master
I will test a build with crate users-0.8.0 and report back here.

providers: Fetch instance-type metadata

Feature Request

Since in fedora-coreos-pinger, we need to fetch the instance type of cloud providers and I am trying to avoid too much duplicate code between fedora-coreos-pinger, Afterburn and Zincati, it would be great if we could fetch cloud instance type here.

Environment

Apply to all cloud providers.

Desired Feature

Fetch cloud instance type for all providers.

Other Information

Link to related fedora-coreos-pinger issue: coreos/fedora-coreos-pinger#30

azure: keep retrying requests a bit longer

On Azure, Afterburn currently sends requests to the metadata/wireserver endpoint with some mildly aggressive retrying logic, which on failures results in sending about 10 requests in ~1 minute.

According to Azure/azure-sdk-for-go#1848, the WireServer routinely goes down for maintenance with a p90 downtime up to 3 minutes (!) for each restart event.

As such, in order to be more lenient toward transient platform issues, we should relax our deadline for retries as to cover extended downtime up to 4 or 5 minutes.

providers: add support for Alibaba Cloud

Fedora CoreOS is targeting Alibaba Cloud as a new platform: coreos/fedora-coreos-tracker#276.

Afterburn should also support that, with the platform ID aliyun.

This cloud platform has a metadata endpoint, documented at https://www.alibabacloud.com/help/doc-detail/49122.htm.

Add mechanism for reporting instance health to cloud platforms

Feature Request

Environment

Azure, Packet

Desired Feature

Consider adding a command-line option or mode which reports a successful boot to the cloud platform and then exits. On platforms that don't require health reporting, the command would silently succeed.

Other Information

The idea is that the command would run as a systemd oneshot after greenboot is happy.

See also coreos/fedora-coreos-tracker#69.

qemu: implement boot-time checkin (via ovirt-guest-agent protocol)

Currently if FCOS fails in very early boot (e.g. in the bootloader or kernel, before switching to the initramfs), it's...hard to detect consistently. For example, we have a test for Secure Boot, but it turns out that if the kernel fails to verify then...coreos-assembler today hangs for a really long time.

We can scrape the serial console in most cases, but we configure the Live ISO not to log to a serial console...so we'd end up having instead to do something like OpenQA and do image recognition on the graphics console 😢

Now we debated this somewhat in
coreos/ignition-dracut#170
and I argued strongly that the most important thing was to cover the "failure in initramfs" case, and we could support the "stream journal in general" by injecting Ignition.

In retrospect...I think I was wrong. It would be extremely useful for us to stream the journal starting from the initramfs at least by default on qemu.

In particular, what we really want is some sort of message from the VM that it has entered the initramfs, but before we start processing Ignition. If we're doing things like reprovisioning the rootfs, it becomes difficult to define a precise "timeout bound". But I think we can e.g. reliably time out after something quite low (like 10 seconds) if we haven't seen the "entered initramfs" message.

So here's my proposal:

• Add a variant of coreos/ignition-dracut#170 that streams the journal to a differently named virtio channel
• Update coreos-assembler to use it
• Eventually drop the old "only dump on failure" channel and code

sshkeys: do not fail if no keys are deployed

Bug

Operating System Version

30.20191014.1 (CoreOS preview)
(happened in previous versions as well)

Afterburn Version

4.1.3

Environment

aws

Expected Behavior

no failing systemd unit after boot

Actual Behavior

● [email protected] - Afterburn (SSH Keys)
   Loaded: loaded (/usr/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-11-28 10:25:13 UTC; 8min ago
  Process: 934 ExecStart=/usr/bin/afterburn ${AFTERBURN_OPT_PROVIDER} --ssh-keys=core (code=exited, status=1/FAILURE)
 Main PID: 934 (code=exited, status=1/FAILURE)

Nov 28 10:25:12 ip-10-100-20-105.eu-west-1.compute.internal systemd[1]: Starting Afterburn (SSH Keys)...
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Nov 28 10:25:13.314 INFO Fetching http://169.254.169.254/2009-04-04/meta-data/public-keys: Attempt #1
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Nov 28 10:25:13.327 INFO Fetch failed with 404: resource not found
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Error: writing ssh keys
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Caused by: failed to open "/var/home/core/.ssh/authorized_keys.d" for syncing
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal afterburn[934]: Caused by: No such file or directory (os error 2)
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal systemd[1]: [email protected]: Failed with result 'exit-code'.
Nov 28 10:25:13 ip-10-100-20-105.eu-west-1.compute.internal systemd[1]: Failed to start Afterburn (SSH Keys).

Reproduction Steps

1. start new ec2 instance without providing ssh keys
2. ssh into instance and check for failed systemd units

Other Information

Release 4.5.1

Release process

This project uses cargo-release in order to prepare new releases, tag and sign the relevant git commit, and publish the resulting artifacts to crates.io.
The release process follows the usual PR-and-review flow, allowing an external reviewer to have a final check before publishing.

In order to ease downstream packaging of Rust binaries, an archive of vendored dependencies is also provided (only relevant for offline builds).

Requirements

This guide requires:

• a web browser (and network connectivity)
• git
• GPG setup and personal key for signing
• cargo (suggested: latest stable toolchain from rustup)
• cargo-release (suggested: cargo install -f cargo-release)
• A verified account on crates.io
• Write access to this GitHub project
• Upload access to this project on GitHub and crates.io

Release checklist

These steps show how to release version x.y.z on the origin remote (this can be checked via git remote -av).
Push access to the upstream repository is required in order to publish the new tag and the PR branch.

• make sure the project is clean and prepare the environment:
  • cargo test
  • cargo clean
  • git clean -fd
  • RELEASE_VER=x.y.z
  • UPSTREAM_REMOTE=origin

⚠️: UPSTREAM_REMOTE should reference the locally configured remote that points to the upstream git repository.

• create release commits on a dedicated branch and tag it:

  • git checkout -b release-${RELEASE_VER}
  • cargo release (and confirm the version when prompted)

• open a PR for this release:

  • git push ${UPSTREAM_REMOTE} release-${RELEASE_VER}
  • open a web browser and create a PR for the branch above
  • make sure the resulting PR contains exactly two commits
  • in the PR body, write a short changelog with relevant changes since last release

• get the PR reviewed, approved and merged

• publish the artifacts (tag and crate):

  • git checkout v${RELEASE_VER}
  • verify that grep "^version = \"${RELEASE_VER}\"$" Cargo.toml produces output
  • git push ${UPSTREAM_REMOTE} v${RELEASE_VER}
  • cargo publish

• assemble vendor archive:

  • cargo vendor
  • tar -czf target/afterburn-${RELEASE_VER}-vendor.tar.gz vendor

• publish this release on GitHub:

  • open a web browser and create a GitHub Release for the tag above
  • write a short changelog (i.e. re-use the PR content)
  • upload target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • record digests of local artifacts:
    • sha256sum target/package/afterburn-${RELEASE_VER}.crate
    • sha256sum target/afterburn-${RELEASE_VER}-vendor.tar.gz
  • publish release

• clean up:

  • cargo clean
  • rm -rf vendor
  • git checkout master
  • git pull ${UPSTREAM_REMOTE} master
  • git push ${UPSTREAM_REMOTE} :release-${RELEASE_VER}
  • git branch -d release-${RELEASE_VER}