amazonlinux / amazon-ec2-utils Goto Github PK

There are situations where you require the block device mapping names to exist before you can mount the root volume in the early-boot process, and currently this package cannot be used there because it would require copying in Python.

One solution to this problem would be to rewrite the script to use nvme-cli >= 1.13, where the block device name can be grabbed with nvme amzn id-ctrl /dev/nvme0n1 | grep '^bdev' | awk '{ sub("/dev", ""); print $3 }' (sometimes the block device name begins with /dev, and this script is expected to strip that off). A more complicated Bash-only version could also be written to avoid needing to depend on grep and awk in the compact initrd/initramfs.

This solution works for EL8 and EL9 out of the box because both ship an nvme-cli that is new enough, but AL2023 and earlier ship too old of an nvme-cli for this to work. Since this repo primarily supports AL and AL does not typically support folks using non-default filesystems like ZFS, I figured contributing a PR that made this change would not be terribly useful. Since we need it (and in fact have already had this in place for years), these changes will exist, and I'm happy to strategize some way to make it work more broadly in this package (maybe different RPM specs? an RPM spec that uses macros to choose a strategy based on the OS?)

We spent quite some time debugging a storage test regression in Fedora rawhide which essentially breaks scsi_debug and other devices, but only on RedHat's/Fedora's Testing Farm infrastructure -- which is essentially AWS EC2 machines with an API.

Latest Fedora rawhide instances now have amazon-ec2-utils-2.2.0-2.fc41.noarch (which got introduced into Fedora very recently), which ships /usr/lib/udev/rules.d/70-ec2-nvme-devices.rules with

KERNEL=="nvme[0-9]*n[0-9]*",        ENV{DEVTYPE}=="disk",      ATTRS{model}=="Amazon Elastic Block Store", PROGRAM="/usr/sbin/ebsnvme-id -u /dev/%k", SYMLINK+="%c"
KERNEL=="nvme[0-9]*n[0-9]*p[0-9]*", ENV{DEVTYPE}=="partition", ATTRS{model}=="Amazon Elastic Block Store", PROGRAM="/usr/sbin/ebsnvme-id -u /dev/%k", SYMLINK+="%c%n"

These instances have an NVME block device, and these rules cause the following symlinks to be created:

lrwxrwxrwx. 1 root root 7 May 29 03:52 /dev/sda1 -> nvme0n1
lrwxrwxrwx. 1 root root 9 May 29 03:52 /dev/sda11 -> nvme0n1p1
lrwxrwxrwx. 1 root root 9 May 29 03:52 /dev/sda12 -> nvme0n1p2
lrwxrwxrwx. 1 root root 9 May 29 03:52 /dev/sda13 -> nvme0n1p3
lrwxrwxrwx. 1 root root 9 May 29 03:52 /dev/sda14 -> nvme0n1p4

This is problematic in multiple ways:

• Pretending that these are SCSI drives tramples on the kernel's namespace. udev symlinks should never create names which the kernel uses.
• "nvme0n1" is the raw block device, not a partition. So it's very confusing to name it "sda1", it should be "sda". Likewise, the first partition should be "sda1", not "sda11".

If then a real sda comes along (e.g. with modprobe scsi_debug), this will create an actual /dev/sda, but then it's impossible to create/see partitions on that, as the sda1 etc. names are already taken.

This is most easily reproduced with

# /usr/sbin/ebsnvme-id -u /dev/nvme0n1
sda1

Curiously, it also does that for a partition:

# /usr/sbin/ebsnvme-id -u /dev/nvme0n1p2
sda1

that explains how the second udev rule can even work -- but this is really hackish!

My recommendation as former udev co-upstream is to just entirely remove these rules. They are not helpful, confusing, and break stuff. You can of course create symlinks in subdirs of /dev all you like, but please don't collide with kernel names.

Thanks!

Hi. After a yum upgrade triggered by security updates, my AWS Linux 2 (amzn2-ami-ecs-hvm-2.0.20191212-x86_64-ebs) instance's containers failed to start.
I traced the problem to a failure of udev rules to create an alias for EBS volumes that appear as NVMe drives.
Which then led me to find the ebsnvme-id tool installed in this yum package is failing.

https://github.com/aws/amazon-ec2-utils/blob/3cd27a273407d4aeadc9b330a463686c0e71f4d5/ebsnvme-id#L167

That line, to my knowledge is invalid syntax. Python 2.7.18 is the default python on Amazon Linux2 AMI and its standard stream of yum updates. If one tries to run this ebsnvme-id tool by hand, this is the result:

[myprompt]# /sbin/ebsnvme-id
  File "/sbin/ebsnvme-id", line 167
    print(err, file=sys.stderr)
                   ^
SyntaxError: invalid syntax

And when one runs it like the official AMI installed udev rule /etc/udev/rules.d/70-ec2-nvme-devices.rules, it also fails, and therefore the udev rule fails, so the alias isn't made, leading to a cascade of volume failures

[myprompt]# /sbin/ebsnvme-id -u /dev/nvme1n1
  File "/sbin/ebsnvme-id", line 167
    print(err, file=sys.stderr)
                   ^
SyntaxError: invalid syntax

Referencing the official Python 2.7 docs https://docs.python.org/2/reference/simple_stmts.html#the-print-statement
the print statement does not support that syntax.

Only Python 3 supports a file= on the print() statement. And since Python 2.7.18 is the official default python of the AWS Linux 2 AMI, and the top of this python file #!/usr/bin/env python doesn't specific a version, that means this file is trying to use Python3 language features on Python2. Therefore, python won't run it, and a cascade of failures occur.

udev rule and python program ebsnvme-id are missing in package nvme-cli on Oracle Linux, this is causing proper device names mapping (e.g., /dev/sd*) not working when compared to other Linux OS running on AWS.

If rules are manually created together with python program /sbin/ebsnvme-id, they work properly on OEL

Hello,

The commit, 9b1ff7d, has introduced an rpmbuild error.

rpmbuild -bb --nodeps --define '__python /usr/libexec/platform-python' --define '_topdir /tmp/amazon-ec2-utils/rpmbuild' /tmp/amazon-ec2-utils/rpmbuild/SPECS/amazon-ec2-utils.spec
warning: bogus date in %changelog: Thu Jul 14 2021 Sai Harsha <[email protected]> 1.3.3
error: %changelog not in descending chronological order

fidisk and cfdisk can list the partion table properly but the lsblk won't list all the partitions, underlying was the /dev/xvda128 itself not created.

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /

Disk /dev/xvda: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 84B5F398-58C6-4F8A-9DB3-02258ADEBC84

Device Start End Sectors Size Type
/dev/xvda1 4096 16777182 16773087 8G Linux filesystem
/dev/xvda128 2048 4095 2048 1M BIOS boot

Partition table entries are not in disk order.

Looks like the Udev rules needs attention, not sure what should be done, please suggest.

ec2-metadata currently uses 169.254.169.254 unconditionally. In the case that an instance has disabled its IPv4 stack, it has no access to IMDS, even in environments that do provide a v6 endpoint.

With the availability of Amazon Linux 2022 in the amazonlinux GitHub organization, I'd like to propose changes to this repository to make it more consistent with the other packages including with Amazon Linux.

1. The package should move from aws to amazonlinux in GitHub. Moving a repository automatically creates redirects from the old location, so existing references will continue to work.
2. The existing master branch will be retired. A new 1.x branch will be created from this branch, and releases targeting Amazon Linux 2 will be made from this branch.
3. A new main branch will be created. This will be the default destination for pull requests and will be the branch from which 2.x tags, targeting Amazon Linux 2022 and other reasonably modern Linux distributions, will be created.

Any additional suggestions for structural or organizational changes to this repository are welcome at this time.

CC @fred-lefebvre @halfdime-code @stewartsmith

Hi Folks. I'm a relatively new EC2/AWS user, and I arrived here by way of wanting to get the latest version of your package. (The Amazon Linux 2 AMI has v0.1.2 of ec2-metadata installed, and I'd like to get the tags support you've added. In fact, I'm pretty surprised that this package is so out of date on Amazon's own OS!) Anyway, I want to install it on my instance but I do not see instructions in the readme, nor do I see a makefile. Thanks for your kind help!

#9 changed several of the nvme udev rules to only trigger on on "add" events. However, this causes the links to be lost on "change" events, which occur whenever the corresponding device is opened in read/write mode (e.g. when creating or mounting a filesystem).

In cases where IMDS is disabled or otherwise unreachable, ec2-metadata prints an error messages beginning with [ERROR] Could not get IMDSv2 token. to stdout. This should go to stderr instead in order to avoid causing confusion for tools trying to consume the output of this command.

Observed on the current master branch, as well as versions packaged in Debian and Amazon Linux 2.

By opening the nvme block devices in read/write mode, ebsnvme-id triggers a kernel "change" event, which translates to the device nodes corresponding to the partitions being deleted and recreated. Similarly, the sysfs files corresponding with the partitions are also deleted and recreated.

To reproduce:

1. Launch an EC2 instance with a partitioned secondary EBS volume attached (seen as /dev/nvme1n1 here):

    admin@ip-10-0-0-75:~$ lsblk 
    NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
    nvme0n1      259:0    0    8G  0 disk 
    ├─nvme0n1p1  259:1    0  7.9G  0 part /
    └─nvme0n1p15 259:2    0  127M  0 part /boot/efi
    nvme1n1      259:3    0    8G  0 disk 
    ├─nvme1n1p1  259:4    0  7.9G  0 part 
    ├─nvme1n1p14 259:5    0    3M  0 part 
    └─nvme1n1p15 259:6    0  124M  0 part 
   

2. Note the inode number of one of the partition device nodes in /dev, e.g:

    admin@ip-10-0-0-75:~$ stat --format="%i" /dev/nvme1n1p1
    566
   

3. Start udevadm monitor in a second session.

4. Run sudo /sbin/ebsnvme-id /dev/nvme1n1

5. Note the inode number of the partition device:

    admin@ip-10-0-0-75:~$ stat --format="%i" /dev/nvme1n1p1
    598
   

6. Note the output of udevadm monitor, which will indicate the child device removal and recreation similar to the following:

    KERNEL[5283.274948] remove   /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p1 (block)
    KERNEL[5283.275107] remove   /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p14 (block)
    KERNEL[5283.275215] remove   /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p15 (block)
    KERNEL[5283.279198] change   /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1 (block)
    KERNEL[5283.279340] add      /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p1 (block)
    KERNEL[5283.279420] add      /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p14 (block)
    KERNEL[5283.281014] add      /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p15 (block)
    UDEV  [5283.339467] remove   /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p15 (block)
    UDEV  [5283.345292] remove   /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p14 (block)
    UDEV  [5283.356890] remove   /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p1 (block)
    UDEV  [5283.410338] change   /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1 (block)
    UDEV  [5283.526077] add      /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p14 (block)
    UDEV  [5283.555895] add      /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p1 (block)
    UDEV  [5283.570766] add      /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1/nvme1n1p15 (block)
   

Similar behavior can be observed with respect to the files in the /sys/block/nvme1n1/nvme1n1p*/ directories.

Since the ebsnvme-id script is only supposed to be gathering metadata about volumes in a read-only manner, its invocation should not trigger a kernel change event.

This commit cf8c1a2 causes failures across a number of amis that we use, including:

• freebsd-12,13
• sles-12,15
• ubuntu-1604,1804
• rhel-6

Specifically, during instance initialization the script exits with this error:

/opt/aws/bin/ec2-metadata: /usr/bin/bash: bad interpreter: No such file or directory

Note the commit comment:

bash is in /usr/bin everywhere we care about. The /bin path is only there for backwards compatibility.

This is not true for the above images that my organization cares about; on these images /usr/bin/bash does not exist, and /bin/bash is not present as a backwards compatibility. What criteria do you use to determine if you 'care about' an OS?

Was this change made to fix any real problem in new OS images?

After building the latest RPM, on a centos 7 machine, I started receiving the following errors in the system log:

Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid ACTION operation
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid rule '/etc/udev/rules.d/70-ec2-nvme-devices.rules:8'
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid ACTION operation
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid rule '/etc/udev/rules.d/70-ec2-nvme-devices.rules:11'
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid ACTION operation
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid rule '/etc/udev/rules.d/70-ec2-nvme-devices.rules:12'
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid ACTION operation
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid rule '/etc/udev/rules.d/70-ec2-nvme-devices.rules:15'
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid ACTION operation
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid rule '/etc/udev/rules.d/70-ec2-nvme-devices.rules:16'
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid ACTION operation
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid rule '/etc/udev/rules.d/70-ec2-nvme-devices.rules:19'
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid ACTION operation
Nov 23 15:13:44 ip-10-17-180-126 systemd-udevd: invalid rule '/etc/udev/rules.d/70-ec2-nvme-devices.rules:22'

I believe the syntax error is due to ACTION="add", introduced in #9, which should use double equals.