Coder Social home page Coder Social logo

ec2-metadata's Introduction

ec2-metadata

image

image

image

image

pre-commit

An easy interface to query the EC2 metadata API (version 2), with caching.

A quick example:

>>> from ec2_metadata import ec2_metadata
>>> print(ec2_metadata.region)
us-east-1
>>> print(ec2_metadata.instance_id)
i-123456

Working on a Django project? Improve your skills with one of my books.


Installation

Use pip:

python -m pip install ec2-metadata

Python 3.8 to 3.12 supported.

Why?

boto came with a utility function to retrieve the instance metadata as a lazy loading dictionary, boto.utils.get_instance_metadata, but this has not been ported to boto3, as per this issue. I thought that rather than building a new version inside boto3 it would work well as a standalone library.

Instance Metadata Service Version 2

In November 2019, AWS released version 2 of the instance metadata service. It's more secure against Server Side Request Forgery (SSRF) attacks.

ec2-metadata now uses it exclusively. You can therefore consider disabling version 1, as per AWS' guide.

Note: Instance Metadata Service v2 has a default IP hop limit of 1. This can mean that you can see requests.exceptions.ReadTimeout errors from within Docker containers. To solve this, reconfigure your EC2 instance’s metadata options to allow three hops with aws ec2 modify-instance-metadata-options__:

aws ec2 modify-instance-metadata-options  --instance-id <instance-id> --http-put-response-hop-limit 3

API

EC2Metadata(session=None)

A container that represents the data available on the EC2 metadata service. Attributes don't entirely correspond to the paths in the metadata service -they have been 'cleaned up'. You may also want to refer to the metadata service docs to understand the exact contents.

There's a singleton instance of it at the name ec2_metadata which should cover 90% of use cases. Use it like:

from ec2_metadata import ec2_metadata

ec2_metadata.region

The session argument, if provided, should be an instance of requests.Session, allowing you to customize the way requests are made.

Most of the attributes are cached, except where noted below. This is because they are mostly immutable, or at least require an instance stop to change. However some cached attributes do represent things that can change without an instance stop, but rarely do, such as network devices.

The caching is done with @cached_property, so they cache on first access. If you want to clear the cache of one attribute you can just del it:

del ec2_metadata.network_interfaces

To clear all, use the clear_all() method as per below.

account_id: str

The current AWS account ID, e.g. '123456789012'.

ami_id: str

The ID of the AMI used to launch the instance, e.g. 'ami-123456'.

autoscaling_target_lifecycle_state: str | None

Uncached. The target Auto Scaling lifecycle state that the instance is transitionioning to, or None if the instance is not in an autoscaling group. See AWS docs page Retrieve the target lifecycle state through instance metadata.

availability_zone: str

The name of the current AZ e.g. 'eu-west-1a'.

availability_zone_id: str | None

The unique, cross-account ID of the current AZ e.g. 'use1-az6'. See AWS docs page AZ IDs for your AWS resources.

ami_launch_index: int

The index of the instance in the launch request, zero-based, e.g. 0.

ami_manifest_path: str

The path to the AMI manifest file in Amazon S3, or '(unknown)' on EBS-backed AMI's.

clear_all() -> None

Clear all the cached attributes on the class, meaning their next access will re-fetch the data from the metadata API. This includes clearing the token used to authenticate with the service.

domain: str

The domain for AWS resources for the region. E.g. 'amazonaws.com' for the standard AWS regions and GovCloud (US), or 'amazonaws.com.cn' for China.

iam_info: IamInfoDict | None

A dictionary of data for the IAM role attached to the instance, or None if no role is attached. The dict has this type, based on what the metadata service returns:

class IamInfoDict(TypedDict):
    InstanceProfileArn: str
    InstanceProfileId: str
    LastUpdated: str

iam_security_credentials: IamSecurityCredentialsDict | None

A dictionary of data for the security credentials associated with the IAM role attached to the instance, or None if no role is attached. See the AWS docs section “Retrieve security credentials from instance metadata” for details. The dict has this type, based on that document:

class IamSecurityCredentialsDict(TypedDict):
    LastUpdated: str
    Type: str
    AccessKeyId: str
    SecretAccessKey: str
    Token: str
    Expiration: str

instance_action: str

Uncached. A state that notifies if the instance will reboot in preparation for bundling. See the AWS docs section “Instance Metadata Categories” for the valid values.

instance_id: str

The current instance's ID, e.g. 'i-123456'

instance_identity_document: InstanceIdentityDocumentDict

A dictionary of dynamic data about the instance. See the AWS docs page “Instance Identity Documents” for an explanation of the contents. The dict has this type, based on that document:

class InstanceIdentityDocumentDict(TypedDict):
    accountId: str
    architecture: Literal["i386", "x86_64", "arm64"]
    availabilityZone: str
    billingProducts: list[str] | None
    marketplaceProductCodes: list[str] | None
    imageId: str
    instanceId: str
    instanceType: str
    kernelId: str | None
    pendingTime: str
    privateIp: str
    ramdiskId: str | None
    region: str
    version: str

instance_profile_arn: str | None

The ARN of the IAM role/instance profile attached to the instance, taken from iam_info, or None if no role is attached.

instance_profile_id: str | None

The ID of the IAM role/instance profile attached to the instance, taken from iam_info, or None if no role is attached.

instance_profile_name: str | None

The instance profile name, extracted from instance_profile_arn, or None if no role is attached.

instance_type: str

The current instance's type, e.g. 't2.nano'

kernel_id: str | None

The current instance's kernel ID, or None if it doesn't have one, e.g. 'aki-dc9ed9af'.

mac : str

The instance's MAC address, e.g. '0a:d2:ae:4d:f3:12'

network_interfaces: dict[str, NetworkInterface]

A dictionary of mac address to NetworkInterface, which represents the data available on a network interface - see below. E.g. {'01:23:45:67:89:ab': NetworkInterface('01:23:45:67:89:ab')}

partition: str

The AWS partition where the instance is running. E.g. 'aws' for the standard AWS regions, 'aws-us-gov' for GovCloud (US), or 'aws-cn' for China.

private_hostname : str

The private IPv4 DNS hostname of the instance, e.g. 'ip-172-30-0-0.eu-west-1.compute.internal' .

private_ipv4: str

The private IPv4 of the instance, e.g. '172.30.0.0'.

public_hostname : str | None

The public DNS hostname of the instance, or None if the instance is not public, e.g. 'ec2-1-2-3-4.compute-1.amazonaws.com'.

public_ipv4: str | None

The public IPv4 address of the instance, or None if the instance is not public, e.g. '1.2.3.4'.

region: str

The region the instance is running in, e.g. 'eu-west-1'.

reservation_id: str

The ID of the reservation used to launch the instance, e.g. 'r-12345678901234567'.

security_groups : list[str]

List of security groups by name, e.g. ['ssh-access', 'custom-sg-1'].

spot_instance_action: SpotInstanceAction | None

Uncached. An object describing an action about to happen to this spot instance. Returns None if the instance is not spot, or not marked for termination.

The SpotInstanceAction object has two attributes:

  • action: str - the action about to happen, one of "hibernate", "stop", or "terminate".
  • time: datetime - the approximate UTC datetime when the action will occur.

See AWS docs section for a little more information.

tags: InstanceTags

A dict-like mapping of the tags for the instance (documented below). This requires you to explicitly enable the feature for the instance. If the feature is not enabled, accessing this attribute raises an error.

(It also seems that there is a bug where if the feature is enabled and then disabled, the metadata service returns an empty response. This is indistinguishable from “no tags”, so beware that in that case, InstanceTags will just look like an empty mapping.)

user_data: bytes | None

The raw user data assigned to the instance (not base64 encoded), or None if there is none.

InstanceTags

A dict-like mapping of tag names to values (both strs). To avoid unnecessary requests, the mapping is lazy: values are only fetched when required. (Names are known on construction though, from the first request in EC2Metadata.tags.)

The metadata service will receive tag updates on some instance types, as per the AWS documentation:

If you add or remove an instance tag, the instance metadata is updated while the instance is running for instances built on the Nitro System, without needing to stop and then start the instance. For all other instances, to update the tags in the instance metadata, you must stop and then start the instance.

Because InstanceTags is cached, it won’t reflect such updates on Nitro instances unless you clear it first:

del ec2_metadata.tags
ec2_metadata.tags["Name"]  # fresh

NetworkInterface

Represents a single network interface, as retrieved from EC2Metadata.network_interfaces. Again like EC2Metadata all its attributes cache on first access, and can be cleared with del or its clear_all() method.

device_number: int

The unique device number associated with that interface, e.g. 0.

interface_id: str

The unique id used to identify the Elastic Network Interface, e.g. 'eni-12345'.

ipv4_associations: dict[str, list[str]]

A dictionary mapping public IP addresses on the interface to the list of private IP addresses associated with that public IP, for each public IP that is associated with the interface, e.g. {'54.0.0.1': ['172.30.0.0']}.

ipv6s: list[str]

The IPv6 addresses associated with the interface, e.g. ['2001:db8:abcd:ef00::1234'].

mac: str

The MAC address of the interface, e.g. '01:23:45:67:89:ab'.

owner_id: str

The AWS Account ID of the owner of the network interface, e.g. '123456789012'.

private_hostname: str

The interface's local/private hostname, e.g. 'ip-172-30-0-0.eu-west-1.compute.internal'.

private_ipv4s: list[str]

The private IPv4 addresses associated with the interface, e.g. ['172.30.0.0'].

public_hostname: str | None

The interface's public DNS (IPv4), e.g. 'ec2-54-0-0-0.compute-1.amazonaws.com'.

public_ipv4s: list[str]

The Elastic IP addresses associated with the interface, e.g. ['54.0.0.0'].

security_groups: list[str]

The names of the security groups to which the network interface belongs, e.g. ['ssh-access', 'custom-sg-1'].

security_group_ids: list[str]

The names of the security groups to which the network interface belongs, e.g. ['sg-12345678', 'sg-12345679'].

subnet_id: str

The ID of the subnet in which the interface resides, e.g. 'subnet-12345678'.

subnet_ipv4_cidr_block: str | None

The IPv4 CIDR block of the subnet in which the interface resides, or None if there is none, e.g. '172.30.0.0/24'.

subnet_ipv6_cidr_blocks: list[str]

The list of IPv6 CIDR blocks of the subnet in which the interface resides, e.g. ['2001:db8:abcd:ef00::/64']. If the subnet does not have any IPv6 CIDR blocks or the instance isn't in a VPC, the list will be empty, e.g. [].

vpc_id: str

The ID of the VPC in which the interface resides, e.g. 'vpc-12345678'.

vpc_ipv4_cidr_block: str | None

The IPv4 CIDR block of the VPC, or None if the instance isn't in a VPC, e.g. '172.30.0.0/16'.

vpc_ipv4_cidr_blocks: list[str]

The list of IPv4 CIDR blocks e.g. ['172.30.0.0/16']. If the interface doesn’t have any such CIDR blocks, the list will be empty.

vpc_ipv6_cidr_blocks: list[str]

The list of IPv6 CIDR blocks of the VPC in which the interface resides, e.g. ['2001:db8:abcd:ef00::/56']. If the VPC does not have any IPv6 CIDR blocks or the instance isn't in a VPC, the list will be empty, e.g. [].

ec2-metadata's People

Contributors

adamchainz avatar arossert avatar cole14 avatar dependabot[bot] avatar epf avatar griffint61 avatar jsjeannotte avatar levisbakalinsky avatar mvanbaak avatar pre-commit-ci[bot] avatar quimey avatar renanqts avatar silverlyra avatar tedder avatar thrawnca avatar zoltan-kecskemethy-epam avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

ec2-metadata's Issues

Provide a CLI?

A CLI to print out the metadata as env vars, or to run programs with said env vars, could be very useful. The only thing is which vars to export. Perhaps it should need a whitelist.

Examples:

$ ec2-metadata print INSTANCE_ID AWS_REGION
INSTANCE_ID=i-123456
AWS_REGION=eu-west-2
$ ec2-metadata exec AWS_REGION -- env | grep AWS_
AWS_REGION=eu-west-2

availability_zone_id may return 404

Python Version

3.8.10

Package Version

2.7.0

Description

In some cases ec2_metadata.availability_zone_id returns 404

----> 1 ec2_metadata.availability_zone_id
File functools.py:967, in __get__(self, instance, owner)
File ec2_metadata/__init__.py:87, in availability_zone_id(self)
File ec2_metadata/__init__.py:65, in _get_url(self, url, allow_404)
File requests/models.py:960, in raise_for_status(self)
HTTPError: 404 Client Error: Not Found for url: http://169.254.169.254/latest/meta-data/placement/availability-zone-id

I assume that all that is needed is to use allow_404 in _get_url

Provide a test mock

It would be useful to provide a test mock version of the EC2Metadata class that returns dummy data without making requests. This could also be provided as a pytest fixture so it's easy for pytest users to integrate it.

Filter out non-200 responses

Currently we don't filter out error responses from the metadata API, which means it can return data from error responses. There should be a resp.raise_for_status() in there, or even filter for exactly resp.status_code == 200

Pypi wheel republished?

Sorry for filing this as a bug but I wasn't really sure how to report it.

Did the ec2-metadata package on pypi get updated? Our CI failed with an "invalid hash" error. I see that on pypi the wheel package was updated after the tarball. If this was intentional then I can just update the hash in the constraints file, I just want to make sure this isn't an event-stream sort of situation.

HTTPConnectionPool(host='169.254.169.254') timeout in docker bridge mode

Python Version

3.9.13

Package Version

2.10.0

Description

I tried use ec2-metadata inside docker container with bridge mode. Curl "169.254.169.254" got ok but ec2-metadata raised error "requests.exceptions.ReadTimeout: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=5.0)"

image

Get empty object

Hi Adam,
I'm new to Python, so bear with me: I'm trying to use your library to get the ec2 metadata when running on EC2. But when I'm running locally, I need an 'empty' version of it that doesn't throw exceptions at me. How do I do this?
thx,
k

is it possible to use ec2_metadata from lambda?

Hi Adam,

I stumbled on your code after having issues querying metadata in lambda. I'm always getting a 'connection refused' (both using requests to query the metadata directly and using your library). I can understand if it's a security issue AWS have put in place as with lambda you're using a shared tenancy, but i'm not sure. Do you know what the limitation is and do you know if and how this can be solved?

Thanks,
Marcel

Instance Role Name

Would it be possible to return the role name from instance meta data? That way i don't have to make a boto3 call?

Use timeout for talking to the metadata API

Use a timeout so that processes don't hang indefinitely if they can't hit the API but crash after a sensible number of seconds. Just a few should be enough since it's normally responsive within millieconds.

Check to see if host is an ec2 instance

Hi -- thanks, project this is great.

I think this is a hard problem, but is there some way to add a method that allows one to check if the current host is an ec2 instance?

thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.