aws-samples / aws-emr-utilities

Amazon EMR Utilities

This repository contains sample code and utilities for using Amazon EMR on EC2. This package is structured based on the following directories:

• applications - application specific patches, plugins, etc.
• utilities - administrative and maintenance utilities for working with EMR

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

Disclaimer

The examples provided in this repository are not supported by AWS EMR. The use of this code is your responsibility and at your own risk.

aws-emr-utilities's Issues

include ebs calculations with serverless calculation

EMR Cost Calculations

It would be nice to also have a non serverless EMR Cost calculator to go along with the serverless calculator.

This repository served as an excellent starting point. There are many forks floating around so it would be awesome to centrally locate it here.
https://github.com/mauropelucchi/aws-emr-cost-calculator

I converted that codebase to a Lambda and added EMR runtime and EMR Step runtimes to it and fixed the EBS calculations. The Lambda sends the metrics to our monitoring solution. It get's triggered though AWS Eventbridge Notifications for EMR terminated events.

New Guide for "Custom Python3 Version on EMR 7.x"

Currently this guide works well on EMR 6.x
https://github.com/aws-samples/aws-emr-utilities/blob/main/utilities/emr-ec2-custom-python3/README.md

It would be great to update to support MR 7.x!

Here is what I succeed:

# systemtap-sdt-devel is required for `./configure --with-dtrace`, https://docs.python.org/3/howto/instrumentation.html
# zlib-devel is required for `sudo make altinstall`

PYTHON_VERSION=3.11.7
sudo yum --assumeyes install \
  bzip2-devel \
  expat-devel \
  gcc \
  libffi-devel \
  make \
  systemtap-sdt-devel \
  tar \
  zlib-devel
curl --silent --fail --show-error --location "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz" | tar -x -J -v
cd "Python-${PYTHON_VERSION}"
export CFLAGS="-march=native"
./configure \
  --enable-loadable-sqlite-extensions \
  --with-dtrace \
  --with-lto \
  --enable-optimizations \
  --with-system-expat \
  --prefix="/usr/local/python${PYTHON_VERSION}"
sudo make altinstall
sudo "/usr/local/python${PYTHON_VERSION}/bin/python${PYTHON_VERSION%.*}" -m pip install --upgrade pip

Note no need to upgrade openssl any more, as EMR 7.0 comes with openssl version 3.0.8

Hope to save people some time in future! ☺️

How do I get the cluster-id? This is for serverless

I am referring to the README at https://github.com/aws-samples/aws-emr-utilities/blob/main/utilities/emr-serverless-estimator/README.md
Isn't a cluster-id available only when you use a cluster, which is not the case this script is for ?

Could not find SparkEventLogParser-slim-1.0.jar in emr-serverless-estimator

Hi, I think when you created the emr-serverless-estimator project. The .gitignore actually prevented upload jar files.

Also, I could not seem to find SparkEventLogParser-slim-1.0.jar in Maven Central Either.

Can you help explaining where can I find those jars in order to run the estimator? (Or upload the jar to the repo if it is not too big)

Thank you and have a nice day. :)

Steps provided are to calculate the cost of spark jobs for the emr cluster and not for the emr serverless

Steps provided are calculate cost of spark jobs for emr cluster and not for emr serverless. Could you provide the steps to sync the spark event logs to S3 and calculate the cost for the emr serverless jobs?.

Typo in Readme file

The comment say that "presto" should be replaced with "trino", but it actually opposite. "trino" should be replaced with "presto"
"If using Presto, modify trino_cloudwatch.sh and replace all "presto" occurences with "trino""