aws-samples / foundation-model-benchmarking-tool Goto Github PK

It would be interesting to see the effect of scaling to multiple instances behind the same endpoint. How does inference latency change as endpoints start to scale (automatically, we could also add parameters for scaling policy), can we support the more transactions with auto-scaling instances while keeping the latency below a threshold and what are the cost implications of doing that. This needs to be fleshed out but this is an interesting area.

This would also need to include support for the Inference Configuration feature that is now available with SageMaker.

The tool can easily allow to compare performance across regions if we allowed region to be a part of the per experiment config.

FMBench needs to be configurable so that no S3 access is needed.

The config file has lots of parameters and while some of them are explained via inline comments, we really need a better way for documenting this.

Would be good to have a CFT template that works in different AWS Region. Would be good to provide support in us-west-2

While this tool was never thought of as testing models hosted on anything other than SageMaker, but technically there is nothing preventing this. There are two things that need to happen for this.

1/ The models have to be deployed on the platform of choice. This part can be externalized, meaning the deployment code in this repo does not deploy those models, they are deployed separately, outside of this tool.

2/ Have support for bring your own inference script which knows how to query your endpoint. This inference script then runs inferences against the endpoints on platforms other than SageMaker. And so at this point it does not matter if the endpoint is on EKS or EC2.

Visualizations are powerful, so the message that it is the full serving stack that results in a particular performance benchmark can get lost unless explicitly called out. It is possible that someone could take the impression that a given instance type always performance better without considering that it is the instance type+inference container +parameters and so the results should not be taken out of context.

This includes adding support for s3 readability and interaction, including all data and metrics being accessible via your personal s3 bucket. The goal of this issue is to abstract out the code on this repo in a way where you can bring your own script, your own model, source data files, upload it to s3, and then expect this repo to run, in order to generate these test results within your s3 bucket that you define. Aim for this is to have a folder in a bucket where you upload your source data files, a folder where you upload your bring your own model script, prompt template, and other model artifacts as need be, and then run this repo to generate test results within the 'data' folder that is programmatically generated containing information on metrics, per chunk and inference results, deployed model configurations, and more.

1. It would be good to have a native integration in the Config.Yaml file of FMBench for integrating with other 3rd party tools for pulling synthetically generated datasets.
2. How to split this synthetically generated datasets for FMBench based evaluation of FM's would be important functionality to have.
3. Final Results (results.md) generated by this tool could have some Visual comparison capability with other 3rd party tools which could be used for holistic evaluation of FM's.

The fact that we are running inference means that we can also measure the accuracy of those inferences i.e. through rouge score, cosine similarity (to an expert generated response) or other metrics. If we add that then this tool can provide a complete bechmarking solution that includes accuracy as well as cost.

src/fmbench/config_filepath.txt
and
manifest.txt
Both show the config files being located in the config directory, but they are now split up under subdirectories.

This causes
from fmbench.utils import *
in
src/fmbench/0_setup.ipynb

to fail with:

config file current -> configs/config-bedrock-claude.yml, None
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[9], line 10
      8 import fmbench.scripts
      9 from pathlib import Path
---> 10 from fmbench.utils import *
     11 from fmbench.globals import *
     12 from typing import Dict, List, Optional

File ~/.fmbench/lib/python3.11/site-packages/fmbench/utils.py:11
      9 import unicodedata
     10 from pathlib import Path
---> 11 from fmbench import globals
     12 from fmbench import defaults
     13 from typing import Dict, List

File ~/.fmbench/lib/python3.11/site-packages/fmbench/globals.py:53
     51     CONFIG_FILE_CONTENT = response.text
     52 else:
---> 53     CONFIG_FILE_CONTENT = Path(CONFIG_FILE).read_text()
     55 # check if the file is still parameterized and if so replace the parameters with actual values
     56 # if the file is not parameterized then the following statements change nothing
     57 args = dict(region=session.region_name,
     58             role_arn=arn_string,
     59             write_bucket=f"{defaults.DEFAULT_BUCKET_WRITE}-{region_name}-{account_id}",
     60             read_bucket=f"{defaults.DEFAULT_BUCKET_READ}-{region_name}-{account_id}")

File /usr/lib/python3.11/pathlib.py:1058, in Path.read_text(self, encoding, errors)
   1054 """
   1055 Open the file in text mode, read it, and close the file.
   1056 """
   1057 encoding = io.text_encoding(encoding)
-> 1058 with self.open(mode='r', encoding=encoding, errors=errors) as f:
   1059     return f.read()

File /usr/lib/python3.11/pathlib.py:1044, in Path.open(self, mode, buffering, encoding, errors, newline)
   1042 if "b" not in mode:
   1043     encoding = io.text_encoding(encoding)
-> 1044 return io.open(self, mode, buffering, encoding, errors, newline)

FileNotFoundError: [Errno 2] No such file or directory: 'configs/config-bedrock-claude.yml'

The need of this issue is to have granular access to each notebook while having the repo being a package that can be run via pip. This is for advanced users in the space who want to change the code for different metrics and modifications so they can run each notebook one by one along with having the option to pip install the fmbt package.

This tool currently supports the HF TGI container, and DJL Deep Speed container on SageMaker and both use the same format but in future other containers might need a different payload format.

Goal: To give user full flexibility to bring their payloads or contain code that generalizes payload generation irrespective of the container type that the user uses. Two options for solution to this issue here:

1/ Have the user bring in their own payload
2/ Have a generic function defined to convert the payload in support for the container type the user is using to deploy their model and generate inference from.

Hi team,

Since FMBench today requires Python 3.11+, it can't be installed on the current SageMaker Studio JupyterLab (default Python kernel at SM Distribution v1.7 runs Python 3.10)

Are you really using newly launched features in Python 3.11? Would you be able to add 3.10 support so we can use the library in SageMaker Studio notebooks?

Thanks!

To add another dataset support out of the box using the OpenOrca HF dataset. Link: https://huggingface.co/datasets/Open-Orca/OpenOrca

We want to add and test functionality for an already deployed SageMaker endpoint and test it with FMBench without having the need to re deploy the endpoint/create a new endpoint

To calculate the cost per config file run for this FMBT harness. This includes model instance type, inference, cost per transactions and so on to sum up the entire run's total cost.

Emit live metrics so that they can be monitored through Grafana via live dashboard. More information to come on this issue but the goal here is to provide full flexibility to the user to be able to view metrics in ways that best suits the needs of their business and technological goals.

[TBD] --> Some sort of an analytics pipeline sending and emitting live results for different model configurations, their results and different metrics based on the needs of the user.

Update the readme to clarify that it is the complete serving stack that we are comparing across different experiments. The serving stack includes everything from the instance type, the inference container and the backend configuration parameters.

Add code to determine the cost of running an entire experiment and include it in the final report. This would only include the cost of running the SageMaker endpoints based on hourly public pricing (the cost of running this code on a notebook or a EC2 is trivial in comparison and can be ignored).

Running this entire benchmarking test, we can add a couple of lines to calculate the total cost being used to run the specific experiment from an end to end perspective to answer simple questions like:

I am running the experiment for this config file and got the benchmarking results successfully in 'x' time. What is the cost that will be incurred to run this experiment?

Containerize FMBT and provide instructions for running the container on EC2 - Once all of the files are integrated via S3 and all the code is abstracted out in terms of generating metrics for any deployable model on SageMaker (including bring your own model/scripts), we want to be able to containerize this and run the container on EC2.

Goal: To choose the specific config file and prompt to run the container on it, generating results without any heavy lifting of any development efforts.

This issue represents the need to benchmark for embedding models that are hosted via sagemaker, bedrock and bring your own custom embedding models.

FLAN-T5 XL is still used by multiple customers so a comparison of this model across g5.xlarge and g5.2xlarge instances would be very useful and so a config file for this should be provided.

To have a functionalit that enables FMBT to support sagemaker endpoint autoscaling via config files for experiment runs.

Currently only the LLama tokenizer is supported but we want to allow users to bring their own token counting logic for different models. This way regardless of the model type, or token count methodology, the user should be able to get accurate results based on their token counter that they use.

Goal: Abstracting out the repo and tool to a point where no matter what token counter type the user uses, you can bring that and run the container to get accurate test results.

• To add support for an REST predictor to run inferences on a model deployed out of sagemaker/bedrock

There is nothing in FMBench which prevents different experiments in the same config file to use different models, but the generated report is not architected in the same way i.e. it is not created to compare different models but rather compare the same model across serving stacks, so that would need to change. This has been requested by multiple customers, the idea being if we find different models that are fit for task, we now want to find the model and serving stack combination which provides the best price:performance.

Create a Python Package out of the 'FMBT' tool to be able to use it as a PyPi Library.

The work for this repo started as a skunkworks project done over the holidays in the Winter of 2024 and so at the time this was just a bunch of notebooks but now that it has transformed into a formal open-source project with a ton of functionality and a bunch of roadmap items, the notebooks have turned unwieldy!

We need to replace the notebooks with regular python scripts and plus there is a whole bunch of code cleanup that needs to happen to replace global imports, use type hints everywhere, optimizations of functions, etc., the list is long.

Assigning this to myself for now, and would create issues for specific items.

Can we add Bedrock to this tool? While support for bring your own inference script would do that but we need to think through Bedrock specific options such as provisioned throughput, auto-generated reported formats, do we want to compare Bedrock and SageMaker side by side.

For models other than Llama and Mistral (say BERT) we need datasets other than LongBench and these models have their own response format.

1. Add support for bring your own dataset by parameterizing the prompt template.
2. Add support for custom inference scripts.