meesho / llm_calculator

LLM Calculator

This repository contains code to find the optimal runtime to fine-tune the Llama2-7b model and optimal codebase.

llm_calculator's Issues

Documentation about how the numbers were calculated.

llm_calculator/optimal_llm_calculator/optimal_runtime_calculator.py

Lines 156 to 168 in 1edc5cc

	total_time_taken_by_train_steps = (
	total_number_of_steps
	* 93
	* (self.tokenized_size / 550)
	* (total_number_datapoints_in_1_step / 4096)
	)
	total_time_taken_by_save_steps = total_number_of_save_steps * 23
	total_time_taken_by_eval_steps = (
	total_number_of_eval_steps
	* 44
	* (self.tokenized_size / 550)
	* (self.number_of_datapoints_to_keep_in_eval / 6868)
	)

Having difficulty understanding how some of the numbers here were computed, like 93, 23 and 44.