The Accelerate backend should not be required to use the application. Accelerate is not compatible with many ML libraries.

Write tests for RawFile located in src/benchkit/data/file_saver

Write a unit test for function get_local_dataloader in src/benchkit/data/helpers

Make Sure Pytorch is not essential to using the library, allows users to train models using things like Jax, TensorFlow, etc.

Create Design Document before tackling this issue

Write a unit test for function remove_all_temps in src/benchkit/data/helpers

Write a unit test for function create_dataset_dir in src/benchkit/data/helpers

Write tests for BooleanFile located in src/benchkit/data/file_saver

Write a unit test for function compress_directory in src/benchkit/data/helpers

Write tests for NumpyFile located in src/benchkit/data/file_saver

Write tests for the Iterable Chunk class located in src/benchkit/data/datasets/

Users should be able to train and deploy their models through Tensorflow, mainly they should be able to use the IterableChunk class with TensorFlow

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Write a unit test for function test_dataloading in src/benchkit/data/helpers

Write tests for TextFile located in src/benchkit/data/file_saver

Refactor and test the save_file_and_label method located in src/benchkit/data/helpers

Write a unit test for function get_folder_size in src/benchkit/data/helpers

Yaml file

Premise

The Yaml File should structure their projects with the proper dependencies, integrations, and template code based on the settings provided.

Future

• In the future, you can use this to pull other user's code and train them just by modifying some settings in the YAML.
• You should also be able to detail cloud settings from this file alone and start cloud training sessions.

Implementation

The Yaml File will hold configuration details on the project, that can be used to streamline the model training process.

In the file, you will detail:

• Which framework to use: PyTorch, TensorFlow, JAX, ...
• The type of model that is being used: LLM, YOLO, ...
• the actions you wish to perform Train, Deploy, FineTune
• The distributed framework being used: Accelerate, Horovod, DDP, None
• Distributed Settings File
• additional dependencies
• dataset configurations

The same configuration should be able to be done without a YAML simply through an object

Unicode error while creating a new project in windows

Write a unit test for function get_directory_size in src/benchkit/data/helpers

Write tests for JsonFile located in src/benchkit/data/file_saver

Everything done through the CLI should be doable through an API object.

Make design documentation before implementing.

Write a unit test for function get_dir_size in src/benchkit/data/helpers

if only one page of experiments is present the user is not able to access the logs for that experiment. Need to make sure they can access the logs in this scenario as well.

Write a unit test for function get_total_file_count in src/benchkit/data/helpers

Users should be able to track experiments that do not take place on the managed cloud

Add integrations and templates that make it easier to train LLMs on Bench AI, should integrate well with YAML file.

Write a unit test for function create_dataset_zips in src/benchkit/data/helpers

Speedup the Processor Dataset using multi processing/threading

Write a unit test for function get_dataloader in src/benchkit/data/helpers

Make sure there are restriction put in place to make sure presigned urls arent given after a certain numbe and so file sizes have a minimum

show ds does not show dataset properly

Write tests for TorchFile located in src/benchkit/data/file_saver

Add the ability to record the graphs using tensor-board and pull the graphs to local using bench kit

Write tests for NumericFile located in src/benchkit/data/file_saver

Write a unit test for function iterate_directory in src/benchkit/data/helpers

Make bench-kit not use docker

Write Tests for the ProcessorDataset class, located at src/benchkit/data/datasets