elgrhy / advancedneural

This repository introduces the AdvancedAccelerator, a lightweight and efficient accelerator designed to enhance the performance of transformer-based neural network models, particularly large language models (LLMs).

AdvancedAccelerator: Enhanced Transformer Accelerator

Overview

The AdvancedAccelerator is an enhanced accelerator designed to improve the efficiency and performance of transformer-based models. It incorporates advanced features such as dropout, residual connections, and layer-wise normalization to enhance the robustness and generalization of the accelerator.

Features

• Initial feature transformation with linear layers.
• Multi-Head Self Attention mechanism with dropout for regularization.
• Feedforward layer with dropout and ReLU activation.
• Layer normalization after each linear layer for stability.
• Residual connections with a learnable scaling factor.
• Configurable dropout rate and other hyperparameters.

Usage

from advancedNeural import AdvancedAccelerator

# Instantiate the accelerator
accelerator = AdvancedAccelerator(input_size, output_size, hidden_size=256, dropout_rate=0.1)

# Forward pass
output = accelerator(input_data)

Please check the usage.md file

Parameters

• input_size: Input dimensionality of the data.
• output_size: Output dimensionality of the accelerator.
• hidden_size: Dimensionality of the hidden layer (default: 256).
• dropout_rate: Dropout rate for regularization (default: 0.1).

# Example instantiation
accelerator = AdvancedAccelerator(input_size=512, output_size=256, hidden_size=128, dropout_rate=0.2)

# Forward pass with sample input
output = accelerator(torch.randn(32, 512))

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

Ahmed Elgarhy - Initial work.