hkproj / pytorch-llama

LLaMA 2 implemented from scratch in PyTorch

pytorch-llama

LLaMA 2 implemented from scratch in PyTorch

pytorch-llama's Issues

will it work on windows OS with only CPU?

will it work on windows OS with only CPU?

alternative implementation of group-query attention (GQA)

here is a probably-better implementation for group-query attention which might run faster. It utilize a property of pytorch that matmul is broadcasting.

Multiplication is not dividing in the RMS norm.

I appreciate you producing such intriguing and valuable articles. an excellent learning resource.

Just wanted to confirm: At present, we have multiplication and not divide in the RMS norm.

Current:
x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

Is this true:
x / torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

[Question] Is this line a bug?

pytorch-llama/inference.py

Line 90 in 067f8a3

next_token = torch.argmax(logits[:, -1], dim=-1)

I have a question about the above line. Shouldn't the logits go through a softmax here?
Is this line correct?
Thanks a lot!

Can I turn off KV cache?

Hello,

Could you please advise me on how to disable the KV cache?
I would also appreciate any guidance on how to implement this change in code.

Thank you for your assistance.

what is minimal computer can be used for only inference

what is minimal computer can be used for only inference
i do have ubuntu with 3060 GPU 8 GB
can I use it?

Error in rotary matrix multiplication formula of slide 25

First of all, thank you for the great resources and Youtube videos.

I wanted to point out that in slide 25 of the Llama notes, regarding the computational efficient realization of rotary matrix multiplication, there is a typo in the third vector, as the order of the last two entries ($x_d$ and $x_{d-1}$) is inverted.

This is what is currently in the slide:

$$R_{\theta, m}^d x = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \\ ... \\ x_{d-1} \\ x_d \end{bmatrix} \otimes \begin{bmatrix} cos (m\theta_1) \\ cos (m\theta_1) \\ cos (m\theta_2) \\ cos (m\theta_2) \\ ... \\ cos (m\theta_{d/2}) \\\ cos (m\theta_{d/2})\end{bmatrix} + \begin{bmatrix} -x_2 \\ x_1 \\ -x_4 \\ x_3 \\ ... \\ \mathbf{-x_{d-1}} \\ \mathbf{x_{d}} \end{bmatrix} \otimes \begin{bmatrix} sin (m\theta_1) \\ sin (m\theta_1) \\ sin (m\theta_2) \\ sin (m\theta_2) \\ ... \\ sin (m\theta_{d/2}) \\\ sin (m\theta_{d/2})\end{bmatrix}$$

And this is how it should be:

$$R_{\theta, m}^d x = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \\ ... \\ x_{d-1} \\ x_d \end{bmatrix} \otimes \begin{bmatrix} cos (m\theta_1) \\ cos (m\theta_1) \\ cos (m\theta_2) \\ cos (m\theta_2) \\ ... \\ cos (m\theta_{d/2}) \\\ cos (m\theta_{d/2})\end{bmatrix} + \begin{bmatrix} -x_2 \\ x_1 \\ -x_4 \\ x_3 \\ ... \\ \mathbf{-x_{d}} \\ \mathbf{x_{d-1}} \end{bmatrix} \otimes \begin{bmatrix} sin (m\theta_1) \\ sin (m\theta_1) \\ sin (m\theta_2) \\ sin (m\theta_2) \\ ... \\ sin (m\theta_{d/2}) \\\ sin (m\theta_{d/2})\end{bmatrix}$$

I imagine the error was already in the original paper and the screenshot comes from there. I have checked the RoFormer paper (page 7 equation 34) and they have since fixed the error.

Where did the factor 2 go in rotary embedding?

Well, thanks for you great video on youtube. I just got one question, in your implementation, in method precompute_theta_pos_frequencies. The equation of theta contains a factor 2 like you wrote in your comment, theta_i = 10000^(-2(i-1)/dim) for i = [1, 2, ... dim/2]. However, in your code, I don't see a factor 2 added, how is that?

an unexpected keyword argument 'rope_theta'

Hi, After running the download script, I pasted the URL from the email and encountered the following errors in my terminal:

Enter the URL from email: https://download.llamameta.net/*?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoid2p5eTF5YjQwZDRxYXJwZWc5ZXhxaHE2IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDQ5OTQxM319fV19&Signature=PHLzzsM%7EBsX9EzzxAH1MbEzQjBWoMf4OM%7Eb7M0-auOub1jEBXL8X3xbJx24uaKGrCC-6VLPyjqpReHVcRkNOt6IH543eeypZwAgSH9io7P-Q-JQGKjHaXYjgD0A8n6330VRC07wQGC9VU1Lv%7E0RzpZDdPMtfkrEas3W9bLUTfE0BlxY5ilvpcHDSuAFwtQk6MP0AlzLSiqzQQCc0Bj93rn6F%7EG0VKgkN8LAc6Z%7EYP3%7E2ZtuWo5KuoUsqXujB7UnzZTlrW3ZR5ODVWUB%7Ed6AWHlh7KLtczSKJTtLaC6vBOE0Tt9XFwX0sFazctlizlf-s6AQn26tMi0WryOQVXvrOGg__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=1361551944476193

Enter the list of models to download without spaces (7B,13B,70B,7B-chat,13B-chat,70B-chat), or press Enter for all: 7b
Downloading LICENSE and Acceptable Usage Policy
--2023-11-20 23:25:33--  https://download.llamameta.net/LICENSE?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoid2p5eTF5YjQwZDRxYXJwZWc5ZXhxaHE2IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDQ5OTQxM319fV19&Signature=PHLzzsM%7EBsX9EzzxAH1MbEzQjBWoMf4OM%7Eb7M0-auOub1jEBXL8X3xbJx24uaKGrCC-6VLPyjqpReHVcRkNOt6IH543eeypZwAgSH9io7P-Q-JQGKjHaXYjgD0A8n6330VRC07wQGC9VU1Lv%7E0RzpZDdPMtfkrEas3W9bLUTfE0BlxY5ilvpcHDSuAFwtQk6MP0AlzLSiqzQQCc0Bj93rn6F%7EG0VKgkN8LAc6Z%7EYP3%7E2ZtuWo5KuoUsqXujB7UnzZTlrW3ZR5ODVWUB%7Ed6AWHlh7KLtczSKJTtLaC6vBOE0Tt9XFwX0sFazctlizlf-s6AQn26tMi0WryOQVXvrOGg__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=1361551944476193
Resolving download.llamameta.net (download.llamameta.net)... 108.138.51.32, 108.138.51.79, 108.138.51.128, ...
Connecting to download.llamameta.net (download.llamameta.net)|108.138.51.32|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-20 23:25:33 ERROR 403: Forbidden.

--2023-11-20 23:25:33--  https://download.llamameta.net/USE_POLICY.md?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoid2p5eTF5YjQwZDRxYXJwZWc5ZXhxaHE2IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDQ5OTQxM319fV19&Signature=PHLzzsM%7EBsX9EzzxAH1MbEzQjBWoMf4OM%7Eb7M0-auOub1jEBXL8X3xbJx24uaKGrCC-6VLPyjqpReHVcRkNOt6IH543eeypZwAgSH9io7P-Q-JQGKjHaXYjgD0A8n6330VRC07wQGC9VU1Lv%7E0RzpZDdPMtfkrEas3W9bLUTfE0BlxY5ilvpcHDSuAFwtQk6MP0AlzLSiqzQQCc0Bj93rn6F%7EG0VKgkN8LAc6Z%7EYP3%7E2ZtuWo5KuoUsqXujB7UnzZTlrW3ZR5ODVWUB%7Ed6AWHlh7KLtczSKJTtLaC6vBOE0Tt9XFwX0sFazctlizlf-s6AQn26tMi0WryOQVXvrOGg__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=1361551944476193
Resolving download.llamameta.net (download.llamameta.net)... 108.138.51.32, 108.138.51.79, 108.138.51.128, ...
Connecting to download.llamameta.net (download.llamameta.net)|108.138.51.32|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-20 23:25:33 ERROR 403: Forbidden.

Downloading tokenizer
--2023-11-20 23:25:33--  https://download.llamameta.net/tokenizer.model?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoid2p5eTF5YjQwZDRxYXJwZWc5ZXhxaHE2IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDQ5OTQxM319fV19&Signature=PHLzzsM%7EBsX9EzzxAH1MbEzQjBWoMf4OM%7Eb7M0-auOub1jEBXL8X3xbJx24uaKGrCC-6VLPyjqpReHVcRkNOt6IH543eeypZwAgSH9io7P-Q-JQGKjHaXYjgD0A8n6330VRC07wQGC9VU1Lv%7E0RzpZDdPMtfkrEas3W9bLUTfE0BlxY5ilvpcHDSuAFwtQk6MP0AlzLSiqzQQCc0Bj93rn6F%7EG0VKgkN8LAc6Z%7EYP3%7E2ZtuWo5KuoUsqXujB7UnzZTlrW3ZR5ODVWUB%7Ed6AWHlh7KLtczSKJTtLaC6vBOE0Tt9XFwX0sFazctlizlf-s6AQn26tMi0WryOQVXvrOGg__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=1361551944476193
Resolving download.llamameta.net (download.llamameta.net)... 108.138.51.32, 108.138.51.79, 108.138.51.128, ...
Connecting to download.llamameta.net (download.llamameta.net)|108.138.51.32|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-20 23:25:33 ERROR 403: Forbidden.

--2023-11-20 23:25:33--  https://download.llamameta.net/tokenizer_checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoid2p5eTF5YjQwZDRxYXJwZWc5ZXhxaHE2IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDQ5OTQxM319fV19&Signature=PHLzzsM%7EBsX9EzzxAH1MbEzQjBWoMf4OM%7Eb7M0-auOub1jEBXL8X3xbJx24uaKGrCC-6VLPyjqpReHVcRkNOt6IH543eeypZwAgSH9io7P-Q-JQGKjHaXYjgD0A8n6330VRC07wQGC9VU1Lv%7E0RzpZDdPMtfkrEas3W9bLUTfE0BlxY5ilvpcHDSuAFwtQk6MP0AlzLSiqzQQCc0Bj93rn6F%7EG0VKgkN8LAc6Z%7EYP3%7E2ZtuWo5KuoUsqXujB7UnzZTlrW3ZR5ODVWUB%7Ed6AWHlh7KLtczSKJTtLaC6vBOE0Tt9XFwX0sFazctlizlf-s6AQn26tMi0WryOQVXvrOGg__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=1361551944476193
Resolving download.llamameta.net (download.llamameta.net)... 108.138.51.32, 108.138.51.79, 108.138.51.128, ...
Connecting to download.llamameta.net (download.llamameta.net)|108.138.51.32|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-20 23:25:33 ERROR 403: Forbidden.

md5sum: tokenizer_checklist.chk: no properly formatted checksum lines found
Downloading
--2023-11-20 23:25:33--  https://download.llamameta.net//params.json?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoid2p5eTF5YjQwZDRxYXJwZWc5ZXhxaHE2IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDQ5OTQxM319fV19&Signature=PHLzzsM%7EBsX9EzzxAH1MbEzQjBWoMf4OM%7Eb7M0-auOub1jEBXL8X3xbJx24uaKGrCC-6VLPyjqpReHVcRkNOt6IH543eeypZwAgSH9io7P-Q-JQGKjHaXYjgD0A8n6330VRC07wQGC9VU1Lv%7E0RzpZDdPMtfkrEas3W9bLUTfE0BlxY5ilvpcHDSuAFwtQk6MP0AlzLSiqzQQCc0Bj93rn6F%7EG0VKgkN8LAc6Z%7EYP3%7E2ZtuWo5KuoUsqXujB7UnzZTlrW3ZR5ODVWUB%7Ed6AWHlh7KLtczSKJTtLaC6vBOE0Tt9XFwX0sFazctlizlf-s6AQn26tMi0WryOQVXvrOGg__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=1361551944476193
Resolving download.llamameta.net (download.llamameta.net)... 108.138.51.32, 108.138.51.79, 108.138.51.128, ...
Connecting to download.llamameta.net (download.llamameta.net)|108.138.51.32|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-20 23:25:34 ERROR 403: Forbidden.

--2023-11-20 23:25:34--  https://download.llamameta.net//checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoid2p5eTF5YjQwZDRxYXJwZWc5ZXhxaHE2IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDQ5OTQxM319fV19&Signature=PHLzzsM%7EBsX9EzzxAH1MbEzQjBWoMf4OM%7Eb7M0-auOub1jEBXL8X3xbJx24uaKGrCC-6VLPyjqpReHVcRkNOt6IH543eeypZwAgSH9io7P-Q-JQGKjHaXYjgD0A8n6330VRC07wQGC9VU1Lv%7E0RzpZDdPMtfkrEas3W9bLUTfE0BlxY5ilvpcHDSuAFwtQk6MP0AlzLSiqzQQCc0Bj93rn6F%7EG0VKgkN8LAc6Z%7EYP3%7E2ZtuWo5KuoUsqXujB7UnzZTlrW3ZR5ODVWUB%7Ed6AWHlh7KLtczSKJTtLaC6vBOE0Tt9XFwX0sFazctlizlf-s6AQn26tMi0WryOQVXvrOGg__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=1361551944476193
Resolving download.llamameta.net (download.llamameta.net)... 108.138.51.32, 108.138.51.79, 108.138.51.128, ...
Connecting to download.llamameta.net (download.llamameta.net)|108.138.51.32|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-20 23:25:34 ERROR 403: Forbidden.

Checking checksums
md5sum: checklist.chk: no properly formatted checksum lines found

Then, I used a script available in the LLaMA repository (https://github.com/facebookresearch/llama/blob/main/download.sh), which allowed me to correctly download the 7b model. However, after executing the "inference.py" command in the console, I ran into another issue that I'm unable to resolve. It seems to me that the problem might be related to the model weights, but I don't have the means to download others. Below is the error message:

E:\pytorch-llama-main>python inference.py
Loading checkpoint "llama-2-7b\consolidated.00.pth"
Loaded checkpoint in 4.49s
Traceback (most recent call last):
  File "E:\pytorch-llama-main\inference.py", line 155, in <module>
    model = LLaMA.build(
  File "E:\pytorch-llama-main\inference.py", line 32, in build
    model_args: ModelArgs = ModelArgs(
TypeError: __init__() got an unexpected keyword argument 'rope_theta'

No need to use forward method?

pytorch-llama/model.py

Lines 230 to 235 in 067f8a3

	# (B, Seq_Len, Dim) + (B, Seq_Len, Dim) --> (B, Seq_Len, Dim)
	h = x + self.attention.forward(
	self.attention_norm(x), start_pos, freqs_complex
	)
	# (B, Seq_Len, Dim) + (B, Seq_Len, Dim) --> (B, Seq_Len, Dim)
	out = h + self.feed_forward.forward(self.ffn_norm(h))

No need to use forward method?
I mean, we could use nn.Module directly.

h = x + self.attention(self.attention_norm(x), start_pos, freqs_complex)
out = h + self.feed_forward(self.ffn_norm(h))  

what is the license of the code please?

what is the license of the code please?