bug: substract with overflow when compiling on debug mode about miden-vm

[Fumuran]

Hi @mFragaBA, thank you for the bug report!

As far as I can say this error occurs because procedures encrypt_shuffle and remask_shuffle in the player.masm don't specify the number of locals, both of them should have at least 2. Without it program tries to calculate the indexes for the local values, which causes the subtraction with underflow. We should add a proper check in this place to print the real error message.

Although it is still unclear why the —release version works fine, probably it is connected with some optimizations of the compiler and assembler. Are you sure that you use encrypt_shuffle and remask_shuffle procedures during the test?

[mFragaBA]

Are you sure that you use encrypt_shuffle and remask_shuffle procedures during the test?

Yeap! Notes shuffle.masm and remask.mask (they call encrypt_shuffle and remask_shuffle) get consumed during the test.

BTW, thanks for debugging the issue! I tried for a bit but I'm not too familiar with the compiler internals yet

[bobbinth]

Although it is still unclear why the —release version works fine,

In release mode, the compiler don't emit overflow/underflow checks as far as I now.

We should add a proper check in this place to print the real error message.

Yes - let's do that.

[bobbinth]

Closed by #1372.