Differences between LLVM const-eval of floats and runtime behaviour can cause miscompilations about rust

[beetrees]

@RalfJung Non-deterministic NaNs can cause miscompilations. Here is an example that works on x86-64 (playground):

#[inline(never)]
fn print_vals(x: f64, i: usize, vals_i: u32) {
    println!("x={x} i={i} vals[i]={vals_i}");
}

#[inline(never)]
pub fn evil(vals: &[u32; 300]) {
    const INC: f64 = f64::MAX / 90.0;
    let mut x: f64 = -1.0;
    let mut i: usize = 0;

    while x.is_sign_negative() {
        print_vals(x, i, vals[i]);
        // Use a big decrement to reach negative infinity quickly.
        x -= INC;
        // Convert infinity into a NaN (Inf - Inf = NaN)
        x = x + x - x;
        i += 2;
    }
}

pub fn main() {
    let mut vals: [u32; 300] = [0; 300];
    for i in 0..300 { vals[i as usize] = i; }
    evil(&vals);
}

On x86-64 (and 32-bit x86), NaNs produced at runtime will always have a negative sign whereas NaNs produced by LLVM at compile time always have a positive sign. This means LLVM thinks the loop will exit when x becomes a NaN, but at runtime the loop will never exit.

[beetrees]

@rustbot label +A-floating-point +A-llvm +I-unsound +A-cross

[apiraino]

WG-prioritization assigning priority (Zulip discussion).

@rustbot label -I-prioritize +P-high

[saethlin]

This different sin implementations problem was previously reported as #109118

[ds84182]

Shouldn't an LLVM issue be filed for this? Ideally the results of transcendentals should be represented by a "fuzzy" range, making compile time float equality impossible in this case.

[oli-obk]

llvm/llvm-project#89885 was filed on the llvm side

[RalfJung]

Ah, fun. So LLVM assumes that sin is deterministic. Yeah that's clearly incoherent.

Now I wonder if the non-determinism in NaNs (as specified in rust-lang/rfcs#3514) is subject to similar issues. If LLVM assumes float operations are deterministic then indeed it must not do any const-folding that differs from the effective runtime behavior.

[RalfJung]

@beetrees thanks for constructing an example!