Advice on how to substitute manual loops with this library about rust_dct

[ejmahler]

(std::f32::consts::PI * freq * i / 1024.0).cos()

This looks like pi*j*k/2*n, where n is 512. There is no discrete cosine transform or discrete sine transform that maps to this core formula. It also doesn't map to the FFT, because the FFT is 2*pi*j*k/n, IE the 2 is in the numerator instead of the denominator.

If you plugged your data into a FFT of size 4n (using rustfft or some equivalent library), that would get the 2 onto the correct side of the fraction.My intuition says that wouldn't quite do what you wanted though, because the twiddle factors of the full FFT will end up leaking in. But it's worth a shot - create a complex buffer of size 4n, populate the real components of the first n elements with your data, run the fft, and pull the output data out of the first n real elements of the output.

I'm very curious where they got this algorithm from. My intuition says that it's custom, but it would be very easy to hand-derive a matrix factorization - easier, in fact, than the DCT.

[torokati44]

Oh, wow, you were exactly right! 😮

I quickly checked this code:

pub fn fft2(mut hist: [f32; 512]) -> [f32; 512] {
    use rustfft::num_complex::Complex32;
    use rustfft::FftPlanner;

    let mut planner = FftPlanner::new();
    let fft = planner.plan_fft_forward(2048);

    let mut h2 = [Complex32::from(0.0); 2048];
    for i in 0..512 {
        h2[i] = Complex32::from(hist[i]);
    }

    fft.process(&mut h2);

    for i in 0..512 {
        hist[i] = h2[i].re;
    }
    hist
}

And it matches down to 4 fractional digits! Hats off to you!
Thank you for the explanation!

Truth be told, I already sidestepped this problem by making cosf itself "faster":
ruffle-rs/ruffle#9657

But armed with this knowledge, I might make it even better sometime in the future!

[ejmahler]

Great! I'm glad it didn't end up needing a manual factorization.

So yeah, if you end up actually switching to rustFFT to this, I recommend using realfft instead, which wraps RustFFT in an algorithm that cuts the work in half for workloads where all the imaginary components are zero.

[torokati44]

I recommend using realfft instead

Alright, thank you!

I'm very curious where they got this algorithm from. My intuition says that it's custom

Well, it very well might be. The entire computeSpectrum method does not look that well thought out, consistent, or tested. 😶
And many think, for good reason, that this is true for the entirety of Flash Player's code.