strange value on audioArray about lively

[rocksdanister]

Yeah I think I messed up the audio processing.
For reference, its based on this:
https://github.com/filoe/cscore/tree/master/Samples/WinformsVisualization

Although those values are stranger than normal, can you try in a different system?

I'll have a look after a quick update for pathData this week.

[Tanuki33]

i can't test on different system atm.
but hey i think, i am already fix this issue..


yeah the comma ( , ) split the array..
here stringify one

This happens because Visual Studio (or C #?) Uses the currency format on the system,
and guess what? i follow this to Set up dot instead of comma in numeric values

yeah! solved! :D

oh wait, the array value is still problematic .. this because fftsize is too small (i think?) i just changed fftsize to 4096 and the problem was solved.

[rocksdanister]

4096 is very high interms of idle cpu usage.
It looks like its a culture issue with the cscore library, I'll set livelycefsharps thread to Invariantculture/en-US.
What is your system language?

[Tanuki33]

i am using English language but my currency and time are Indonesia.

[Allanfd12]

for some reason, the audioArray is showing several repeated values. In the image below, I drew the values of the array, red is bigger, blue is smaller, the y-axis is time. thus, it is clear that the first 11 values are always the same, and the 10 following, and then the other 9.

I understand that the audioArray has only 128 positions, but I expected some image similar to this:

where each position of the vector has an independent value. I just don't understand, why when closer to the end, the numbers stop repeating, and each 1 or 2 positions has a new value.

both images represent the same sound segment, but the bottom one does not show repetition in the audio array, and has a higher resolution of approximately 512 values.

I would really love 512 values, or even 1024, but for now, I am satisfied with 128 values without repetition, do you have any idea how to improve this?

[rocksdanister]

Yeah the value is off, while accuracy is not the priority since its meant to run realtime without much overhead .. something is still off.
All I'm doing is passing the fft data from cscore library so currently I'm not sure what's the cause.

I would really love 512 values, or even 1024, but for now, I am satisfied with 128 values without repetition, do you have any idea how to improve this?

Higher value is avoided to avoid high cpu usage, I can however add support for external json config file for the initialization class.

[ricksanche2]

from

private void Timer_Tick1(object sender, EventArgs e)
        {
                 ...
          
                        var fftBuffer = new float[(int)fftSize];
                        fftBuffer = _lineSpectrum.livelyGetSystemAudioSpectrum();
                        if(fftBuffer != null)
                            ExecuteScriptAsync("livelyAudioListener", fftBuffer);
                 ...
        }

to

public float[] livelyGetSystemAudioSpectrum()
        {
            var fftBuffer = new float[(int)FftSize];
            if (SpectrumProvider.GetFftData(fftBuffer, this))
            {
                //return fftBuffer;
            }
            else
              ##{
                System.Diagnostics.Debug.WriteLine("livelygetspectrum error");
                return null;
            }

            SpectrumPointData[] spectrumPoints = CalculateSpectrumPoints(2, fftBuffer);
            float[] res = new float[spectrumPoints.Length];
            for (int i = 0; i < spectrumPoints.Length; i++)
            {
                res[i] = (float)spectrumPoints[i].Value;
            }

            return res;
        }

to
protected virtual SpectrumPointData[] CalculateSpectrumPoints(double maxValue, float[] fftBuffer)

The data that is being passed to javascript is not fft data, it's SpectrumPointData[] that is a transformation of the fft result. And because the nature of FastFourierTransform, the fftSize transform yields fftSize/2 size frequency bin data in the case of real valued input(which is the case for audio capture). Meaning fftSize of 128 give 64 frequency bins that hold type float magnitude of those bins. And what protected virtual SpectrumPointData[] CalculateSpectrumPoints(double maxValue, float[] fftBuffer) does is scale this frequency bin data either by decibel scale or linear scale or squareRoot scale. Note that it is doing this for fftSize/2 which is 64, but in the code the bar count is 128, so data points necessarily becomes duplicated. And 64 bins is not a lot to divide the 24000hz(sampling rate/2) frequency range, so at the beginning(low frequency end) the fft result itself is very close if not the same. Add to that there is weird useAverage and xLogScale ..

Anyway, the CPU usage is not really related to fft calculation. I used FFTW(fastest fft) instead of CSCore implementation and I used a mock array of floats, the transmission of data is the problem.Because chrome itself can do fft(size2048) using trivial amount of CPU and fftw on itself can do it using similar.

I thought about using WebRTC to stream captured audio to chrome and it can handle fft on its own. Do you find that viable?
The maintainer of CefSharp suggested

The other option would be to use a SchemeHandlerFactory/ResourceHandler possibly you should stream the data using a range request or go with the old school AJAX requests and pull the data

ResourceHandler has low overhead and can transfer large amounts of data.

If the fft is done on the C# side, it should conform to web audio API. That way users can find random javascript visualizer(codepen, jsFiddle,github..) and plug it in without the drawing getting out of wack due to different transform of fft result. For example, web audio API first apply a blackman window, and after fft the result is smoothed and converted to dB.

[ricksanche2]

I'm really new here, and I want to help. I will try out the two aforementioned method of passing data and if there is anything I should know let me know.

[rocksdanister]

@ricksanche2

The data that is being passed to javascript is not fft data, it's SpectrumPointData[] that is a transformation of the fft result.

Lol that was a very Jerry thing of me to do.

I thought about using WebRTC to stream captured audio to chrome and it can handle fft on its own. Do you find that viable?
The maintainer of CefSharp suggested

Preferably backward compatibility should be maintained so existing wallpaper like Fluid work wont break.

If the fft is done on the C# side, it should conform to web audio API. That way users can find random javascript visualizer(codepen, jsFiddle,github..) and plug it in without the drawing getting out of wack due to different transform of fft result. For example, web audio API first apply a blackman window, and after fft the result is smoothed and converted to dB.

That would be cool.. even could try injecting js code to make them work without any user change.

I'm really new here, and I want to help. I will try out the two aforementioned method of passing data and if there is anything I should know let me know.

Not a lot other than wallpaper players being separate programs:
https://github.com/rocksdanister/lively/wiki/Building

Help is appreciated.