sigsep / sigsep-mus-2018-analysis Goto Github PK

Hey,

Here's a function I'm using to segment the main SiSEC 2018 results generated by this repo, based on the track sample times (csv file) generated by sigsep-mus-previews:

def segment_sisec18_given_previews(sisec18, previews, sample_rate=44100):

    def _filter(track):

        preview = previews[previews.iloc[:, 0].isin(track.track)]

        start_time = preview.iloc[0, 1] // sample_rate
        end_time = preview.iloc[0, 2] // sample_rate
        loc = ((track['time'] >= start_time) & (track['time'] <= end_time))

        return track[loc]

    sisec18 = sisec18[sisec18.track.isin(previews.iloc[:, 0])]

    track_group = sisec18.groupby('track')
    sisec18 = track_group.apply(_filter)

    return sisec18

Just thought you might find it helpful.

If it would be useful here, let me know where you would like it adding and I'll set up a pull request.

Hello,

I noticed that the Conover post-hoc tests yields very close to zero p-values, which seems a bit unusual as the box plots show quite a lot of overlap in SDR values between the methods... Could this be because it is currently computed by taking a vector of all segment-wise observations from each method and comparing them, ignoring that some segments are correlated because they belong to the same song? This is how it looks to me at least:

sp.posthoc_conover(df_voc, val_col='score', group_col='estimate')

I don't know stats very well, but could it be that we need to apply a blocked design, in which the segment-wise observations from the same song are put into one block? I think block assignments are supported by the conover method.

Hey,

Thanks again for all the great work on SiSEC 2018!

Quick one:

When computing summary statistics and comparing distributions of the BSS Eval measures across algorithms, e.g. as done here, are you or do you plan on aggregating the scores over time first (the framewise measures)?

I would not, for example, compare two algorithms by using the pooled data (e.g. all SAR values over time and songs), but rather take the medians of the framewise measures to obtain a set of per-song median scores from which I can compare.

I'm also a little scared about those large negative scores I've seen at the start/end of a song.

Cheers