Introduce JudgeLM (and brief discussion to finalized the structure) about distilabel

[dvsrepo]

cool!
and let's rename to UltraFeedbackRating too?

[alvarobartt]

Hey here! Nice idea to add JudgeLM, I think that we can have a separate judgelm.py file with JudgeLMRating for the moment and then explore whether there's a better solution, which probably there is, so lets explore that before the release!

[dvsrepo]

@alvarobartt now I see the initial naming alignment, they are *Preference so: UltraFeedbackPreference, JudgeLMPreference 😃

[alvarobartt]

IMO the most compliant naming as of now are probably UltraFeedbackTask and JudgeLMTask? WDYT? We can do UltraFeedbackPreference and JudgeLMPreference otherwise, but since those are already imported from distilabel.tasks.preference maybe the "preference" part does not need to be present there?

[dvsrepo]

Maybe even remove the *Task?

[alvarobartt]

I'd keep it because we are assigning that to an arg named task so I feel like it's more intuitive to just go with task=JudgeLMTask rather than task=JudgeLM, WDYT?

[dvsrepo]

perfect