LeaderBoard for submissions so far? about calib_challenge

[haraschax]

If you need feedback to figure out whether your solution is any good, you're probably going about this the wrong way. The zip you submitted has score of over 100%, which means predicting zeros would have been better. There are several ways you can test your own solutions without the answers.

Posting which methods were successful isn't really in the spirit of this challenge, especially since it's a challenge we still use for hiring. Finding a strategy is the fun part :). Though I can say about 5 people have achieved the under 25% benchmark.

[neel04]

5 people have achieved the under 25% benchmark.

5? 😨
Oh no, no feedback - just wanted to test out my submissions here some times before emailing them; Clearly, overfitting is happening here despite good performance on validation and training set.

I am not sure whether iterating over a solution is considered valid; I simply wish to try a bunch of things out and see which one makes the <25% cut on test set. Its really a problem not understanding whether my solution performs well due to the disjoint nature of the train/test set - so all I can try is having multiple submissions to understand how well my methods work.

Right now, clearly my methods are off - but at least now I have a feeling as to the next approach I wish to try. Iterating over solutions - isn't this the engineering spirit @ CommaAI? 🚗

[haraschax]

Trying multiple strategies is of course a good approach. But you need to find your own ways of validating them. In the real world evaluating solutions can often be harder problem than writing the solution, and that is exactly what this challenge intends to simulate.

[neel04]

@HaraldSchafer I was wondering if someone has a simple idea they wanted to try in something like OpenPilot (for real-world impact) would there be a way to communicate that with the CommaAI team training models? For ex. I had a naively simple idea for reducing memory consumption, inference speed and sequence length for transformers used in world-modelling (which I presume CommaAI does for its FSD goals) - anyway I can pitch through, or is the only way doing this challenge and getting an internship; talking to the right people internally? 😅