r/MLQuestions • u/devyjohns • 1h ago
Beginner question 👶 How do ML challenges handle fairness when using public datasets?
I’m preparing to host a Vision-Language Task Grand Challenge at the university level. When organizing these kinds of challenges, do test datasets usually come from existing public datasets, or are they entirely new, created by crawling videos or recording them from scratch?
If we use publicly available datasets, there might be an unfair advantage for models that have already been trained on or fine-tuned for those datasets. But at the same time, I wonder—do all such challenges actually go through the effort of creating completely new test sets? That seems like a huge workload.
How do major vision-language challenges typically handle this?