So, this is a good idea. Having opinions about what AGI benchmarks should look l... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

curious_cat_163 on June 12, 2024 | parent | context | favorite | on: ARC Prize – a $1M+ competition towards open AGI pr...

So, this is a good idea. Having opinions about what AGI benchmarks should look like is a great way to argue about the kind of technology we want to build for the future.

However, why are the 100 test tasks secret? I don't understand why how resisting “memorization” techniques requires it. Maybe someone can enlighten me.

muglug on June 12, 2024 | [–]

If the tasks were public then it would be trivial to have a human figure out the answers, and then to train an LLM to memorise those answers.

andoando on June 12, 2024 | [–]

Test date is always a secret no, otherwise you can train it on the test data and prod your algo to match the results closely as possible

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact