How should incomplete timestamps be handled?
If the publish timestamp or date doesn’t have time zone information, we assume the latest possible time zone (Anywhere on Earth) to be the time zone. If the publish time only has the date, we assume the latest possible time (23:59:59) to be the publish time.
How can the publish time of a data source be determined?
The collection time for large scale datasets is sometimes provided in the datasheet in the paper. For datasets that grow in size (e.g., Common Crawl, CC-NEWS), the publish time is usually one of the accessible fields. Note some don’t have timezone information.
How can the release time of a pre-trained model be determined?
Aside from finding the release time on Github or Hugging Face, the model date is often specified on the model card in the paper. For example, Gopher’s model card can be found here and it indicates a model date of December 2020 which is acceptable. Chinchilla used the same training dataset as Gopher, so it’s also acceptable even though it was released in 2022.
For your convenience, the following models were verified by organizers and can be used without additional proof: BERT, XLNet, T5-v1.1, DeBERTa models, UnifiedQA models, GPT-3 (001 models), LaMDA, Gopher, Chinchilla, or models that trained only on the pre-training data used by the aforementioned models.
Where do we download data and submit results?
See the Getting Started page.
How many submissions can each team enter each competition phase?
Only one account per team can be used to submit results. Creating multiple accounts to circumvent the submission limits will result in disqualification.
Are participants required to share the details of their method?
We encourage all participants to share their methods and code, either with the organizers or publicly. To be eligible for prizes, teams must open source their methods, code, models, data, and any additional information to reproduce the experimental results.
What are the current rules?
They can be found on the official rules page.
Can the organizers change the rules?
Yes. We reserve the right to change the rules for any reason. This area is relatively new and adjustments may need to be made to the competition as we go along.
How do I contact the organizers?
Don't hesitate to contact us at forecasting@mlsafety.org.
When will prizes be distributed?
The winning teams will be announced after the organizers verify that the shared code and models of top submissions are legitimate. Prize money will be distributed as soon as possible after the winning teams are announced.
