WSDM Cup Workshop is co-located with the WSDM conference and will be held on February 10, 2017.
|14:00-14:45||Keynote: Fabian Suchanek|
|14:45-15:30||Keynote: Lydia Pintscher|
|15:30-15:45||Overview of the Vandalism Detection Task|
Stefan Heindorf, Martin Potthast, Gregor Engels, and Benno Stein
|15:45-16:00||Overview of the Triple Scoring Task|
Hannah Bast, Björn Buchhold, and Elmar Haussmann
All participants are invited to present their work as a poster
The workshop program is part of the WSDM conference program.
|September 1, 2016||Training data release|
|October 15, 2016||Early bird software submission (optional)|
|December 8, 2016||Final software submission|
|December 22, 2016||Announcement of evaluation results|
|January 21, 2017||Paper submission|
|February 6-10, 2017||Conference and WSDM Cup workshop|
If you submit a working prototype of your software by October 15, 2016, you will be able to tune it against a validation dataset. Based on the results obtained on the validation dataset, you may revise and resubmit your software for the final software submission deadline by December 8, 2016.
If you register for the WSDM conference by a still to-be-determined date, you may do so at a reduced fee.
To be eligible to receive an award for one of the tasks, participating teams have to agree to these terms and comply with them.
We maintain a code repository for the WSDM Cup 2017 on GitHub at github.com/wsdm-cup-2017. Since many participants have expressed interest to share their code with the community, our GitHub organization provides for a central place to do so.
Viewing the repositories is simple; just click the button below and you can browse the various code repositories available so far. To download a given code repository, you will need to log in with your GitHub account, or sign up for a new one. After that, simply follow the checkout instructions provided by GitHub.
Many researchers do not share their code cause it's not polished, they have no time to provide support, or for fear of competition. This may be true, yet, researchers who share their code may have a lasting impact on their community. Others who adopt their code will cite it, ultimately adding to the sharer's reputation. Sharing is thinking long-term.
To get started, send us an email with the name of your GitHub account. We will create a team for you at our GitHub page. You may add as many team members to your team as you like, or be part of multiple teams. After that, you can create repositories our organization and commit to them.
It is our goal to carry out reproducible research. While the evaluation resources created for our shared tasks benefit future evaluations, even shared tasks lack reproducibility. Therefore, we ask participants to submit working executables of their software. To facilitate the submission of software, we employ TIRA (tira.io), a cloud-based evaluation platform that implements the evaluation as a service paradigm.
To get started, send us an email with your preferred choice of operating system. We offer Ubuntu Linux 14 LTS (Desktop or Server) as well as Windows 7. We will set up a virtual machine (VM) for you and pass you the access credentials. You can access your VM via SSH and remote desktop. Once you have access, please install your software inside the virtual machine.
Reproducibility is one of the cornerstones of science; yet, in recent years it was shown that many scientific contributions are irreproducible. For shared tasks in particular, the software developed by participants often go missing soon after the event has passed. Moreover, the test datasets used in shared tasks are released to participants, creating an evaluation bias. All of these problems are solved by submitting software instead of their run output.
To work with TIRA, your software has to be executable from the command line and it must accept the following parameters:
Once your software is installed in your VM, you can remote control its evaluation via TIRA's web interface.