Workshop Program
WSDM Cup Workshop is co-located with the WSDM conference and will be held on February 10, 2017.
Schedule
| 14:00-14:45 | Keynote: Fabian Suchanek |
| 14:45-15:30 | Keynote: Lydia Pintscher |
| 15:30-15:45 | Overview of the Vandalism Detection Task Stefan Heindorf, Martin Potthast, Gregor Engels, and Benno Stein |
| 15:45-16:00 | Overview of the Triple Scoring Task Hannah Bast, Björn Buchhold, and Elmar Haussmann |
| 16:00-16:15 | Poster Setup |
| 16:15-17:15 | Poster Session All participants are invited to present their work as a poster |
| 17:15-17:30 | Discussion |
| 17:30 | Social Dinner |
The workshop program is part of the WSDM conference program.
Important Dates Anywhere on Earth
| September 1, 2016 | Training data release |
| October 15, 2016 | Early bird software submission (optional) |
| December 8, 2016 | Final software submission |
| December 30, 2016 | Final software submission (extended) |
| January 1, 2017 | Announcement of evaluation results |
| January 21, 2017 | Paper submission |
| February 6-10, 2017 | Conference and WSDM Cup workshop |
Benefits for Early Birds
If you submit a working prototype of your software by October 15, 2016, you will be able to tune it against a validation dataset. Based on the results obtained on the validation dataset, you may revise and resubmit your software for the final software submission deadline by December 8, 2016.
If you register for the WSDM conference by a still to-be-determined date, you may do so at a reduced fee.
Award Terms
To be eligible to receive an award for one of the tasks, participating teams have to agree to these terms and comply with them.
- A team is a group of people who have registered for the WSDM Cup 2017. A team is eligible for an award if it fulfills all below requirements.
- Every member of an eligible team is a member of at most one team per task.
- A scientific paper has been submitted to the WSDM Cup 2017 workshop by the eligible team, describing the approach implemented in detail.
- The eligible team or a designated representative attends the WSDM Cup 2017 workshop held at the WSDM 2017 conference and presents the scientific paper in an oral talk or a poster session as decided by the organizers of the WSDM Cup 2017.
- A working version of the testing component of the software submitted by an eligible team has been deployed to TIRA and the organizers of the WSDM Cup 2017 have used it to successfully replicate the results achieved.
- The source code of the software submitted by an eligible team has been released in full, including the components for training and testing, as well as sufficient documentation to compile and run it, in a repository located on the GitHub organization of WSDM Cup 2017. The source code shall include third party materials required to compile and run it, where applicable, and otherwise refer to them in the documentation, describing how to obtain them. If a preprocessing of third party materials is required, the corresponding source code must be included and documented. The use of proprietary third party materials is allowed as long as it can be publicly obtained by anyone.
- The source code of the software submitted by an eligible team has been published under an open source software license chosen from https://opensource.org/licenses. It is the sole responsibility of the authors of the software to ensure that the chosen license is in compliance with the licenses of all third party materials used. The authors of the software must ensure that no copyrights of third parties are infringed by publishing their source code and by granting the organizers of the WSDM Cup 2017 usage rights as specified below. In cases where third party rights are infringed or where a third party raises such claims, the authors of the source code assume full responsibility, take over all charges, and agree to indemnify the organizers of the WSDM Cup 2017 of all costs they may incur on such an occasion.
- Eligible teams grant the organizers of the WSDM Cup 2017 the absolute an irrevocable right and permission to use, re-use, publish, and re-publish their code, their deployed software, and their paper, in full or in part, in any form and medium known now or hereafter, only for non-commercial purposes, to achieve the scientific goals of the WSDM Cup 2017, also after the workshop has passed, each time including the team's members' names and affiliations to indicate the authorship of their source code, software, and paper.
- Eligible teams have complied with all additional rules set for the individual tasks.
- Eligible teams have not attempted to cheat.
- The organizers of the WSDM Cup 2017 have no legal obligation to award prizes and retain the right to withdraw from awarding prizes and to change the award terms at any time. The choice of awardees is at their full discretion and may include criteria not listed here or on the task web pages.
- Recourse to the courts is excluded.
Open Source Proceedings
We maintain a code repository for the WSDM Cup 2017 on GitHub at github.com/wsdm-cup-2017. Since many participants have expressed interest to share their code with the community, our GitHub organization provides for a central place to do so.
How to get access?
Viewing the repositories is simple; just click the button below and you can browse the various code repositories available so far. To download a given code repository, you will need to log in with your GitHub account, or sign up for a new one. After that, simply follow the checkout instructions provided by GitHub.
Why share at all?
Many researchers do not share their code cause it's not polished, they have no time to provide support, or for fear of competition. This may be true, yet, researchers who share their code may have a lasting impact on their community. Others who adopt their code will cite it, ultimately adding to the sharer's reputation. Sharing is thinking long-term.
How to share my code?
To get started, send us an email with the name of your GitHub account. We will create a team for you at our GitHub page. You may add as many team members to your team as you like, or be part of multiple teams. After that, you can create repositories our organization and commit to them.
What are the terms?
- Authors retain copyright of all their code. We assume no ownership.
- The choice of license rests with the authors. Respect the license.
- When you use some code, cite the authors. If you don't know how, ask.
- When you find bugs, fix them and share bugfixes with the authors.
Software Submission
It is our goal to carry out reproducible research. While the evaluation resources created for our shared tasks benefit future evaluations, even shared tasks lack reproducibility. Therefore, we ask participants to submit working executables of their software. To facilitate the submission of software, we employ TIRA (tira.io), a cloud-based evaluation platform that implements the evaluation as a service paradigm.
How to get access?
To get started, send us an email with your preferred choice of operating system. We offer Ubuntu Linux 14 LTS (Desktop or Server) as well as Windows 7. We will set up a virtual machine (VM) for you and pass you the access credentials. You can access your VM via SSH and remote desktop. Once you have access, please install your software inside the virtual machine.
Why submit software?
Reproducibility is one of the cornerstones of science; yet, in recent years it was shown that many scientific contributions are irreproducible. For shared tasks in particular, the software developed by participants often go missing soon after the event has passed. Moreover, the test datasets used in shared tasks are released to participants, creating an evaluation bias. All of these problems are solved by submitting software instead of their run output.
What to prepare?
To work with TIRA, your software has to be executable from the command line and it must accept the following parameters:
- A path to an input dataset.
- A path to an output directory.
- A path to the output of a previous run of the software (optional).
Once your software is installed in your VM, you can remote control its evaluation via TIRA's web interface.
What are the terms?
- Authors retain copyright of their software.
- Authors provide usage rights to us and TIRA for the purpose of evaluation.
- Authors may revoke usage rights at any time.
- Usage of TIRA is at the author's risk.
