Distribution Format Specification
Question Format - Organizer will distribute topics(questions) in this format
Gold Standard Format - Organizer will distribute gold standard in this format
- Evaluation Result Format - Evaluation result will be distributed in TREC-compatible format just like the previous ACLIA. Details TBA.
Submission Format Specification
Run ID Format - Run ID definition for Question Analysis, IR4QA and CCLQA output submission
Question Analysis Format - All ACLIA participants are recommended to submit question analysis results in this format
IR4QA Format - IR4QA (Embedded CLIR) participants will submit their output (retrieval results) in this format
CCLQA Format - CCLQA participants will submit their output (answers) in this format. Also use this format for the IR4QA+CCLQA collaboration run.
- Technical Description - Short description about your system. TBA.
Format Checker - Useful tools to check if your output follows the format specification.
Question Analysis Restrictions
Please submit one answer type per topic per run. You can submit up to 3 runs, but we may evaluate the first run only.
We will accept up to 1,000 documents IDs, although we may not be able to evaluate all of them.
Submit at least one T-run (where only QUESTION field is used).
You can submit up to 3 runs, but the first run (01) should be the T-run as mandatory. You can submit up to top 30 answers for each topic. However, due to resource constraints, we may not be able to evaluate all 30 answers in each topic.
See RunIDFormat for more details.
Distribution files are encoded in UTF-8. Please encode submission files in UTF-8 as well.
Xinhua Corpus Doc ID
Xinhua corpus has different DOCID format depending on distributions (even among different versions of Chinese Gigawords). Use the official corpus named LDC2009E75, which has the prefix "XIN_CMN_".
Changes from the previous ACLIA
- Supporting document ID is mandatory for CCLQA
- Topic ID prefix "ACLIA1" has been changed to "ACLIA2"
- Output xml should contain dependencies among runs (if any) under the metadata field.