抄録RD-003
Designing Test Collections That Provide Tight Confidence Intervals
Designing Test Collections That Provide Tight Confidence Intervals
○酒井哲也（早大）
In this study, we show a method for determining the topic set size n for researchers who are trying to build a new test collection, by requiring a tight confidence interval (CI) for the difference between any given pair of systems X and Y.
By applying this method with estimates of the population variance from past data (i.e., existing test collections), more reliable test collections can be built for similar tasks.
Specifically, we show that evaluation measures should be chosen at the test collection design phase, and that
a tight CI can be achieved at a low cost by having many topics with shallow document pools.
By applying this method with estimates of the population variance from past data (i.e., existing test collections), more reliable test collections can be built for similar tasks.
Specifically, we show that evaluation measures should be chosen at the test collection design phase, and that
a tight CI can be achieved at a low cost by having many topics with shallow document pools.
