Publications [#258512] of Li Ma
Papers Published
- Ma, L; Mease, D; Russell, DM. "A four group cross-over design for measuring irreversible treatments on web search tasks." Proceedings of the Annual Hawaii International Conference on System Sciences (March, 2011). [doi]
(last updated on 2025/07/05)Abstract:
When trying to measure the effect of irreversible treatments such as training interventions, the choice of the experimental design can be difficult. A two group cross-over experimental design cannot be used due to longitudinal effects during the course of the experimental run, which can be especially large in dynamic web search environments. A standard case/control two group design also can be problematic because it is negatively impacted by variability among participants. Our solution uses a four group cross-over design that combines features from standard cross-over and case/control designs. We illustrate the effectiveness of this design through a case study in which participants are shown a video on how to use "control-F" to search for text within a web page. We quantify the improvement of our four group cross-over design compared to the standard case/control two group design with respect to measuring the effect of this video on participants. Finally, we compare the magnitude of our estimated "control-F" effect to similar studies on web ranking and user-interface changes, revealing that teaching this skill produces a potentially large improvement in web searchers' ability to search rapidly.