Order effect in interactive information retrieval evaluation: An empirical study
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Order effect in interactive information retrieval evaluation : An empirical study. / Clemmensen, Melanie Landvad; Borlund, Pia.
In: Journal of Documentation, Vol. 72, No. 2, 2016, p. 194-213.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Order effect in interactive information retrieval evaluation
T2 - An empirical study
AU - Clemmensen, Melanie Landvad
AU - Borlund, Pia
PY - 2016
Y1 - 2016
N2 - Purpose – The purpose of this paper is to report a study of order effect in interactive information retrieval (IIR) studies. The phenomenon of order effect is well-known, and it is the main reason why searches are permuted (counter-balanced) between test participants in IIR studies. However, the phenomenon is not yet fully understood or investigated in relation to IIR; hence the objective is to increase the knowledge of this phenomenon in the context of IIR as it has implications for test design of IIR studies.Design/methodology/approach – Order effect is studied via partly a literature review and partly an empirical IIR study. The empirical IIR study is designed as a classic between-groups design. The IIR search behaviour was logged and complementary post-search interviews were conducted.Findings – The order effect between groups and within search tasks were measured against nine classic IIR performance parameters of search interaction behaviour. Order effect is seen with respect to three performance parameters (website changes, visit of webpages, and formulation of queries) shown by an increase in activity on the last performed search. Further the theories with respect to motivation, fatigue, and the good-subject effect shed light on how and why order effect may affect test participants’ IR system interaction and search behaviour.Research limitations/implications – Insight about order effect has implications for test design of IIR studies and hence the knowledge base generated on the basis of such studies. Due to the limited sample of 20 test participants (Library and Information Science (LIS) students) inference statistics is not applicable; hence conclusions can be drawn from this sample of test participants only.Originality/value – Only few studies in LIS focus on order effect and none from the perspective of IIR.Keywords Evaluation, Research methods, Information retrieval, User studies, Searching, Information searches
AB - Purpose – The purpose of this paper is to report a study of order effect in interactive information retrieval (IIR) studies. The phenomenon of order effect is well-known, and it is the main reason why searches are permuted (counter-balanced) between test participants in IIR studies. However, the phenomenon is not yet fully understood or investigated in relation to IIR; hence the objective is to increase the knowledge of this phenomenon in the context of IIR as it has implications for test design of IIR studies.Design/methodology/approach – Order effect is studied via partly a literature review and partly an empirical IIR study. The empirical IIR study is designed as a classic between-groups design. The IIR search behaviour was logged and complementary post-search interviews were conducted.Findings – The order effect between groups and within search tasks were measured against nine classic IIR performance parameters of search interaction behaviour. Order effect is seen with respect to three performance parameters (website changes, visit of webpages, and formulation of queries) shown by an increase in activity on the last performed search. Further the theories with respect to motivation, fatigue, and the good-subject effect shed light on how and why order effect may affect test participants’ IR system interaction and search behaviour.Research limitations/implications – Insight about order effect has implications for test design of IIR studies and hence the knowledge base generated on the basis of such studies. Due to the limited sample of 20 test participants (Library and Information Science (LIS) students) inference statistics is not applicable; hence conclusions can be drawn from this sample of test participants only.Originality/value – Only few studies in LIS focus on order effect and none from the perspective of IIR.Keywords Evaluation, Research methods, Information retrieval, User studies, Searching, Information searches
KW - Faculty of Humanities
KW - Evaluation
KW - Research methods
KW - Information retrieval
KW - User studies
KW - Searching, Information searches
U2 - 10.1108/JD-04-2015-0051
DO - 10.1108/JD-04-2015-0051
M3 - Journal article
VL - 72
SP - 194
EP - 213
JO - Journal of Documentation
JF - Journal of Documentation
SN - 0022-0418
IS - 2
ER -
ID: 150713165