Towards reproducibility in recommender-systems research

Beel, Joeran; Breitinger, Corinna; Langer, Stefan; Lommatzsch, Andreas; Gipp, Bela

Towards reproducibility in recommender-systems research

Dateien

Beel_0-324818.pdfGröße: 1.76 MBDownloads: 1248

Datum

2016

Autor:innen

Beel, Joeran

Breitinger, Corinna

Langer, Stefan

Lommatzsch, Andreas

Gipp, Bela

Open Access-Veröffentlichung

Open Access Green

Sammlungen

Informatik und Informationswissenschaft

Publikationstyp

Zeitschriftenartikel

Publikationsstatus

Published

Erschienen in

User Modeling and User-Adapted Interaction : umuai. 2016, 26(1), pp. 69-101. ISSN 0924-1868. eISSN 1573-1391. Available under: doi: 10.1007/s11257-016-9174-x

Zusammenfassung

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system research. We conduct experiments using Plista’s news recommender system, and Docear’s research-paper recommender system. The experiments show that there are large discrepancies in the effectiveness of identical recommendation approaches in only slightly different scenarios, as well as large discrepancies for slightly different approaches in identical scenarios. For example, in one news-recommendation scenario, the performance of a content-based filtering approach was twice as high as the second-best approach, while in another scenario the same content-based filtering approach was the worst performing approach. We found several determinants that may contribute to the large discrepancies observed in recommendation effectiveness. Determinants we examined include user characteristics (gender and age), datasets, weighting schemes, the time at which recommendations were shown, and user-model size. Some of the determinants have interdependencies. For instance, the optimal size of an algorithms’ user model depended on users’ age. Since minor variations in approaches and scenarios can lead to significant changes in a recommendation approach’s performance, ensuring reproducibility of experimental results is difficult. We discuss these findings and conclude that to ensure reproducibility, the recommender-system community needs to (1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments, (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research.

Fachgebiet (DDC)

004 Informatik

Zitieren

ISO 690

BEEL, Joeran, Corinna BREITINGER, Stefan LANGER, Andreas LOMMATZSCH, Bela GIPP, 2016. Towards reproducibility in recommender-systems research. In: User Modeling and User-Adapted Interaction : umuai. 2016, 26(1), pp. 69-101. ISSN 0924-1868. eISSN 1573-1391. Available under: doi: 10.1007/s11257-016-9174-x

BibTex

@article{Beel2016-03-12Towar-33528,
  year={2016},
  doi={10.1007/s11257-016-9174-x},
  title={Towards reproducibility in recommender-systems research},
  number={1},
  volume={26},
  issn={0924-1868},
  journal={User Modeling and User-Adapted Interaction : umuai},
  pages={69--101},
  author={Beel, Joeran and Breitinger, Corinna and Langer, Stefan and Lommatzsch, Andreas and Gipp, Bela}
}

RDF

<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/33528">
    <dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/>
    <dc:contributor>Gipp, Bela</dc:contributor>
    <dc:creator>Lommatzsch, Andreas</dc:creator>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dc:contributor>Breitinger, Corinna</dc:contributor>
    <dcterms:title>Towards reproducibility in recommender-systems research</dcterms:title>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2016-04-01T08:21:57Z</dc:date>
    <dcterms:abstract xml:lang="eng">Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system research. We conduct experiments using Plista’s news recommender system, and Docear’s research-paper recommender system. The experiments show that there are large discrepancies in the effectiveness of identical recommendation approaches in only slightly different scenarios, as well as large discrepancies for slightly different approaches in identical scenarios. For example, in one news-recommendation scenario, the performance of a content-based filtering approach was twice as high as the second-best approach, while in another scenario the same content-based filtering approach was the worst performing approach. We found several determinants that may contribute to the large discrepancies observed in recommendation effectiveness. Determinants we examined include user characteristics (gender and age), datasets, weighting schemes, the time at which recommendations were shown, and user-model size. Some of the determinants have interdependencies. For instance, the optimal size of an algorithms’ user model depended on users’ age. Since minor variations in approaches and scenarios can lead to significant changes in a recommendation approach’s performance, ensuring reproducibility of experimental results is difficult. We discuss these findings and conclude that to ensure reproducibility, the recommender-system community needs to (1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments, (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research.</dcterms:abstract>
    <dc:creator>Gipp, Bela</dc:creator>
    <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/33528/1/Beel_0-324818.pdf"/>
    <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/33528/1/Beel_0-324818.pdf"/>
    <dc:contributor>Langer, Stefan</dc:contributor>
    <dc:rights>terms-of-use</dc:rights>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/33528"/>
    <dc:contributor>Beel, Joeran</dc:contributor>
    <dc:creator>Breitinger, Corinna</dc:creator>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dc:creator>Langer, Stefan</dc:creator>
    <dc:contributor>Lommatzsch, Andreas</dc:contributor>
    <dc:creator>Beel, Joeran</dc:creator>
    <dcterms:issued>2016-03-12</dcterms:issued>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2016-04-01T08:21:57Z</dcterms:available>
    <dc:language>eng</dc:language>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
  </rdf:Description>
</rdf:RDF>

Universitätsbibliographie

Ja