Critical analysis on the reproducibility of visual quality assessment using deep features

Götz-Hahn, Franz; Hosu, Vlad; Saupe, Dietmar

Critical analysis on the reproducibility of visual quality assessment using deep features

Dateien

Goetz-Hahn_2-1slsft0xvpsz07.pdfGröße: 1.86 MBDownloads: 76

Datum

2022

Open Access-Veröffentlichung

Open Access Gold

Sammlungen

Informatik und Informationswissenschaft

Publikationstyp

Zeitschriftenartikel

Publikationsstatus

Published

Erschienen in

PLoS ONE. Public Library of Science (PLoS). 2022, 17(8), e0269715. eISSN 1932-6203. Available under: doi: 10.1371/journal.pone.0269715

Zusammenfassung

Data used to train supervised machine learning models are commonly split into independent training, validation, and test sets. This paper illustrates that complex data leakage cases have occurred in the no-reference image and video quality assessment literature. Recently, papers in several journals reported performance results well above the best in the field. However, our analysis shows that information from the test set was inappropriately used in the training process in different ways and that the claimed performance results cannot be achieved. When correcting for the data leakage, the performances of the approaches drop even below the state-of-the-art by a large margin. Additionally, we investigate end-to-end variations to the discussed approaches, which do not improve upon the original.

Fachgebiet (DDC)

004 Informatik

Zitieren

ISO 690

GÖTZ-HAHN, Franz, Vlad HOSU, Dietmar SAUPE, 2022. Critical analysis on the reproducibility of visual quality assessment using deep features. In: PLoS ONE. Public Library of Science (PLoS). 2022, 17(8), e0269715. eISSN 1932-6203. Available under: doi: 10.1371/journal.pone.0269715

BibTex

@article{GotzHahn2022Criti-59105,
  year={2022},
  doi={10.1371/journal.pone.0269715},
  title={Critical analysis on the reproducibility of visual quality assessment using deep features},
  number={8},
  volume={17},
  journal={PLoS ONE},
  author={Götz-Hahn, Franz and Hosu, Vlad and Saupe, Dietmar},
  note={Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 251654672 –TRR 161 (Project A05). Article Number: e0269715}
}

RDF

<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/59105">
    <dc:rights>Attribution 4.0 International</dc:rights>
    <dc:language>eng</dc:language>
    <dcterms:rights rdf:resource="http://creativecommons.org/licenses/by/4.0/"/>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2022-11-10T13:22:25Z</dcterms:available>
    <dcterms:abstract xml:lang="eng">Data used to train supervised machine learning models are commonly split into independent training, validation, and test sets. This paper illustrates that complex data leakage cases have occurred in the no-reference image and video quality assessment literature. Recently, papers in several journals reported performance results well above the best in the field. However, our analysis shows that information from the test set was inappropriately used in the training process in different ways and that the claimed performance results cannot be achieved. When correcting for the data leakage, the performances of the approaches drop even below the state-of-the-art by a large margin. Additionally, we investigate end-to-end variations to the discussed approaches, which do not improve upon the original.</dcterms:abstract>
    <dc:creator>Hosu, Vlad</dc:creator>
    <dc:creator>Saupe, Dietmar</dc:creator>
    <dcterms:title>Critical analysis on the reproducibility of visual quality assessment using deep features</dcterms:title>
    <dc:creator>Götz-Hahn, Franz</dc:creator>
    <dc:contributor>Saupe, Dietmar</dc:contributor>
    <dc:contributor>Hosu, Vlad</dc:contributor>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/59105"/>
    <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/59105/1/Goetz-Hahn_2-1slsft0xvpsz07.pdf"/>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dc:contributor>Götz-Hahn, Franz</dc:contributor>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2022-11-10T13:22:25Z</dc:date>
    <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/59105/1/Goetz-Hahn_2-1slsft0xvpsz07.pdf"/>
    <dcterms:issued>2022</dcterms:issued>
  </rdf:Description>
</rdf:RDF>

Kommentar zur Publikation

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 251654672 –TRR 161 (Project A05).

Universitätsbibliographie

Ja

Begutachtet

Ja