Preview

PDF, English (Dissertation) - main document Download (4MB) | Terms of use

Abstract

The assessment of highly domain-general problem solving skills is increasingly important as problem solving is increasingly demanded by modern workplaces (e.g., Autor, Levy, & Murnane, 2003) and increasingly present in international large-scale assessments such as the Programme for International Student Assessment (PISA, e.g., OECD, 2014). This thesis is about the computer-based assessment of problem solving skills based on Multiple Complex Systems (MCS, Greiff, Fischer, Stadler, & Wüstenberg, 2014): The main idea of the MCS approach is to present multiple computer-simulations of “minimally complex” problems (Greiff, 2012) in order to reliably assess certain problem solving skills. In each simulation, the problem solver has to interact with a problem in order to find out (a) how to adequately represent the problem, and (b) how to solve the problem. Up to now, two instances of the MCS approach have been proposed: (1) the MicroDYN approach (based on simulations of linear equation systems) and – more recently, in the second paper of this thesis – (2) the MicroFIN approach (based on simulations of finite state machines). In the current thesis I will elaborate on three research questions regarding the validity (cf. Bühner, 2006) of the MCS approach: (1) its content validity with regard to the concept of complex problem solving; (2) the convergent validity of different instances of the MCS approach; (3) the discriminant validity of the interactive problems of the MCS approach with regard to traditional static measures of reasoning and analytic problem solving skills. Each research question will be addressed in one corresponding paper:

In a first paper (Fischer, Greiff, & Funke, 2012) complex problem solving is defined as the goal-oriented control of systems that contain multiple highly interrelated elements. After reviewing some of the major strands of research on complex problem solving (e.g., research on strategy selection, information reduction, intelligence, or on the interplay of implicit and explicit knowledge in the process of complex problem solving) a theoretical framework outlining the most important cognitive processes involved in solving complex problems is derived. The theoretical framework highlights both interactive knowledge acquisition (problem representation) and interactive knowledge application (problem solution) as the two major phases in the process of complex problem solving. Both phases are represented in all current instances of the MCS approach.

In a second paper (Greiff, Fischer et al., 2013) the convergent validity of MicroDYN and MicroFIN is investigated (thereby introducing MicroFIN as an alternative to MicroDYN) in order to demonstrate that both instances address the same kind of problem solving skills. Based on a multitrait-multimethod analysis of a sample of university students (N = 339) it is demonstrated that – in addition to method-specific skills – both instances assess a common set of skills (method-general traits) related to (1) representing and (2) solving different kinds of interactive problems. In a regression of science grades on reasoning and the skills assessed by the instances of the MCS approach it is demonstrated that only the method-general representation trait and reasoning have substantial unique contributions. Thus, MicroDYN and MicroFIN seem to address a common set of skills and this set of skills is relevant for explaining school grades in science classes even beyond reasoning.

In a third paper (Fischer et al., in press) the discriminant validity of the interactive MicroDYN test is investigated by relating it to reasoning and traditional static measures of Analytic Problem Solving skills (APS) as they were applied in PISA 2003 (OECD, 2004). Besides a common core of problem solving skills addressed by both kinds of tasks (e.g., analyzing complex information about the information given at a certain moment in time) Fischer et al. (in press) expected to find evidence for additional skills that were related to interactive problems only (e.g., systematically generating information and interactively testing hypotheses). Results indicate that MicroDYN shares a lot of variance with APS even after controlling for reasoning in a sample of high-school students (N = 577) and the university student sample (see above). With regard to the explanation of school grades MicroDYN had an incremental value compared to reasoning and APS in the high-school student sample but not significantly so in the university student sample (whereas APS had an incremental value in both samples).

Basically these findings highlight both potential and limitations of the MicroDYN approach in its current form. Current instances of the MCS approach address a small set of problem solving skills reliably, but it takes more than these skills to competently solve complex problems. Implications for future research on the assessment of problem solving skills are discussed.