RESEARCH ARTICLE


Cognitive Loads and Training Success in a Video-Based Online Training Course



Klaus D. Stiller*, Annamaria Köster
Department of Educational Science, University of Regensburg, Regensburg, Germany


Article Metrics

CrossRef Citations:
4
Total Statistics:

Full-Text HTML Views: 4716
Abstract HTML Views: 3000
PDF Downloads: 1377
ePub Downloads: 973
Total Views/Downloads: 10066
Unique Statistics:

Full-Text HTML Views: 1997
Abstract HTML Views: 1482
PDF Downloads: 916
ePub Downloads: 611
Total Views/Downloads: 5006



Creative Commons License
© 2017 Stiller and Köster.

open-access license: This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0), a copy of which is available at: (https://creativecommons.org/licenses/by/4.0/legalcode). This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

* Address correspondence to this author at the Department of Educational Science, University of Regensburg, 93040 Regensburg, Germany, Tel: +49 941 943 3719, Fax: 49 941 943 2450; E-mail: klaus.stiller@ur.de


Abstract

Background:

According to research based on cognitive load theory, the way of presenting information in an instructional environment is essential to the learning outcome. By avoiding unnecessary extraneous load caused by badly designed instructions and other sources, learners are more likely to successfully construct knowledge. In addition, learner characteristics are known to affect learning.

Objective:

This study explores the effects of learners’ online learning experience, domain-specific prior knowledge, computer attitude and computer anxiety on their perceived intrinsic, extraneous and germane load and on their learning outcome in a video-based training course about media design for employees.

Method and Results:

Learning outcome was assessed by ratings of subjective learning success, ratings of professional competence, the number of completed modules and performance. None of the learning outcome variables could be modelled when entering learner characteristics in a regression analysis, but all could be modelled using the cognitive load ratings.

Conclusion:

Thus, extraneous, intrinsic and germane load were the most important factors for explaining the learning outcome. This result points to the importance of instructional design and particularly to managing cognitive load in online training scenarios.

Keywords: Online training, Computer attitude, Computer anxiety, Prior knowledge, Online learning experience, Cognitive load.



1. INTRODUCTION

Nowadays, online learning is established in higher education and vocational training in addition to the traditional ways of learning and teaching [1]. Although ample theoretical and practical knowledge about online learning can be found in the literature, the problem of fostering successful student learning persists [2]. One research focus that can inform educators is to explore the extent that learners’ characteristics and skills related to cognitive, motivational and affective processes as well as observable usage behavior directly determine learning outcomes. The following study explores the effects of various learner characteristics on cognitive load while learning and learning outcomes in a video-based online training course about media design for employees. To determine which learner characteristics should be involved, we combined characteristics that have already shown to be empirically relevant for (online) learning and that address motivational, affective, cognitive and skill aspects. Thus, computer attitude, computer anxiety, domain-specific prior knowledge and online learning experience were selected. Our investigation was conducted against the background of the Cognitive Load Theory (CLT) and cognitive load assessments [3-5].

1.1. Cognitive Load Theory

CLT emphasizes the role of working memory in learning and the problem of occupying working memory by useful und unnecessary cognitive processes, that is, cognitive loads [3-5]. The cognitive processes executed in working memory are thought to determine the effectiveness of learning. Only when information processing occurs without disturbances can learners build an adequate mental representation that will be stored effectively in long-term memory. Working memory has a limited capacity, which allows processing of only a few information elements at a time. If rehearsal of memory items is not possible, items fade within seconds. Learning complex tasks is often difficult because of the working memory limitations. Learners must hold numerous elements (e.g. rules and states) in working memory while relating them, which often exceeds working memory capacity. The components of cognitive load are intrinsic, germane and extraneous loads. Intrinsic load is commonly assumed to be created by the difficulty of the subject matter related to the number of elements to be learned and their interactivity [3]. Intrinsic load is the basic amount of processing required for understanding a presentation. Germane load goes beyond understanding and is bound to processing information used to build schemas and store them in long-term memory [3]. Extraneous load is assumed to result from the presentation manner of the material and is often the main source that inhibits learning, because it is unrelated to the construction or automation of schemas [3]. Successful learning occurs when working memory capacity is not overburdened by overall cognitive load and when as much capacity as is available can be dedicated to schema acquisition and automation, which creates germane load. Cognitive overload is mostly created by extraneous and intrinsic load.

1.2. Assessments of Cognitive Load

Martin [4] proposed a classification of cognitive load measuring methods with respect to causal relationship (direct vs. indirect) and objectivity (objective vs. subjective). Causality of methods describes whether the measures directly reflect the effects of cognitive load or are indirectly affected by them via an intermediary link. Objectivity of methods describes whether the measures reflect subjective, self-reported data or objective assessments of behavior, physiological conditions or performance. Using indirect and direct subjective measures of cognitive load combined with indirect objective measures is a widely used combination for measuring cognitive load in cognitive load research, whereas direct objective assessments are rarely used [3, 4].

1.2.1. Direct, Subjective Assessments of Intrinsic, Extraneous and Germane Cognitive Load

Subjective ratings of material difficulty (a method to assess intrinsic load) or the presentation format of the material (a method to assess extraneous load) are proposed for measuring the cognitive load of the learning material [4, 5]. Often, these loads are measured by one-item ratings [4, 5]. de Jong [3] summarized a multitude of issues related to subjective cognitive load ratings, particularly when using one-item rating scales. He questioned the validity, reliability and sensitivity of the instrument, particularly the variation of scales used in self-report assessments (e.g., number of points, anchor terms), and how often and when self-reporting is used (e.g., while or after learning). Furthermore, scores are not consistently interpreted (e.g., as a high or low cognitive load score) and easily comparable (e.g., when scores are associated with poor performance conditions in one study and with high performance conditions in another study) across studies. These aspects should be considered when interpreting results involving cognitive load.

One surprising finding is the limited exchange between cognitive load research and research on learning strategies in the literature [4]. A long history of theory and research on learning strategies in general and in specific situations exists. Conceptualizations and diagnostics in that field could be fruitful for cognitive load research, especially when measuring germane load. Germane load could be assessed directly and subjectively by assessing learning strategies, especially elaboration. Elaboration strategies aim at an active integration of new information with prior knowledge [6]. Techniques like paraphrasing sentences, creating images corresponding to texts, finding analogies, looking for examples or referring new information to personal experience belong to this category. Learning strategies are commonly assessed by using multiple items. Instruments that are sufficiently reliable and valid have been developed for measuring learning strategies [7].

1.2.2. Perceived Usability as a Direct, Subjective Measure of Extraneous Load

In online learning, the learning environment could especially create extraneous load because of the difficulty in managing it, commonly referred to as a lack of usability. The extraneous load results from the mental processing power needed to handle an online learning environment. Thus, the assessment of usability requiring students to rate the difficulty of managing a learning environment could be categorized as a direct, subjective measure of extraneous load. For measuring usability, well developed instruments exist that are sufficiently reliable and valid [8]. This type of extraneous load is distinguished from the extraneous load created by the presentation of the material to be learned, because it is produced by another source of load, namely the learning environment.

Usability of software is an international concern reflected by standards set by the International Organization for Standardization (ISO). Central for usability are the standards defined in ISO 9241 ‘Ergonomics of human-system interaction’ [9], which states that usability is the “extent to which a product can be used by specific users to achieve specific goals with effectiveness, efficiency and satisfaction in a specific context of use” (ISO 9241 - Part 11). In particular, Part 110 of ISO 9241 focuses on principles related to the ergonomic design of the dialogue between a user and an interactive system [10]. The design principles are formulated in general terms without reference to situations of use, application, environment or technology. ISO 9241-110 is regarded as applicable to all types of interactive systems, thus also to online courses. Seven features are listed in ISO 9241-110 (see Table 1) [8]. Applying the ISO definition of usability to online learning environments, a more specific definition could be the extent to which an environment can be used by learners to achieve their learning goals with effectiveness, efficiency and satisfaction in a training context.

Table 1. The seven design aspects according to ISO 9241 - Part 110.
Principles Explanations
Suitability for the task An interactive system is suitable for the task when it supports the user in the completion of the task, i.e. when the functionality and the dialogue are based on the task characteristics (rather than the technology chosen to perform the task).
Self-descriptiveness A dialogue is self-descriptive to the extent that at any time it is obvious to the users which dialogue they are in, where they are within the dialogue, which actions can be taken and how they can be performed.
Conformity with user expectations A dialogue conforms with user expectations if it corresponds to predictable contextual needs of the user and to commonly accepted conventions.
Suitability for learning A dialogue is suitable for learning when it supports and guides the user in learning to use the system.
Controllability A dialogue is controllable when the user is able to initiate and control the direction and pace of the interaction until the point at which the goal has been met.
Error tolerance A dialogue is error-tolerant if, despite evident errors in input, the intended result may be achieved with either no or minimal corrective actions by the user.
Suitability for individualization A dialogue is capable of individualization when users can modify interactions and presentations of information to suit their individual capabilities and needs.

1.2.3. Learning Outcome Measures as Indirect Objective and Subjective Load Measures

Learning outcome measures are often used as dependent variables in educational studies to, for example, show the effectiveness of training methods or to analyze which learner characteristics lead to learning success [11-20]. Assessments of performance can be viewed as indirect, objective measures of cognitive load, because performance indirectly depends on storage and retrieval processes that may be affected by cognitive load [3, 4]. Although performance measures bear a great amount of uncertainty for interpreting cognitive load [3], they are perfectly suitable for their intended purpose of measuring performance. Performance measures could then be particularly used to correlate other load measures with it to clarify which load factors might influence performance. The number of completed modules in an online training analogously reflects such an indirect, objective measure.

Indirect, subjective measures of performance, such as self-reported learning success or self-reported competence development, are also used in field studies. These assessment types could especially be administered when performance is not easily assessed, when feedback given to students is not very differentiated, when the learner’s personal perspective can provide important information about motivational or evaluative aspects of the learning environment or when an additional personal view on performance might be fruitful. Performance assessments might reliably reflect learning success, but subjective reports might more easily reflect, for example, the relevance of the learned material (i.e., whether students believe that they have learned something important, useful, practical or relevant for their personal development). One main problem in general with self-report questionnaires is the difference between learners’ current judgments and their actual behavior [4]. Research on judgements of learning shows that learners can be biased in their judgement [21]. Judgements may be based on what learners have learned, but they might also be influenced by subjective experiences like interest, frustration, flow or invested effort [21]. Nevertheless, in the context of academic teaching and evaluation, subjective self-reports of competencies and learning success have been successfully and systematically developed with acceptable reliability and validity [22-25] and in accordance with performance measures [22]. Paechter et al. [22, pp. 223], for example, focused on learning achievements and described them “as different facets of competences such as theoretical and methodical knowledge as well as skills required for problem solving, personal/social competences (e.g., in self-regulated or collaborative learning), and/or media competence.” These subjective self-reports are expected to be more adequate for evaluating learning outcomes than simple judgements of one-item percentage or x-point rating scales [21].

1.3. Learner Characteristics Influencing Learning

1.3.1. Computer Attitude and Anxiety

Attitudes consist of affective, conative and cognitive components [26]. From a cognitive perspective, attitudes are often defined as beliefs that are organized in topics, hence the computer as a self-experienced instrument for working and learning might be of interest concerning learning [26]. Attitudes toward computers are assumed to reflect a bipolar structure [26]. A person might concurrently have negative and positive beliefs about the same point. Computer attitude is a learner feature that might affect working memory load while learning. Negative computer attitudes might engender a more extraneous load because of, for example, disturbing thoughts about the computer malfunctioning or even crashing. Learners with positive attitudes should be less prone to such thoughts and thus less likely to suffer from extra working memory load. As a result, the learner is able to adequately engage in information processing and knowledge construction. Only a few studies have been conducted on the effects of computer attitudes on learning performance. These studies have found negative effects of negative attitudes [19, 20] and positive effects of positive attitudes (e.g., toward e-learning, internet use, information technology, or technology use) on course usage and persistence [27].

Similarly, computer anxiety is also a learner feature that might influence online learning. Computer anxiety has been defined as a trait, which comprises both cognitive and affective components such as feelings of anxiety and worrisome thoughts [26]. The anxiety might engender a more extraneous load because the learner must cope with negative emotions and negative thoughts about the computer. In contrast, learners without computer anxiety are less likely to suffer from extra working memory load and are assumed to adequately engage in information processing and knowledge construction. To date, most studies have focused on the relationships between anxiety, computer self-efficacy, and performance or learning system usage (also computer usage), mostly assuming that anxiety has only an indirect negative effect on performance and course usage but has a direct influence on self-efficacy which directly influences performance and course usage [13, 17, 18].

1.3.2. Online Learning Experience and Domain-specific Prior Knowledge

The learner’s experience with online learning might also affect working memory load while learning. Online learning experience could comprise various aspects that accompany participation in online learning courses such as knowledge about online learning portals and their usage, typical course processes, and adequate learning strategies. From this point of view, having some online learning experience could help a learner expend less working memory resources on determining, for example, the correct usage of an online portal, when and what action should be performed and how to perform an efficient learning action in the course progression. Hence, experienced learners might be able to concentrate more on learning than less experienced learners. Good design and usability are particularly important for learners with a lack of experience in online training. Ample evidence exists in the literature showing positive effects of online learning experience on performance and course persistence [12, 16, 18, 28, 29].

The domain-specific prior knowledge is known to influence program usage, information processing and performance [11]. Studies from hypertext research have reported that prior knowledge has a positive impact on a diversity of performance measures [11, 19, 20, 30, 31]. Research based on CLT has revealed expertise reversal effects in a variety of instructional designs [14]. An expertise reversal effect refers to all instructional design effects that are dependent on prior knowledge and result in a learning environment that is effective for learners with little prior knowledge but ineffective for learners with considerable prior knowledge. In experienced learners, guidance of information processing by presented instructions conflicts with guidance by prior knowledge. These two active guidance strategies overlap in components that are cross-related and integrated and thus generate unnecessary load, because ignoring the instructional guidance is difficult. Inexperienced learners lack the schemata to help them handle new information. Consequently, well-designed instructions can guide information processing by serving as a substitute for a schema [15]. Given that “the involvement of different (schema-based and instruction-based) cognitive constructs for dealing with the same units of information may consume sufficient resources to cause cognitive overload compared with instruction that relies more heavily on pre-existing schemas for guidance” [16, p. 24], eliminating instruction-based guidance for experienced learners has been recommended. One approach to eliminating components of instruction-based guidance is to allow learners to control the pace and sequence of information presentation [20], which is a common feature in online learning courses. Hence, typical online learning scenarios should be adequate for inexperienced and experienced learners, resulting in a positive correlation between prior knowledge and performance [19, 20, 31] but a non-existing correlation when the course aims at mastery learning.

2. RESEARCH QUESTIONS

The purpose of the study was to explore the relations between learner characteristics, cognitive loads and learning outcome in a video-based training course. The study addressed the three research questions.

(1) To what Extent do the Selected Learner Characteristics Account for Cognitive Load?

We expected that a higher level of online learning experience would result in a lower level of extraneous load because of existing schemas about how online learning environments might function and how to learn with videos. Correspondingly, higher scores in negative computer attitude and computer anxiety should result in a higher level of extraneous load because of task irrelevant cognitions. A lower level of extraneous load should result in more available capacity that could be used for germane load. Thus, the levels of online learning experience, computer attitude and computer anxiety are likely to be associated with germane load levels. Intrinsic load is assumed to mainly depend on prior knowledge. The more prior knowledge, the less intrinsic load experienced by the learners.

(2) To what Extent does Cognitive Load Account for Learning Outcomes?

A lower level of extraneous load should result in higher learning outcomes, because the learner will be able to focus on the learning task. Correspondingly, allocating memory resources to germane processes should directly result in higher learning outcomes. In contrast, a higher level of intrinsic load should result in higher learning outcomes, because of exercising more effort in learning to overcome a greater challenge.

(3) To what Extent do the Selected Learner Characteristics Account for Learning Outcomes?

Overall, positive levels of online learning experience, domain-specific prior knowledge, attitude towards computers and computer anxiety should lead to experiencing less overall load (either less extraneous load or intrinsic load; see above), resulting in more learning and consequently higher learning outcomes.

3. METHOD

3.1. Participants

The training course was a continuous vocational training on media design for employees. To develop and promote the training, we announced that partner enterprises would support employee participation in the training. The training was promoted online on various relevant message boards and pages (including press releases) and offline via flyers (distributed in numerous chambers of crafts). A total of 128 employees signed up for the course, of which 26 never logged in and 44 dropped out without sending any solution to the course’s module tasks. The reaming 58 employees worked on at least one module task, of which 28 provided full datasets and 30 failed to answer the questionnaires. The mean age of the 28 employees was 38 years (SD = 12). Twenty-three employees were female (82.1%) and 5 were male (17.9%). Most employees had a post-secondary school diploma (n = 16, 57.1%), followed by a secondary school or commercial college certificate (n = 9, 32.1%) and an entrance qualification for universities for applied sciences (n = 2, 7.1%). One employee (3.6%) had a secondary modern school-leaving certificate. The size of the enterprises spread from one-man enterprises (21.4%, n = 6) up to enterprises having more than 250 employees (10.7%, n = 3). Most enterprises had between two and nine employees (39.3%, n = 11). The most reported branches were freelance, economic or technical services (n = 5) and health and social systems (n = 5).

3.2. Description of the Online Course

The online course is exclusively an online training course without a fixed schedule that uses asynchronous tools for communication between learners and tutors and among learners. It consists of 13 modules (e.g., image design, designing flyers and posters, social media, and open source). The core element of each module is an instructional video lasting about 15 minutes. Learners are asked to complete different module tasks to construct knowledge. In the current study, the evaluation and feedback for each module task solutions were given by the module expert. The expected workload to successfully complete the training was 20 to 25 hours. In addition, an introductory module informed the employees about the content, technical requirements and course organization.

The starting point is a Moodle course portal. The module pages are composed of three sections: course issues, content area, and navigation area. The course issues provide information about the module’s tutor and the student (if added), and an overview of the student’s learning performances of the modules. The navigation area provides an overview of the module’s content and direct access to all content sections. The content area comprises the following six parts: (1) The module profile gives an overview of the content and the teaching objectives. (2) The video presents the contents. A content includes a typical real world problem, exemplified by an employee of an enterprise who attempts to solve it according to clearly formulated goals. The components of the solution procedure are presented to the learner in sequence. Components could be design standards and rules and actions that demonstrate them. Knowledge of design is utilized and skills are used to produce results. Below the video, learners are asked to evaluate the video. (3) Links to external resources and literature that could be used for further elaboration are presented in the supplements section. (4) The training task is used for controlling whether the module objectives were reached by the learners. (5) The support section provides access to an internal message board, which could be used when learners had questions about contents or tasks. (6) The media presentation section enables learners to provide access to their media products. The purpose in the current study was to generate discussion and receive feedback from tutors and other learners.

3.3. Procedure and Means of Measurements

3.3.1. Procedure

One week before starting the training, all registered employees were invited to the first online questionnaire. All employees of the sample except one answered the questionnaire before studying. The results of the study were not influenced by the one person who completed the questionnaire after already studying five of 13 modules. After starting, employees had 12 weeks to work on 13 modules. Each module provided a video questionnaire that was answered after studying the module’s video for the first time. A module was finished when the student submitted a task solution. Five weeks after starting, the participants were invited to the second questionnaire and at the end of the training to the final third questionnaire. The items of all questionnaires are listed in the supplementary file.

3.3.2. First Questionnaire

Employees were asked about their age, sex, level of education and the size and sector of the enterprise they work in. They then rated their experience with online training and instructional videos on 5-point Likert scales. Then their attitude towards computers focusing on the personal experience with using the computer as a means for learning and working was examined [26]. The negative component in the sense of the computer being regarded as an uncontrollable machine was measured by seven statements. Afterwards, computer anxiety was measured by eight statements referring to the cognitive and affective components [26]. All statements were rated on a 5-point Likert scale from agree to disagree.

3.3.3. Second Questionnaire

Usability was measured by using five of the seven scales from the ISONORM 9241-110 questionnaire (suitability for the task, self-descriptiveness, controllability, conformity with user expectations and suitability for learning) [8]. Each scale comprises five bipolar items, rated on a 7-point rating scale. The ends of the rating scale are represented by statements featuring opposite positions.

3.3.4. Third Questionnaire

The final questionnaire included a measure of subjective learning success, which was rated with eight statements concerning the quantity and quality of learning. Students were required to rate whether they could expand their knowledge and whether they had learned new important and useful information [23]. Professional competence was also assessed, which is declarative knowledge that comprises specific expertise such as knowledge about facts, concepts, theories, models and circumstances, as well as knowledge about correlations and principles [24]. Learners rated the extent that they agreed to seven statements on a 5-point Likert scale.

3.3.5. Video Questionnaires

The questionnaires included statements about the level of domain-specific prior knowledge, intrinsic load (difficulty of content) and extraneous load (comfort of presentation format). Participants were required to place check marks on 7-point rating scales. They also indicated the extent that they agreed on a 5-point Likert scale to 10 statements about the use of elaborative learning strategies (reflecting germane load).

3.3.6. Performance

Performance was measured by evaluating the module task solutions. Modules mostly provide transfer tasks, but retention tasks or both types of tasks can also be used for single modules. The transfer tasks require an application of the knowledge learned, thus the learners are instructed to either create a media product or evaluate a given product as the task solution. Scores were calculated as percent correct, because the number of possible points were different for each task.

3.3.7. Annotation to Analyses

The individual scores of the variables were calculated differently.

  1. Means of items were calculated for online learning experience, computer attitude, computer anxiety, subjective learning success and professional competence.
  2. A mean was calculated across the number of tests completed (from 1 to 13 possible scores) for domain-specific prior knowledge, intrinsic load, extraneous load, germane load and performance. Before, the score of germane load was calculated as the mean of items per module. These scores reflected the mean ratings of the modules and thus of the whole training.
  3. Means of items were also calculated for the usability scales. The scale scores strongly correlated with each other, thus the mean of scale scores per individual was used for further analysis. The mean score was considered to reflect the usability experience best.

A high score expresses a higher level of the feature except for computer attitude which indicates a low negative attitude (a high score could be thought of as a “positive” attitude). In addition, the number of modules a learner had completed was counted.

Table 2. Means and standard deviations and the potential score range of each assessment.
Number of items used for assessment Number of assessments an individual score is based on M SD n Range Cronbach’s alpha
Online learning experience 2 1 1.26 1.04 74 0-4 .60(1)
Computer attitude 7 1 4.26 .58 74 1-5 .79
Computer anxiety 8 1 1.73 .44 74 1-5 .66
Prior knowledge 1 1 to 13 4.16 1.08 51 1-7 Not calculated
Usability experience 5*5(2) 5(3) 5.12 1.03 38 1-7 .82 to .90(4)
Intrinsic load 1 1 to 13 4.30 1.23 51 1-7 Not calculated
Extraneous load 1 1 to 13 2.82 1.04 51 1-7 Not calculated
Germane load 10 1 to 13 4.63 .51 51 1-5 .62 to .88(5)
Success of learning 5 1 3.92 .75 36 1-5 .87
Professional competence 7 1 3.73 .73 36 1-5 .90
Number of completed modules 13 1 8.60 5.16 58 0-13 Not calculated
Performance 1(6) 1 to 13 81.47 17.59 58 0-100 Not calculated
Note.(1) Correlation of the two scale items. (2) 5 usability features were assessed by 5 items each. (3) Individual scores were calculated as the mean of the five usability scale scores. (4) Calculated per each usability scale. (5) Calculated per each module. (6) One training task per module.

4. RESULTS

All correlation and regression analyses were computed for the whole sample (n = 28) except for performance. The mean performance score is only meaningful for analyses when students have studied about the same number of modules. Thus, the sample decreased to n = 18 when setting the number of studied modules to at least 12. We therefore performed no detailed correlation analysis of performance, only considering correlations when regression analyses succeed in modelling performance.

4.1. Descriptive Statistics

Tabel (3) shows the descriptive statistics of variables. Overall, the sample reported to be less experienced in online learning, less negative attitudes towards the computer, a low level of computer anxiety and a medium level of prior knowledge. Students rated usability positively, and intrinsic, extraneous and germane load were reported to have been small to medium. Learning outcomes were generally positive.

Table 3. Means and standard deviations (n = 28).
M SD Range
Online learning experience .98 1.06 0-4
Computer attitude 4.32 .62 1-5
Computer anxiety 1.66 .42 1-5
Prior knowledge 3.71 .89 1-7
Usability experience 5.19 1.03 1-7
Intrinsic load 4.45 .92 1-7
Extraneous load 2.92 .91 1-7
Germane load 3.69 .44 1-5
Success of learning 3.91 .79 1-5
Professional competence 3.75 .70 1-5
Number of completed modules 10.64 4.10 0-13
Performance 92.80 23.48 0-100

4.2. Analysis of Correlations Between Variables

The correlations between the learner characteristics, the load measures and the outcome variables are shown in Table (4). In total, 55 correlations were computed. Cumulative Alpha error controlled according to the Bonferroni-Holm correction [32] lead to three significant correlations (i.e. the values of r = .56, r = -.57 and r = .67). This result can be expected because of the small sample size, n = 28. Treating correlations smaller than .56 as null correlations is nevertheless not helpful with interpreting regression results. Alternatively, correlations could be viewed as effect sizes and only large effect sizes could be discussed. According to an empirical analysis by Gignac and Szodorai [33], correlations over r = .30 should be considered as relatively large. Adopting this view changes very little in comparison to the analysis using a non-corrected Alpha level of .05. The latter criterion was taken for discussing correlations.

Table 4. Correlations are shown above the diagonal and one-sided significance levels below (n = 28).
1 2 3 4 5 6 7 8 9 10 11
1 Online learning experience -.09 -.14 .45 .10 -.14 -.07 -.11 -.06 .05 .26
2 Computer attitude .327 -.57 -.15 -.08 -.03 -.12 .03 .22 .09 .27
3 Computer anxiety .238 .001 -.03 .10 -.12 -.11 -.01 -.21 -.10 -.31
4 Prior knowledge .009 .231 .433 .03 -.15 .02 -.14 -.19 -.06 .21
5 Usability .307 .350 .303 .441 -.22 -.55 .00 .56 .48 .32
6 Intrinsic load .239 .434 .280 .228 .128 .27 -.03 -.10 .13 .35
7 Extraneous load .353 .279 .297 .469 .001 .082 -.37 -.42 -.50 .08
8 Germane load .290 .431 .486 .245 .494 .437 .027 .38 .50 -.17
9 Subjective learning success .376 .133 .144 .166 .001 .310 .013 .024 .67 .29
10 Professional competence .398 .326 .315 .386 .005 .248 .004 .004 .001 .39
11 Number of completed modules .092 .086 .056 .138 .047 .034 .339 .190 .068 .020

4.2.1. Learner Characteristics

A clear pattern of correlations was found among the learner characteristics. Computer attitude significantly correlated r = -.57 with computer anxiety, thus the absence of negative attitudes is linked to the absence of computer anxiety. The second significant correlation occurred between online learning experience and prior knowledge, r = .45. The more online learning experience learners reported, the more prior knowledge they had.

4.2.2. Cognitive Load

Again, a clear pattern of correlations was found among the cognitive load variables. Usability experience significantly correlated (r = -.55) with extraneous load. The lower that learners rated the usability, the higher was their experience of extraneous load stemming from the learning material. The second significant correlation occurred between extraneous and germane load, r = -.37. A higher level of extraneous load corresponded with a lower level of germane load.

4.2.3. Learning Outcomes

Significant correlations could be found among the three outcome variables. Subjective learning success and professional competence were highly correlated, r = .67. Thus, the higher learners rated their success of learning, the higher was their experience of professional competence. Furthermore, the number of completed modules correlated insignificantly with subjective learning success (r = .29) and significantly with professional competence (r = .39).

4.2.4. Learner Characteristics with Cognitive Load and Learning Outcome

All correlations between the learner characteristics and the load variables were between r = -.15 and r = .10 and did not differ significantly from r = 0. Additionally, no significant correlations were found between success of learning and professional competence with the learner characteristics of interest. Only the number of completed modules correlated with online learning experience, computer attitude, and computer anxiety in the expected direction. More positive levels of these features corresponded to more completed modules.

4.2.5. Cognitive Load and Learning Outcome

Similar correlation patterns occurred for subjective learning success and professional competence. Both were positively correlated with usability experience and germane load and negatively correlated with extraneous load. Usability experience also correlated in the expected direction with the number of completed modules. In addition, the number of completed modules positively correlated with intrinsic load. The higher that participants rated content difficulty, the more modules they completed.

4.3. Multiple Linear Regression Analyses Using Backward Elimination

4.3.1. Learner Characteristics and Cognitive Load and Learning Outcomes

Neither usability experience, intrinsic load, extraneous load, and germane load nor subjective learning success, professional competence, number of completed modules and performance could be modelled by multiple linear regression with the learner characteristics entered as predictors (all full models: adjusted R2 ≤ .04, F ≤ 1.32, p ≥ .29).

4.3.2. Cognitive Load and Learning Outcomes

Subjective learning success could be modelled by multiple linear regression with usability and germane load entered as predictors (see Table 5). A high multiple correlation of .68 was found, leading to an explained variance of 46%. The strongest contribution to subjective learning success was usability (β = .56), followed by germane load (β = .38), the effect of which was about two-thirds of the effect of usability (see Table 6). Both variables have positive betas, meaning that the higher the usability and germane load scores, the higher the learners rated their subjective learning success.

Table 5. Summary of models.
Adjusted R2 SE F df1 df2 p
Subjective learning success .41 .61 10.52 2 25 .001
Professional competence .49 .50 9.76 3 24 .001
Number of completed modules .24 3.59 5.17 3 24 .013
Performance .19 3.63 4.92 1 17 .041

Similarly, professional competence could be modelled by usability, intrinsic and germane load, accompanied by an also high multiple correlation of .74 and an explained variance of .55. The contribution of usability and germane load were equally high (β = .54 and β = .50), and the contribution of intrinsic load was about half as strong as the contributions of the other variables (β = .27), but it only approached significance. All three variables have positive betas, meaning that the higher the usability, intrinsic and germane load scores, the higher the learners rated their professional competence.

The number of completed modules could be modelled with usability and intrinsic load as predictors. A high multiple correlation of .54 was found with an explained variance of .29. The contribution of usability and intrinsic load were equally high (β = .42 and β = .45). Both variables have positive betas, meaning the higher the usability and intrinsic load scores, the more modules were completed by the learners. Performance could be modelled with usability as the predictor. A correlation of .49 was found with an explained variance of .24. The higher the usability scores, the higher the performance scores.

Table 6. Results of the multiple linear regression analyses with backward elimination of variables.
B SE ß t df p
Subjective learning success (Constant) -.82 1.15 -.72 25 .48
Usability .43 .11 .56 3.81 25 .01
Germane load .67 .27 .38 2.54 25 .02
Professional competence (Constant) -1.99 1.11 -1.79 24 .09
Usability .37 .10 .54 3.86 24 .01
Intrinsic load .20 .11 .27 1.93 24 .07
Germane load .79 .22 .50 3.68 24 .01
Number of completed modules (Constant) -6.88 5.50 -1.25 25 .22
Usability 1.68 .69 .42 2.45 25 .02
Intrinsic load 1.97 .69 .45 2.58 25 .02
Performance (Constant) 74.80 5.71 3.10 16 .01
Usability 2.35 1.06 .49 2.22 16 .04

4.3.3. Self-Correlation, Homoscedasticity, and Non-collinearity

The self-correlation of residuals from the regression models were tested with the Durbin-Watson test, which computes values in the range of 0 to 4. Values close to 2 (commonly used is the interval of 1.5 to 2.5) are assumed to highlight the non-existence of self-correlations of residuals. Thus, the residuals are considered to be independent for subjective learning success (DW = 2.79) and professional competence (DW = 1.59), but the assumption is violated for the number of completed modules (DW = .69) and performance (DW = 3.18). Homoscedasticity, tested by the Koenker test [34], was not violated for subjective learning success (λ2 = 2.66, df = 2, ns), professional competence (λ2 = 3.40, df = 3, ns), the number of completed modules (λ2 = 3.09, df = 2, ns), and performance (λ2 = .26, df = 1, ns). Non-collinearity focuses on the correlations between predictors and could be controlled by the Variance Inflation Factor (VIF). Values near 1 are unproblematic, values that exceed 10 are often regarded as indicating multicollinearity, but values above 2.5 may be a cause for concern in weaker models. For all significant regression models including more than one predictor, the VIF values of subjective learning success (all VIF = 1.00), professional competence (all VIF values between 1.00 and 1.05) and number of completed models (all VIF values 1.05) indicated non-collinearity of the predictors that survived the backward elimination method.

5. DISCUSSION

The aim of the study was to investigate whether the learning characteristics, online learning experience, domain-specific prior knowledge, computer attitude and computer anxiety influence cognitive load when learning and thus contribute to learning outcomes. Generally, we expected that relevant online experience, considerable domain-specific prior knowledge, positive attitudes towards computers and less computer anxiety would lead to lower levels of overall load when learning, particularly extraneous and intrinsic load, which would in turn result in more learning and consequently higher learning outcomes. The results showed unexpectedly that learning outcomes did not depend on any of the learner characteristics, but the outcomes were influenced by cognitive load.

Subjective learning success, professional competence, the number of completed modules and performance could be modelled by regression with the load variables of experienced usability of the online training environment and intrinsic and germane load entered as predictors. Learners who had a lack of usability in an online learning system tended to experience higher levels of extraneous load, which resulted in less learning and lower performance. The result that perceived usability is positively correlated with learning outcomes is consistent with the dearth of research on usability and learning outcomes [22, 24, 35-38]. Moreover, germane load measured by rating strategies of elaboration was positively related to learning outcomes. The purpose of elaboration strategies is to integrate new information with prior knowledge [6]. Therefore, the finding that a higher level of germane load corresponded to higher ratings of subjective learning success and professional competence is not surprising. This result also corresponds to research on learning strategies [7]. In addition, participants passed more modules and felt more competent when they reported to have been confronted with higher intrinsic load (i.e., more difficult tasks). This finding urges us to suggest motivation to be considered in further studies. More challenging content might increase motivation to process the learning content and complete the modules. The result also corresponds to research on task difficulty and motivation, particularly as it pertains to effort and related physiological reactivity [39]. Given that subjective learning success and professional competence were highly correlated, it is not surprising that the regression models using them as response variables were very similar. Moreover, the number of completed modules was weakly correlated with success of learning and professional competence, which was reflected in a similar regression model.

None of the learning outcomes and cognitive loads could be modelled by using the learner characteristics. Why this unexpected result might have occurred [11-13, 17-20, 29-31]? This result could be due to a lack of variance in learner characteristics. One explanation could be that a highly homogeneous group of participants with respect to learner characteristics responded very similarly. Learners were widely free of anxiety and had a positive attitude toward computers with only little variance between participants in these characteristics. Online learning experience scores indicated that learners mostly had only little experience, and their self-reported prior knowledge was also on average in the lower range, thus contributing to the low variance found in learner characteristics. Moreover, only receiving the feedback of having past or failed a module might have hindered learners from rating subjective learning success and professional competence more carefully and with more differentiation.

CONCLUSION

For future practices, we recommend to concentrate on managing load by means of instructional design and didactics. Nevertheless, the results must be treated carefully. The small sample size and lack of experimental design in this field study warrants replication of results under more controlled conditions with more learners, which would also give the analyses more power. In addition, motivation was not assessed in the study, which could be an important variable contributing to the number of completed modules. Thus, it should be considered in further studies. Overall, approaching authentic online learning scenarios under a cognitive load perspective in a broader way was shown to be fruitful.

SUPPORTIVE/SUPPLEMENTARY MATERIAL

The items of the questionnaires used for assessment are listed in the supplementary file.

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

Not applicable.

HUMAN AND ANIMAL RIGHTS

No Animals/Humans were used for studies that are base of this research.

CONSENT FOR PUBLICATION

Not applicable.

CONFLICT OF INTEREST

The author confirms that this article content has no conflict of interest.

ACKNOWLEDGEMENTS

Declared none.

REFERENCES

[1] Allen IE, Seaman J. Online report card: Tracking online education in the United States. Babson Park, MA: Babson Research Group 2016.
[2] Jo I-H, Park Y, Yoon M, Sung H. Evaluation of online log variables that estimate learners' time management in a korean online learning context. International Review of Research in Open and Distributed Learning 2016; 17(1): 195-213. [http://dx.doi.org/10.19173/irrodl.v17i1.2176].
[3] de Jong T. Cognitive load theory, educational research, and instructional design: Some food for thought. Instr Sci 2010; 38: 105-34. [http://dx.doi.org/10.1007/s11251-009-9110-0].
[4] Martin S. Measuring cognitive load and cognition: Metrics for technology-enhanced learning. Educ Res Eval 2014; 20: 592-621. [http://dx.doi.org/10.1080/13803611.2014.997140].
[5] Sweller J. Element interactivity and intrinsic, extraneous, and germane cognitive load. Educ Psychol Rev 2010; 22(2): 123-38. [http://dx.doi.org/10.1007/s10648-010-9128-5].
[6] Pintrich PR. The role of motivation in promoting and sustaining self-regulated learning. Int J Educ Res 1999; 31(6): 459-70. [http://dx.doi.org/10.1016/S0883-0355(99)00015-4].
[7] Griese B, Lehmann M, Roesken-Winter B. Refining questionnaire-based assessment of STEM students’ learning strategies. International Journal of STEM Education 2015; 2(12): 1-12.
[8] Prümper J, Anft M. 9241/110 (Langfassung). Beurteilung von Software auf Grundlage der Internationalen Ergonomie-Norm DIN EN ISO 9241-110 [ISONORM 9241/110 (Long version). Evaluation of software based upon International Ergonomics Standard DIN EN ISO 9241-110]. Available from: http://people.f3.htw-berlin.de/ Professoren/Pruemper/ instrumente/ISONORM%209241-110-L.pdf 2009.
[9] Prümper J, Hurtienne J. Usability is easy to use: Some background on standards and processes. In: Sundblad Y, Walldius A, Eds. User-driven IT design and quality assurance. Stockholm: KTH 2007; pp. 47-52.
[10] International Organization for Standardization. ISO 9241-110:2006. Ergonomics of human-system interaction - Part 110: Dialogue principles. Available from: http://www.iso.org/ iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=38009 . 2006.
[11] Amadieu F, Tricot A, Mariné C. Exploratory study of relations between prior knowledge, comprehension, disorientation and on-line processes in hypertext. Ergonomics Open Journal 2009; 2: 49-57. [http://dx.doi.org/10.2174/1875934300902010049].
[12] Hachey AC, Wladis CW, Conway KM. Do prior online course outcomes provide more information than G.P.A. alone in predicting subsequent online course grades and retention? An observational study at an urban community college. Comput Educ 2014; 72: 59-67. [http://dx.doi.org/10.1016/j.compedu.2013.10.012].
[13] Hauser R, Paul R, Bradley J. Computer self-efficacy, anxiety, and learning in online versus face to face medium. Journal of Information Technology Education: Research 2012; 11: 141-54.
[14] Kalyuga S. The expertise reversal principle in multimedia learning. In: Mayer RE, Ed. The Cambridge handbook of multimedia learning. New York, NY: Cambridge University Press 2014; pp. 576-97.
[15] Kalyuga S, Ayres P, Chandler P, Sweller J. The expertise reversal effect. Educ Psychol 2003; 38(1): 23-31. [http://dx.doi.org/10.1207/S15326985EP3801_4].
[16] Park J-H, Choi HJ. Factors influencing adult learners' decision to drop out or persist in online learning. J Educ Technol Soc 2009; 12(4): 207-17.
[17] Saadé RG, Kira D. Computer anxiety in E-Learning: The effect of computer self-efficacy. J Info Tech Edu: Research 2009; 8: 177-91.
[18] Sam HK, Othman AE, Nordin ZS. Computer self-efficacy, computer anxiety, and attitudes toward the internet: A study among undergraduates in Unimas. J Educ Technol Soc 2005; 8(4): 205-19.
[19] Stiller KD. Mono- und bimodale Textpräsentationen zu Bildern in Hypermedia-Systemen [Mono- and bimodal text presentations accompanying pictures in hypermedia systems]. Psychologie in Erziehung und Unterricht 2009; 56(1): 49-63.
[20] Stiller KD. Fostering learning via pictorial access to on-screen text. J Educ Multimed Hypermedia 2017; 26
[21] Townsend CL, Heit E. Judgements of learning and improvement. Mem Cognit 2011; 39: 204-16. [http://dx.doi.org/10.3758/s13421-010-0019-2].
[22] Paechter M, Maier B, Macher D. Students’ expectations of, and experiences in e-learning: Their relation to learning achievements and course satisfaction. Comput Educ 2010; 54: 222-9. [http://dx.doi.org/10.1016/j.compedu.2009.08.005].
[23] Braun E, Gusy B, Leidner B, Hannover B. Das Berliner Evaluationsmodell für selbst eingeschätzte, studentische Kompetenzen (BevaKomp) [The Berlin Evaluation Instrument for Self-evaluated Student Competences]. Diagnostica 2008; 54(1): 30-42. [http://dx.doi.org/10.1026/0012-1924.54.1.30].
[24] Paechter M, Salmhofer G, Sindler A, Dorfer A, Mauer B. Grazer Evaluationsmodell des Kompetenzerwerbs (GEKo) [The Graz Model for the Evaluation of Student Competences]. In: Graz: Karl-Franzenz-Universität Graz. Graz: Karl-Franzenz-Universität Graz 2006.
[25] Paechter M, Maier B, Macher D. Evaluation von Lehre mittels Einschätzungen des subjektiven Kompetenzerwerbs [Evaluation of university courses by students’ assessment of acquisition of competence]. Psychologie in Erziehung und Unterricht 2011; 58: 128-38. [http://dx.doi.org/10.2378/peu2010.art26d].
[26] Richter T, Naumann J, Horz H. . Eine revidierte Fassung des Inventars zur Computerbildung (INCOBI-R) [A revised version of the Computer Literacy Inventory]. Zeitschrift für Pädagogische Psychologie 2010; 24(1): 23-37. [http://dx.doi.org/10.1024/1010-0652/a000002].
[27] Bernard RM, Brauer A, Abrami PC, Surkes M. The development of a questionnaire for predicting online learning achievement. Distance Educ 2004; 25(1): 31-47. [http://dx.doi.org/10.1080/0158791042000212440].
[28] Lee Y, Choi J. A review of online course dropout research: Implications for practice and future research. Educ Technol Res Dev 2011; 59(5): 593-618. [http://dx.doi.org/10.1007/s11423-010-9177-y].
[29] Dodd C, Kirby D, Seifert T, Sharpe D. The impact of high school distance e-learning experience on rural students’ university achievement and persistence. Online J Distance Learn Adm 2009; 1-12.
[30] McDonald S, Stevenson RJ. Effects of text structure and prior knowledge of the learner on navigation in hypertext. Hum Factors 1998; 40(1): 18-27. [http://dx.doi.org/10.1518/001872098779480541].
[31] Stiller K. Lernstrategien und Lernerfolg beim computerbasierten Wissenserwerb [Learning strategies and achievement in computer-based knowledge acquisition]. Psychologie in Erziehung und Unterricht 2003; 50(3): 258-69.
[32] Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979; 6(2): 65-70.
[33] Gignac GE, Szodorai ET. Effect size guidelines for individual differences researchers. Pers Individ Dif 2016; 102: 74-8. [http://dx.doi.org/10.1016/j.paid.2016.06.069].
[34] Koenker R. A Note on studentizing a test for heteroskedascity. J Econom 1981; 17(1): 107-12. [http://dx.doi.org/10.1016/0304-4076(81)90062-2].
[35] Fredericksen E, Picket A, Pelz W, Swan K, Shea P. Student satisfaction and perceived learning with on-line courses: Principles and examples from the SUNY learning network. J Asynchronous Learn Netw 2000; 4(2): 7-41.
[36] Lakhal S, Bazinet N. Technological factors explaining student dropout from online courses in higher education: A review. In: Carliner S, Fulford C, Ostashewski N, Eds. Proceedings of EdMedia: World Conference on Educational Media and Technology. Chesapeake, VA: AACE 2015; pp. 1806-11.
[37] Meiselwitz G, Sadera WA. Investigating the connection between usability and learning outcomes in online learning environments. J Online Learn Teach 2008; 4: 234-42.
[38] Tselios N, Avouris N, Dimitracopoulou A, Daskalaki S. Evaluation of distance-learning environments: Impact of usability on student performance. Int J Educ Telecommun 2001; 7(4): 355-78.
[39] Capa RL, Audiffren M, Ragot S. The effects of achievement motivation, task difficulty, and goal difficulty on physiological, behavioral, and subjective effort. Psychophysiology 2008; 45: 859-68. [http://dx.doi.org/10.1111/j.1469-8986.2008.00675.x].