Research in Learning Technology

ALT's Open Access Journal

Learning outcomes: online versus classroom learning

The most recent research has focused on whether online learning is as effective as learning in traditional classroom environments. Several studies have found online learning to be comparable to face-to-face learning. Cao and Sakchutchawan (2011) sought to investigate if online and traditional MBA courses were similar in terms of quality and student satisfaction. Results did not reveal significant differences between course passing rates for online versus traditional students, but did reveal significant differences in course satisfaction with lower mean ratings for online courses. The authors concluded that students enrolled in online MBA courses performed as well as students enrolled in traditional MBA courses; however, online MBA students were less satisfied with their learning experiences. In contrast, Ashby, Sadera and McNary (2011) examined performance outcomes across three different contexts (i.e. face-to-face, blended and online) for students enrolled in an intermediate developmental community college math course. They found that students enrolled in the traditional class scored worse than those students in online and blended courses on course tests and earned lower course averages. In addition, more passing grades were earned by the students in the online course followed by blended and then traditional environments suggesting online learning to be more effective than classroom based learning. However, differences in the learning environments themselves (in the online setting students were permitted to take course tests with notes by their side) may explain why the online students received the highest course averages and passing grades over the traditional students; it does not mean, however, that the online students learned more than the traditional students.

A meta-analysis conducted by the US Department of Education on 45 controlled design category 1 (entirely online versus classroom) and 2 (blended versus classroom) studies published between 1996 and 2008 revealed modestly large mean effect sizes for learning outcomes (e.g. grades, scores on midterm/final exams, etc.) for students in online learning (d=0.20) versus traditional face-to-face classes (Means, Toyama, Murphy, Bakia and Jones 2010). Effect sizes were larger for blended (d=0.35) versus fully online courses (d=0.05). Aragon, Johnson and Shaik (2002) sought to determine if students enrolled in graduate level online versus in-person courses (Human Resources classes) differed on learning style preferences and if these learning style preferences affected success in online versus in-person learning contexts. Learning style was comprised of motivation, task engagement, and cognitive control systems based on Curry's (1991) theoretical framework and evaluated outcomes using the Student Learning Style Scale, Learning and Study Strategies Inventory, Learning Style Inventory and final course/project grades. Results indicated that the groups differed in cognitive controls with online students exhibiting greater reflective observations and preferring abstract conceptualisation (i.e. learning by thinking), whereas traditional students reported increased use of active experimentation (i.e. learning by doing). However, results indicated that differences in learning style preferences did not affect learning success for the two groups. The authors concluded that online learning can be as effective as traditional classroom learning even if learning style preferences vary.

Course design characteristics of online learning environments

Several studies have investigated course and instructional design factors that contribute to effective online learning, both in terms of outcomes and satisfaction. Ruey (2010) used a qualitative approach (structured interviews, content analysis of course documents, surveys and online observations) to explore the use of a constructivist-based design for a professional development graduate level online course to investigate how and if online students benefit from this model. Several aspects, including peer moderated discussions, group projects and chat room meetings, used constructivist-based instructional strategies. Results suggested that this model facilitated development of self-directed learning skills and a sense of community among the learners via peer collaboration. It changed some of the learners’ views regarding online learning and learning in general (i.e. students reported being more concerned with what was learned rather than just grades and related what they learned to daily life); however, adequate facilitation and critical feedback on the part of the instructor were reported as critical factors in perceived quality of the online course.

Song, Singleton, Hill and Koh (2004) examined factors related to online learning effectiveness from the graduate students’ perspective. Learners’ perceptions were assessed using a survey generated by the researchers targeting learner characteristics, perceived difficulties and beneficial components of online learning. Results indicated that course design and time management were crucial components to successful online learning, while lack of community and technical problems were most challenging for online learners. Eom, Wen and Ashill (2006) also sought to examine factors that contribute to perceived student success and satisfaction in university online learning contexts, examining both learner and course design characteristics. Self-motivation, learning styles, instructor knowledge/facilitation, instructor feedback, interaction, and course structure were assessed using a researcher-generated survey. Results showed that the model (comprised of the six variables listed above) explained 69% of the variance in learner satisfaction and 63% of the variance in learning outcomes. Further analysis revealed that, for learner satisfaction, all of the factors included in the model were significant; only instructor feedback and learning style were significant for learning outcomes. The authors concluded that online learning is best suited for students with visual and read–write learning styles and that instructor feedback should be integrated into the course design.

Ke and Xie (2009) also examined the perception of learning success and satisfaction among undergraduate and graduate students enrolled in online courses, but examined the effect that course design has on these variables. Ten courses analysed in the study were coded into one of three course design models:

1. integrated – content was unstructured and adaptable; no textbook-weekly readings provided by instructor; online discussions/team projects with active facilitation by instructor,
2. content-support – highly structured with pre-recorded lectures and assignments/quizzes; minimal interaction with other students, or
3. wrap around – moderately structured with weekly virtual lectures and assignments; ≤50% of time dedicated to participation in the discussion board; moderately structured assigned online discussion tasks (integrated [combination of open- and close-ended], open-ended or close-ended).

Results showed that learners were significantly more satisfied with the integrated than the wrap around course model, and were significantly less satisfied with close-ended (versus integrated) discussion types. The authors concluded that the integrated course design model appears to promote the highest level of satisfaction while close-ended discussion tasks promote the opposite. The United States Department of Education meta-analysis (Means et al. 2010) also revealed specific instructional/design practices and conditions associated with improved performance. A narrative synthesis of category 3 studies (online courses with interactive elements, such as videos) suggested that learner control of interactions and prompts facilitating reflecting on information, including problem solving strategies/self-explanation, were found to be successful for online learning performance.

Course content/discipline

In the online learning environment, just as in the traditional courses, consideration of alignment of instructional content and assessment measures with learning outcomes is critical to successful learning outcomes and satisfaction. In other words, course objectives should align with the delivery of content and the manner in which learning is assessed (Blumberg 2009). Blumberg (2009) offers suggestions for instructors with regards to course alignment and maximising learning, including creating objectives based on taxonomies of learning [i.e. Anderson and Krathwohl (2001) revised taxonomy], matching the teaching, learning and assessment activities with the objectives. Objectives are the main driving force of a course; they detail what students should master by the end of a course; assessments are used to determine if students have met the course objectives. Given the importance of objectives, they should guide course planning and approach to teaching of information.

Anderson and Krathwohl (2001) revised taxonomy details levels and types of learning. Levels of learning include remember, understand, apply, analyse, evaluate and create. Types of learning include factual, conceptual, procedural and metacognitive. Conceptual knowledge promotes deeper learning (i.e. application, analysis and evaluation) versus factual knowledge, something instructors should consider when planning and designing a course. Courses become misaligned when objectives do not match teaching, learning and/or assessment methods. For example, objectives may state higher level cognitive processes (e.g. analysis and evaluation); however, instructors may not provide teaching or learning opportunities that facilitate that type of knowledge (e.g. just providing PowerPoint lectures of factual information). According to Reeves (2006), assessment methods are the most misaligned component of courses, regardless of the context. He suggests that instructors tend to focus on assessing things that are easy to measure (i.e. factual information via multiple choice tests), rather than assessing authentic tasks (i.e. problem-based assessments). Course management systems (CMS) used in online courses (e.g. Blackboard) make it easy for instructors to lose sight of alignment, given the automaticity of using templates and drop down tools. Often, instructors are ill informed of ways to effectively use the features present in CMS (Koszalka and Ganesan 2004). Instructors can learn ways to use technology to create instructor–learner, learner–learner and learner–content interactions to promote deeper levels of knowledge (Fabry 2009).

Different disciplines may require different types of knowledge, especially in terms of conceptual and procedural knowledge (Blumberg 2009). The online learning environment presents a unique challenge on how to engage students in developing discipline specific conceptual– procedural knowledge. McCracken, Cho, Sharif, Wilson, and Miller (2012) provided case study examples of ways to engage and develop deeper levels of knowledge in an online dental hygiene undergraduate programme. The design strategy included peer assessment, virtual patients (for assessment and treatment planning), and extended case studies, incorporating problem-based and authentic learning experiences in both instruction/learning activities and assessment. Jones (2015) provided examples of integrating technology with course content in clinical social work online courses to meet discipline specific objectives. Students studying to become social workers need to develop higher level skill sets including interpersonal skills, self-awareness, application, analysis and evaluation skills (for diagnostic assessment, planning and treatment). Jones (2015) used synchronous videoconferencing, service learning (for developing clinical skills) and role playing (with asynchronous video recordings) in the online foundational social work courses. Students in both the online and face-to-face courses rated them favourably. Rogerson-Revell (2015) used e-tivities in an online applied linguistics/TESOL masters programme to build critical thinking and practice skills. The e-tivities were designed to promote collaboration and formative assessment (portfolio of written assignments and activities) in an asynchronous manner with use of an e-moderator (student tutor). E-tivities were module specific with no more than 8–10 students in each cohort, and used voice based discussion boards, podcasts and blogs. Students reported increased motivation, reflection and preparedness for their final module assessment.

Learner characteristics and learning outcomes

Several research studies have also sought to examine learner characteristics which contribute to positive performance outcomes. Berenson, Boyles and Weaver (2008) investigated the relationship between learners’ emotional intelligence (EI), resilience, personality type and success [via surveys/grade point average (GPA)] in the context of online learning for a diverse population of technical community college students. EI was defined as the awareness and self-regulation of feelings and needs, measured by the EI-Q abridged survey. EI accounted for 11% of the variance in GPA and the combination of EI and personality accounted for 18% of the variance. Students with lower amounts of success in online learning were characterised by having lower EI with personalities marked by aggression, tension and external locus of control. In contrast, DeTure (2004) investigated whether spatial skill (in terms of field independence/dependence) and self-efficacy for use of computer technologies could predict success (final course GPA) in online courses for community college students. The participants were assessed on two measures: the GEFT (Group Embedded Figures Test: to determine field independence/dependence) and the OTSES (Online Technologies Self-efficacy Scale). Results indicated that neither the GEFT nor the OTSES significantly predicted success in online courses. There was, however, a significant positive relationship between GEFT and OTSES, suggesting that those with higher field independence demonstrated greater self-efficacy with online technologies. This can be explained by the fact that spatial measures can tap general intelligence: if general intelligence is the product of prior academic success this would explain why spatial skills and self-efficacy are correlated. DeTure concluded that students with greater field independence and hence greater computer technology self-efficacy would not necessarily receive higher grades in online courses than those students with more field dependence and less computer self-efficacy. Similarly to DeTure's findings, Liu, Magjuka and Lee (2008) found that cognitive style did not affect online learning performance. Muilenburg and Berge (2005) examined perceived barriers to learning online for adult learners engaged in a variety of web-based learning settings (university courses, community college courses, business/corporate courses) using a self-report survey. Exploratory factor analysis revealed time management and lack of motivation as significant underlying barriers. Less important barriers included technical skills, which support DeTure's findings regarding technology self-efficacy and online learning outcomes. Results of other studies have similarly demonstrated the important roles that self-regulation (including time management skills, active learning style and self-discipline) and motivation play in online learning (Lim and Kim 2002; Park and Choi 2009; Waschull 2005; Yukselturk and Bulut 2007).

Conclusions

The main findings from the literature supported learner characteristics and skills needed to be successful and satisfied with online learning. What emerged following this narrative synthesis was a profile of the successful online student: one who demonstrates greater EI, that is, self-awareness of needs, adequate management of feelings (Berenson, Boyles, and Weaver 2008), self-regulation skills, self-discipline, time management, organisation, planning, self-evaluating (Eom, Wen, and Ashill 2006; Kerr, Rynearson, and Kerr 2006; Muilenburg and Berge 2005; Ruey 2010; Song et al. 2004; Waschull 2005; Yukselturk and Bulut 2007), reflective/visual learning styles (Eom, Wen, and Ashill 2006; Means et al. 2010) and internal locus of control (Berenson, Boyles, and Weaver 2008; Kerr, Rynearson, and Kerr 2006).These findings are not surprising considering the nature of online learning. More responsibility is placed on the learner, especially in asynchronous courses. The student is responsible for reviewing course material, taking exams at scheduled intervals etc., which requires adequate self-regulation skills. Self-regulated students take control of their learning, developing appropriate metacognitive strategies such as planning, staying organised and motivated (Yukselturk and Bulut 2007).

EI also plays a role in online learning; it is easy for students to get frustrated with the technology and/or challenges to understanding information without the option of immediate feedback and/or assistance. Students who are able to control their emotions and get proper assistance prior to becoming frustrated will have better control over their learning (Berenson, Boyles, and Weaver 2008). Learning style characteristics such as being a visual learner make sense in the online learning context. Information is typically presented visually versus aurally with learners coming to understand information via descriptions. Online learning may not be the best context for students who are not visual learners (Eom, Wen, and Ashill 2006).

Some surprising results surfaced regarding success in online learning. While preferred learning styles may vary between students in online and traditional classes, with online students preferring reflective learning and traditional students preferring active learning (i.e. learning by doing), these preferences for learning did not affect success: students can perform just as well online as in face-to-face courses regardless of learning style. This demonstrates that some students have flexibility and can adapt to the different learning environments being successful in both contexts (Aragon, Johnson, and Shaik 2002). Another surprise finding concerns cognitive style and technology self-efficacy. Students with field independent cognitive styles are independent learners, using internal cues to understand information and demonstrate greater confidence in using online technology; field dependent students depend on others to learn information, and may not be as comfortable with online technology; however, these differences did not translate to differences in success within online learning (DeTure 2004; Liu, Magjuka, and Lee 2008). Other studies have demonstrated mixed results regarding comfort with online technology and success in online learning, that is, no difference in learning outcomes for those students with high versus low technology efficacy (Kerr, Rynearson, and Kerr 2006; Muilenburg and Berge 2005; Waschull 2005). Perhaps students enter the world of online learning equipped with requisite skills for using the technology, and because of the high level of self-regulation skills they possess, they become self-motivated to learn how to navigate through the course.

In terms of satisfaction, learners were pleased with online courses that were structured (Ke and Xie 2009; Song et al. 2004), interactive, that is, constructivist instructional design (Aragon, Johnson, and Shaik 2002; Ke and Xie, 2009; Muilenberg and Berge 2005; Ruey 2010), relevant, that is, application based with practical significance (Park and Choi 2009; Ruey 2010), and instructor facilitated in terms of interactions/feedback (Eom, Wen, and Ashill 2006; Muilenberg and Berge 2005; Ruey 2010; Song et al. 2004). Courses that facilitated increased performance and satisfaction were interactive and allowed for collaboration, organised/structured with prompts to facilitate reflection as learning is occurring. All of these factors involved in satisfaction come down to instructional and course design. Courses should be structured around reading materials, lectures and assignments organised into units with clear learning goals in mind. In other words, course instructors need to ensure that there is alignment of objectives with instructional methods, learning activities and assessment methods (Blumberg 2009). Instructors should provide timely feedback and serve as facilitators of discussion and interaction just as they do in traditional courses. Courses should provide opportunities for peer collaboration and sharing of ideas in order to develop an online community of learners, rather than feelings of isolation (Song et al. 2004).

Implications

Yukselturk and Bulut (2007) offered some recommendations regarding design of online courses to meet learner expectations, including integrating strategies to facilitate self-regulation within the course, monitoring of students performance with prompt feedback, regular interactions with peers and instructors while making use of current technology, and ensuring content of practical significance so as to allow students to make real life connections. Self-regulation strategies can be facilitated through the use of learner reflection prompts, providing learners a way to explicitly reflect on what they are learning (Means et al. 2010). Dabbagh (2007) suggested two learning models to accommodate and support the characteristics of successful online learners: exploratory and dialogical models. The exploratory model is based on inquiry/authentic problem-based learning methods which support the attributes of successful online learners (i.e. reflective and self-directed/self-regulated learning). This model can be implemented through use of internet resources including hypermedia/multimedia links and specific instructional modules to facilitate exploration. Dialogical models of learning focus on learning information through interactions: i.e. conversations supported through group forums/discussion boards, listservs, chat groups and document sharing, emphasising collaboration and reflection. Ruey (2010) proposed a constructivist-based model for online learning in which the instructor functions as the facilitator/mentor, assisting in the development of reflective/critical thinking skills; the TA (teaching assistant) functions in a supporting/monitoring role; and the students function in a moderating role, keeping one another engaged in interactive learning.

Limitations of prior studies

The aforementioned studies did not account for instructor attributes and the role these characteristics play in online satisfaction and other outcomes. In addition, these studies did not separately analyse the learning or needs of students with disabilities or those from various socioeconomic backgrounds. The importance of instructor feedback emerged from the narrative synthesis; however, the importance of peer feedback has not been systematically investigated. It is still not known what role peer feedback plays in knowledge building within online classrooms. While we know the importance of instructor feedback on learners’ satisfaction and performance, we do not know how timing affects this, i.e. if there is a limit to when feedback is effective, considering the time delay that is ubiquitous in online learning. The above studies failed to investigate how the quality of instructor feedback affects satisfaction and learning outcomes. Finally, many of the studies used surveys/self-report measures to assess learner characteristics and satisfaction. Many of these surveys were not adequately validated, which in effect, casts doubt on the findings.

What is also not yet known is if students in online learning courses/programmes learn course material deeply versus superficially. Many of the studies included in this synthesis used course GPA as indicators of success, which in of itself does indicate deep knowledge and necessarily greater outcomes. Many of the course design suggestions for developing quality online classes provide opportunities for deep learning experiences; however, many of the studies did not investigate whether deep learning occurred. Ke and Xie (2009) found that the majority of students included in their study reported a deep learning approach on questionnaires, but the content analysis of the online discussions suggested otherwise. We do not fully understand how reading and writing vary for online versus traditional learning modes and how this affects online performance. The former studies did not account for students’ level of technological literacy (literacy for computer/information technology) and how this affects learning. It is also not known how various suggested pedagogical models for online learning compare in terms of learner satisfaction and knowledge building.

Future research and practice

In the future, researchers can create valid and reliable instruments to empirically assess student attributes specific to online learning to determine if students are ready to take online courses. One such measure, the Test of Online Learning Success (TOOLS), has been created and validated (Kerr, Rynearson, and Kerr 2006) for just this purpose. The test taps five constructs: computer skills, independent learning, need for online learning, dependent learning and academic skills, assessing strengths and weaknesses with regards to online learning readiness. Across the 3 years of development and validation of the TOOLS, the researchers’ identified critical student characteristics for predicting online success were all related to independent learning (self-regulation skills, time management, multitasking, self-motivation/discipline). Reading and writing proficiency was also crucial to student performance. Following administration of such a measure, administrators could create orientation courses to explicitly teach strategies that have been demonstrated to be helpful in online learning including self-regulation skills, and instruct students on the differences between traditional and web-based courses, including the more autonomous nature of learning in online courses.

Professors/instructors should be knowledgeable regarding design characteristics that promote student success and satisfaction including course alignment and integration of content with technology to facilitate engagement. Institutions should provide faculty with instructional courses regarding making the transition to teaching online courses including the aforementioned design characteristics. Universities should not just assume that faculty can teach effectively online. Instructors may need explicit guidance regarding the importance of such factors as prompt feedback, a primary factor for online satisfaction. This makes sense considering that feedback can motivate students (Eom, Wen, and Ashill 2006). According to the Sloan Consortium report on online learning, only 6% of universities surveyed did not provide training and or mentoring programmes of faculty teaching online courses (Allen and Seaman 2011). Another consideration that institutions should take into account is course size. The nature of online courses makes it easy to enrol more students per course than traditional courses. Large enrolment courses may be challenging for instructors to effectively provide enough attention/ interaction to individual students and manage the course; students may find it challenging to manage discussion boards and interaction with fellow classmates. Studies have demonstrated the negative impact of large course size on online course satisfaction measures (Arbaugh and Duray 2002; Arbaugh and Rau 2007). More research is needed examining the effect of course size on student performance and satisfaction within the online learning environment, including the ‘optimal’ course size and effects on satisfaction and learning outcomes of placing students enrolled in larger online courses into cohorts.

Another area that warrants further investigation involves the use of technology within online courses to meet the needs of students with disabilities.

In conclusion, many studies have documented that online courses can be just as or more successful than traditional face-to-face courses (Aragon, Johnson, and Shaik 2002; Ashby, Sadera, and McNary 2011; Cao and Sakchutchawan 2011; Jones 2015; Means et al. 2010; Rogerson-Revell 2015). Success in online learning requires interaction between the learner, instructor, learners and technology (Fabry 2009). Fully online degree programmes are emerging at a fast rate due to the increased demand (Allen and Seaman 2011). More and more students are turning to online learning as it represents convenience in a busy and demanding society. Technology is also changing at a rapid rate; new ways of obtaining and sharing information are being developed and applied to the higher education setting. With all of these changes, it becomes even more important for colleges and universities to find ways to improve the quality of online learning to maximise learning, including effectively aligning technology with course content and instruction (Rogerson-Revell 2015). Ultimately, future employers will expect that students with a degree, no matter if completed online or in person, will have obtained the necessary skills and knowledge to perform the job requirements. In addition, students with no other choice but to complete degrees online will expect to receive the same standard of education as students in traditional degree programmes; decreased satisfaction with internal factors including instructional design, course organisation, online teaching skills and use of technology, are predictors of attrition, which can lead to loss of money and time/investment for both students and universities/colleges (Moody 2004; Park 2007). This review has demonstrated that many factors play a role in satisfaction with and success in online learning. The challenge remains for institutions to design courses to meet students’ needs/expectations, for instructors to effectively integrate technology into content, including features of CMS, in order to facilitate engagement and deeper learning, and for students to be self-disciplined and use technology to engage with other students and the instructor.