Veletsianos, G., Collier, A., & Schneider, E. (2015). Digging Deeper into Learners' Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption. British Journal of Educational Technology 46(3), 570-587. Digging Deeper into Learners’ Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption Abstract Researchers describe with increasing confidence what they observe participants doing in Massive Open Online Courses (MOOCs). However, our understanding of learner activities in open courses is limited by researchers’ extensive dependence on log file analyses and clickstream data to make inferences about learner behaviors. Further, the field lacks an empirical understanding of how people experience MOOCs and why they engage in particular activities in in the ways that they do. In this paper, we report three findings derived by interviewing 13 individuals about their experiences in MOOCs. We report on learner interactions in social networks outside of MOOC platforms, notetaking, and the contexts that surround content consumption. The examination and analysis of these practices contributes to a greater understanding of the MOOC phenomenon and to the limitations of clickstream-based research methods. Based on these findings, we conclude by making pragmatic suggestions for pedagogical and technological refinements to enhance open teaching and learning. Keywords: MOOCs, learner experiences, notetaking, social networks, content consumption, qualitative, interpretive research Note: This is an author’s version of the final. The final version is available at http://onlinelibrary.wiley.com/doi/10.1111/bjet.12297/abstract Digging Deeper into Learners’ Experiences in MOOCs 1 Digging Deeper into Learners’ Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption The term Massive Open Online Course (MOOC) describes an evolving ecosystem of open online learning environments, encompassing a spectrum of course designs ranging from networks of distributed online resources (cMOOCs) to structured learning pathways centralized on proprietary or open source platforms (xMOOCs) (Rodriguez, 2012). These courses represent an expansion in the institutional and technological opportunities available in the greater landscape of online learning. Institutionally, MOOCs are being offered by a wide array of universities and non-accredited providers1. Learners enroll with no admissions requirements and participate outside of a credentialing program or a visible cohort. Technologically, there is a move away from traditional, text-based platforms to multimedia platforms with built-in assessment tools. MOOCs appear to offer opportunities for new and evolving threads of research into online learning, and indeed administrators and researchers extol MOOCs as sites for exploring how people learn, highlighting the great potential of large datasets, learning analytics, and experiments to support that endeavor (e.g., Breslow et al., 2013; Kay et al., 2013). However, while researchers can say with increasing confidence what they observe learners doing in MOOCs, empirical evidence on why they do what they do, how they do what they do, and what it is like to participate in MOOCs is scarce. Understanding these aspects of the learning experience is essential for improving scholarly understanding of learning in online settings that share the technological or institutional characteristics of MOOC (i.e. multimediarich platforms, minimal barriers to entry, etc). Further, technological or pedagogical designs to 1 Examples include: Coursera, edX, Udacity, FutureLearn, Open2Study, P2PU, and iversity Digging Deeper into Learners’ Experiences in MOOCs 2 improve online learning in emerging contexts such as open courses may be curbed if designers lack an in-depth understanding of individuals’ learning experiences. We address this gap in the literature and present findings that contribute to a deeper understanding of learners’ activities and experiences in MOOCs. Through interviews with 13 learners who had participated in a variety of MOOCs, we uncovered three consistent aspects of their experience: learner interactions in social networks outside of MOOC platforms, notetaking, and the contextual factors that surround content consumption. To contextualize these findings, we first review relevant MOOC research, acknowledging that this is a rapidly-evolving area of inquiry, and discuss insights from research on traditional online courses that helped to inform our analyses and conclusions. Next, we discuss our methods and findings. We close by discussing the implications of these findings for research and design. Review of relevant literature Online learning and research on the topic are not new (Means et al, 2013; Simonson, Schlosser, & Orellana, 2011; Zhao et al., 2005). While a systematic review of the literature on learner experiences in online settings is beyond the scope of this paper, it is important to note some significant aspects of pre-MOOC online learning research that is relevant to the current investigation. Social interactions are a central component of research on online learning. In pre-MOOC theories of online learning, researchers proposed that learner-learner, learner-instructor, learnercontent, and instructor-content interactions are key features of the learning environment (Anderson, 2003). Online and distance learners have historically faced a number of social challenges, including feelings of isolation (Galusha, 1997), high dropout rates (Peters, 1992), and unsatisfying and impersonal interactions (Vonderwell, 2003; Paechter, et al. 2010; Song, et al. Digging Deeper into Learners’ Experiences in MOOCs 3 2004, Lee, et al. 2011). Researchers have also reported that online learners may seek flexibility in the “timing of learning” over close interaction with instructors (Mandernach, Donnelli, & Dailey-Hebert, 2006). Nonetheless, though online courses were sought for their flexibility, some online learners may still struggle with time management and self-discipline needed to be successful. Researchers and designers have attempted to address these problems via pedagogical and technological interventions such as developing collaborative learning approaches and providing “live” office hours using chat rooms or video-over-IP services like Skype. While online learning is not a new phenomenon, the rise of MOOCs has increased the visibility and interest in online environments as spaces for learning and research to broader audiences (Gasevic et al, 2014). In our examination of the MOOC literature, we find that while research on the first generation of MOOCs (cMOOCs) uses methods and examines issues similar to those used and examined in traditional online learning research, the bulk of education-focused research on xMOOCs to date uses log file analyses and presents an aggregate picture of learner behavior. Currently, research focused on xMOOCs also provides minimal insight into learner experiences. Early research on MOOCs Research on many of the early MOOCs (2008-2012) focused on cMOOCs (Ebben & Murphy, 2014), which offered distributed learning opportunities across multiple online platforms, and encouraged open-ended exploration of central course questions. In this research, social interactions continue to be salient themes for learners’ experiences. Much of this research used case study approaches (Gasevic et al., 2014; Liyanagunawardena, Adams, & Williams, 2013). Using surveys, interviews, virtual ethnographies, platform data, and interpretive analyses of participants’ writing on various online platforms, this set of work provided detailed reports on learner experiences. For example, Kop, Digging Deeper into Learners’ Experiences in MOOCs 4 Fournier and Mak (2011) studied learners’ experience in two cMOOCs which encouraged learners to explore a distributed set of resources and to participate in conversations through multiple social networking tools. Learner-learner connections and interactions were a central feature of the course; the researchers reported that these connections took place on Facebook, on Twitter, on a Moodle discussion forum established for the course, and through participants’ individual blogs. These platforms enabled smaller learner groups to form, though more recent literature suggests that some learners have been finding these smaller groups alienating or impenetrable (Mackness & Bell, 2015). Learner-instructor interactions were also fairly minimal, leading to a perceived lack of support for some participants. In a different study, Mackness, Mak, and Williams (2010) found that the central design tenets of the cMOOCs - autonomy, diversity, openness, and connectedness - were realized in the experience of learners, but also created challenges. For instance, some participants expressed feeling lost without traditional guidance for navigating course materials. Research on other cMOOCs likewise showed that navigating the distributed design of these courses was a challenge, though participants who had more experience with open online courses generally had an easier time adjusting to this format (Milligan, Littlejohn, & Margaryan, 2013; Waite et al., 2013). In all of the courses in which these studies were conducted, self-directed learning strategies were deemed to be vital and necessary. xMOOCs and the ever-evolving research landscape With the launch in 2012 of a new wave of MOOCs (now often called xMOOCs), scale became the defining feature of MOOCs. Generally, the number of students who register for these courses far outnumber the number of students who complete them (Jordan, 2014). The centralized platforms on which these courses are offered provide educators and researchers the Digging Deeper into Learners’ Experiences in MOOCs 5 opportunity to access and study large amounts of data on learners. As a result, the scholarly literature on the MOOC phenomenon experienced a surge of research using quantitative, clickstream2, and observational data to describe learner behaviors and practices. The availability of large-scale data sets appears to have shaped the research questions that are being asked about MOOCs. A number of big data reports on MOOCs examine learner activities and behaviors in aggregate terms, using a rationale that is exemplified in the following quote by Koller, Ng, Do, and Chen (2013): “While some students engage with course content in ways that defy grouping, the majority exhibit behaviors that fall into clear categories.” Much of this research observes participants enrolled in courses on a particular platform (e.g., Brinton et al., (in press) investigate discussion forum behavior on 73 open courses on an anonymous platform) or in a particular university (e.g., Reich et al., (2014) examine learners enrolled in Harvard University’s courses). Studies of MOOC participants across multiple platforms paint a portrait of the wide variation in patterns of student engagement (Kizilcec, Piech, & Schneider, 2013; Seaton et al. 2014). These studies define engagement in terms of learners’ use of instructional materials – lectures, texts, and assessments – and use computational analysis techniques, developed for use with big data, to discover trends and patterns in engagement with materials. Discussion forums and social opportunities for participation are often considered a separate aspect of the course for purposes of analysis, though strong correlations are observed between staying engaged to the end of the course and discussion forum activity (Kizilcec, Piech, & Schneider, 2013). Among discussion forum participants, current research suggests that there is a smaller subset of “superposters” who contribute far more frequently and whose participation appears to inspire 2 A clickstream is a recording of what a user clicked on while visiting a website (e.g., clicked on the assignments page, clicked on the play button of a video, paused the video). Clickstream data can be transformed into multiple representations (total numbers of videos downloaded, etc.) in order to generate insights into user activities and behaviors. Digging Deeper into Learners’ Experiences in MOOCs 6 more conversations among other learners (Huang et al., 2014). The core and persistent evidence highlighted in these clickstream-driven studies is that each course has a small and selective group of committed learners, and a far greater number of individuals who pick and choose amongst course materials. An in-depth exploration of learner experiences in xMOOCs is just beginning to arise. For example, in a phenomenological study of xMOOC completers’ experiences of watching lecture videos, researchers found that some MOOC learners develop a sense of belonging and ‘eventedness’ in MOOCs (Adams et al., 2014). Taking another approach, Veletsianos (2013) presents a set of reflection essays from participants in a wide variety of MOOCs. In these essays, participants described the pleasure of learning novel topics for learning’s sake, the role of prior knowledge in their learning experiences, the tradeoffs of self-paced and synchronous courses, and the frequent frustrations and challenges they experienced in connecting with other learners in the discussion forums. These student-centered research efforts are sometimes accompanied with student-centered design efforts. For instance, Salmon et al (in press) report on participant experiences in a MOOC that was designed using Carpe Diem (a student-centered methodology). Reflecting on methods: A rationale for the current study Understanding learner experiences in MOOCs is essential for improving scholarly understanding of learning and teaching online. Early research on MOOCs was generally qualitative, with some incorporation of analytics, and focused on learners’ experiences in those courses. But, ease of access to large data sets from xMOOCs offered through an increasing number of centralized platforms has shifted the focus of MOOC research primarily to data science and computational methodologies, giving rise to a discourse suggesting that teaching and Digging Deeper into Learners’ Experiences in MOOCs 7 learning can be fully analyzed, understood, and designed for by examining clickstream data. While not exhaustive, the following quotes are illustrative and representative of this discourse: ● “What 6.9 million clicks tell us about how to fix online education” (Conner-Simons, 2014), ● “By collecting every click, homework submission, quiz and forum note from tens of thousands of students, Coursera [a MOOC platform] is a data mine that offers a new way to study learning” (May, 2012), ● “The 3 billion — and counting — data records produced within the edX MOOC platform are providing educators around the globe with insights into how students learn best and how they prefer to learn” (Buck, 2014), ● “‘We can study things like how much of a textbook they read, and what they said to their peers, which we can’t study on campus,’ Pritchard says. ‘We can see everything the students do’ (emphasis added).” (Chu, 2013). Even though data science and computational methodologies offer exciting and fruitful prospects for understanding learning and participation, a variety of methods, including qualitative approaches, are still necessary. Using a diverse range of methods allows researchers to triangulate findings, to investigate phenomena from multiple angles, and to capitalize on the strengths of each method (e.g., clickstream analytics methods can help researchers identify learner subpopulations of interest to interview). Technological or pedagogical designs may be restricted or misdirected if we lack an in-depth understanding of how individuals assign meaning to and experience their learning, participation, activities, and presence in MOOCs (Adams et al., 2014; Veletsianos, 2013). Our research helps address this gap in the literature by reporting on Digging Deeper into Learners’ Experiences in MOOCs 8 learners’ experiences in MOOCs and examining some of the ways and the reasons that learners engage with MOOCs. Methods The goal of this study was to investigate learners’ experiences and activities in MOOCs. To achieve this goal, we used a basic qualitative study (Merriam, 2002), which enabled us to identify learners’ experiences, activities, and perspectives about others (e.g., instructors and peers), and to understand how they make meaning of their experiences (Crotty, 1998). Qualitative inquiry is ideal for this goal because it enables researchers to generate rich descriptions of emerging phenomena such as MOOCs and enable individuals to share their experiences in their own voices. Methodologically, this study falls under the broad framework of the interpretive research paradigm. It is part of a larger research effort to gain a deeper understanding of participation and learning in open online learning and participation. Participants Individuals who participated in open courses were invited to participate in this study via a brief demographic survey shared on two of the researchers’ social media channels. In addition, one MOOC instructor shared the invitation via her personal Twitter account and via the Facebook group dedicated to the MOOC she taught. There was one inclusion criterion: To be included in the study, learners had to report having attended at least three weeks in at least one MOOC. Learners who responded to the survey but did not complete at least three weeks in a MOOC were excluded from the study. Three weeks was used as a demarcation point because prior research suggests that learners either commit to or abandon MOOCs by that point (Kizilcec, Piech, and Schneider, 2013). Digging Deeper into Learners’ Experiences in MOOCs 9 Forty-one individuals completed the brief demographic questionnaire. Twenty respondents were identified to be interviewed. As these individuals were being interviewed, the researchers were discussing participant responses. At the end of the 13th interview, the researchers agreed that the findings were increasingly repetitive and, in line with qualitative methodology recommendations, decided to end data collection (c.f. Baker & Edwards, 2012). Thus, the final sample consists of thirteen participants (table 1). Participants' ages ranged from 25 to 67 (Mean = 41, SD =13.8). Nine of the participants were female and 4 were male. Six respondents resided in the United States and two resided in the United Kingdom. The rest of the participants resided in Canada, El Salvador, India, Ireland, and the Netherlands. All but one of the participants had completed some portion of a college degree. Participants reported enrolling in MOOCs on all of the major platforms including Coursera, edX, FutureLearn, P2P University, NovoEd, and Blackboard CourseSites. Several participants had used multiple platforms. At the time of the study, all participants had completed at least one MOOC, all of them were enrolled in a MOOC, and all but one had enrolled in multiple MOOCs over the past year (three enrolled in 2 MOOCs, two enrolled in 3 MOOCs, three enrolled in 6-8 MOOCs, and four enrolled in more than 10). Table 1. Participants’ demographic information3 Name Age Employment Highest grade completed Country Ava 30 A homemaker College graduate United States 2 Chloe 38 A homemaker College graduate United States 1 Daniel 36 Employed for wages Some college (1 to 3 years of college) Ireland 1 3 All names used in this paper are pseudonyms. MOOCs completed to date Digging Deeper into Learners’ Experiences in MOOCs 10 Deven 25 Employed for wages College graduate India 7 Joanne 67 Self-employed Graduate School United Kingdom 1 Jacob 44 Self-employed College graduate The Netherlands 5 Lily 25 Unable to work Graduate School United States 2 Luis 25 A student High school graduate El Salvador 1 Olivia 45 Employed for wages Graduate School United States 3 Sophia 49 Employed for wages Graduate School United States 2 Mandy 50 Employed for wages Graduate School United Kingdom 1 Mia 35 Employed for wages Graduate School Canada 1 Nicole 63 Retired Graduate School United States 1 Data sources Two researchers interviewed participants using a semi-structured interview protocol that focused on asking learners to recount and describe particular moments in their MOOC experiences (appendix A). The protocol allowed interviewers to probe for additional comments and reflections while ensuring that questions were consistent and followed the pre-determined script. Interviews lasted between 25 and 50 minutes, were conducted via voice-over-IP, were recorded, and were transcribed verbatim. Data analysis We used the constant comparative method (Glaser & Strauss, 1967) to analyze the data collected. Individually, we read the interview transcripts in full to gain a holistic understanding Digging Deeper into Learners’ Experiences in MOOCs 11 of each interviewee’s experience. Next, again individually, we read the first sentence in our data set and tentatively coded it in an open manner. In particular, for each sentence, we wrote down possible categories in response to the guiding question: “What are learners’ experiences and practices in these courses?” The process followed thereafter was the same: Each new sentence was read and compared to the existing codes. If the sentence needed a new code to describe it, a new code was created. If the sentence fit the existing code, it was marked with that code. During this process, existing codes were reused and new ones were created. In total, we created 151 codes; some examples of codes created were “anxiety”, “professor”, and “use of backchannel”. The process of constantly comparing codes across data sources was used to confirm or disconfirm patterns. Codes were then consolidated into themes addressing the research questions. As we analyzed the data, we held regular meetings to discuss emerging codes and patterns. During these meetings we each discussed emerging patterns that we saw in the data and searched for alternate understandings of the phenomena arising. At the end of the fifth meeting, we realized that a number of the activities that were described by learners could not be easily captured by clickstream data. We believed that this warranted further elucidation. Thus, we returned to the transcripts and sought to identify further activities and experiences that fit this criterion. We continued coding data until we felt that further analysis and coding generated no new insights (i.e. we reached data saturation). Codes were then compiled into themes. Rigor We took a number of steps to reduce the potential of individual biases and ensure the accuracy of the reported results. First, we analyzed data individually and then met to discuss our findings. This step helped us challenge or corroborate our interpretations of learners’ experiences, and triangulate our findings by investigator. Second, we conducted member checks Digging Deeper into Learners’ Experiences in MOOCs 12 with participants and asked them to comment on whether the findings reflected their experience. Each participant was provided an email summary of major findings. Three participants responded to this email, and all indicated that the findings reflected their experiences and feelings. Finally, we present our findings using rich descriptions in order to allow others to examine the plausibility of our results, and evaluate the degree to which these results apply to their own context (Merriam, 1995). Findings All learners participating in this research were prompted to describe their day-to-day MOOC experiences. In describing their learning activities and participation, learners described (a) activities that are digital, but occur in platforms other than the ones on which MOOCs are taught (e.g., support groups on social networking sites), (b) activities that are not digital (e.g., taking notes on paper), (c) activities that are social (e.g., discussing MOOC experiences with others), and (d) activities that are individual (e.g., locating a study space at home). In the sections that follow, we examine three activities and experiences: Interactions in social networks outside of the MOOC platform, notetaking, and consuming content. We focus our analysis on these three because they were consistently discussed by participants in extensive detail and are novel in the MOOC literature. Interactions in social networks outside of the MOOC platform A number of learners alluded to interactions they have had with individuals who are part of their social networks. These include digital connections with other participants in a MOOC, face-to-face interactions with friends and family, and face-to-face interactions with new connections in a MOOC. Learners reported that these interactions occurred before, during, or after a particular course (frequently as a result of learners connecting with each other on social Digging Deeper into Learners’ Experiences in MOOCs 13 media prior to the beginning of the course or as a result of the course remaining open after its conclusion), suggesting that learner experiences with a particular MOOC may both precede and extend beyond the official start and end dates of a particular MOOC. Joanne described connections she made with other learners in a MOOC, and how those connections led to ongoing and impactful interactions that have sustained for much longer than the duration of the course: One person on that course contacted me by email...and said something nice about a blog post I'd made or something. And we have been having email discussions and private wiki discussions ever since...I've never met this person, but I would say that a month doesn't go by when we don't have an interaction, usually on a wiki that we've got. And it's usually related to some sort of theme that we picked. So it might be a book that we're interested in and we want to discuss certain ideas about it. But it's all in the background, not in public at all, but that connection came from [MOOC]. And there have been others like that as well...people who I'm connected with not very visibly, but have followed as a result of [MOOC]...And one or two of them I still haven't met. You know it's been totally online work that we've done together. But these people are quite close people now and we know a lot about each other. Sophia also noted that she maintained connections with individuals after the end of the course. In particular, she said that she “met some really great people, people that I am still in touch with now, many months or a year more after the course.” Mandy had a similar experience. She stated that a number of the individuals with whom she had MOOC forum interactions and Twitter interactions “are now part of her personal learning network.” Mandy maintains an “ongoing Digging Deeper into Learners’ Experiences in MOOCs 14 connection” with these individuals and has an “ongoing awareness” of them. It’s important to note that learners’ ongoing interactions with others may have little relationship to the quality or effectiveness of the MOOC in which individuals met each other. In Sophia’s words: “Sometimes it’s funny because I will be interacting with them and we will be talking about something and be like, ‘Oh God, we met in that horrible MOOC!’ But it is where we met. So, that’s a good thing.” Participants also discussed MOOC materials and experiences with family and friends. For instance, Joanne enrolled in a course because her “husband said how good it was” and Chloe discussed courses with her father- and mother-in-law. Chloe also shared worthwhile information she learned with her husband: “If there's something I want to share with my husband I'll go into the video transcript and I'll highlight and copy the text for what I want to share with my husband.” Ava shared information that she discovered in MOOCs with friends. With “one friend in particular,” who had questions about the Measles, Mumps, and Rubella (MMR) vaccine. Ava would “pop in my notebook and find my little MMR page and [tell my friend] ‘okay well this is kind of what they have talked about in the course that I took’.” Finally, Olivia described how she took a course with her boyfriend: We watched those videos together, and we talked about it. We just, we watched the videos together, mostly, on a weeknight or, you know, occasionally and probably on the weekend. And so there was just very simple quizzes in each video, and you know, one or two questions, and we answered them together. And so it was very simple. So but it mostly just involved watching the videos. Digging Deeper into Learners’ Experiences in MOOCs 15 Participants also reported multiple ways to find and connect with each other outside of MOOC platforms. For example, Deven noted that learners “post their email IDs and then you can communicate with them on Gmail or some other, through some other source, like on a Facebook study group or something.” Online study groups sometimes spill over into face-to-face meetings. In Deven’s case, he actively sought to meet people, attended face-to-face study groups in his hometown, and became friends with a person he met through these groups. In other cases, social media connections were significant. Joanne, for example, noted that “there’s an active Twitter group ...[a]nd I am reading. I am reading, but not participating in Facebook, and I am following Twitter.” Mandy described similar behavior, when she noted that she participated in “forum discussions and Twitter discussions” and “from time to time, [I] go look at their blogs, or if they are not bloggers, I do just look them up now and again.” For Sophia, social media participation served as a refuge from a “depressing and discouraging” situation. In the MOOC in which she was participating she became friends with a number of people “mostly those who fled to Google Plus and decided that we couldn’t participate in the Coursera discussion board but we kept on our discussion amongst ourselves over in Google Plus.” Notetaking Despite the fact that none of the popular MOOC platforms support integrated notetaking at the time of writing this paper, nearly all interviewees reported taking notes while watching lecture videos. Only one interviewee never took notes. However, the tools used to take notes and the subsequent use of notes varied substantially by learner. Learners took notes primarily from the video lectures, though Olivia also mentioned keeping notes of insightful comments from the discussion forum. Joanne summed up the ease of taking notes from video lectures when she said, “I quite often prefer to watch recordings than be Digging Deeper into Learners’ Experiences in MOOCs 16 in on a live session, to be honest. Because then I can control it. I can pause, I can say, ‘I want to think about that, I want to write a note about that.’” Learners took notes on paper and digitally. Paper notes appeared in notebooks designated for the course, on loose sheets of paper that the learner would later organize by course or topic (when taking multiple courses on a single topic), and, in Mia’s case, in the “to-do list” notebook that she carried with her at all times. Learners who took digital notes used word processing software or took screenshots when a lot of information was presented. Some would also excerpt the digital materials themselves, by copy-pasting text from the video lecture transcripts into their notes or by making notes directly on PDFs of the presenters’ slides. The choice to take digital or paper notes was one of personal preference, but was also influenced by the use of the notes. One aspect of Luis’ preference for paper notes was that he got tired of being in front of the computer. He also emphasized the benefits of paper as more versatile for studying and sharing: “I believe it's better for me to write it in a notebook because when I go out, if I have time I can read it. I can share it with my friend or with someone in the gym, things like that.” As a non-native English speaker, Luis also used his notes as an opportunity to practice the language and work on his reading and writing skills, so it was valuable to have access to his notes at all times. On the digital side, Joanne and Sophia used their course notes as the foundation for blog posts. The notes that Joanne took during lectures captured information and served as a site for developing ideas. Every few days, she would write a blog post by rereading her notes, adding key points to them, and synthesizing the concepts presented in the course along with her own perspective. Having her notes in digital form made it far easier to produce a final piece of digitized writing. Digging Deeper into Learners’ Experiences in MOOCs 17 Sharing notes and information with others was mentioned by several learners. Notably, “sharing with others” was described just as frequently as using notes to support studying, taking quizzes or doing writing assignments. Learners also planned to use their notes for personal and professional purposes after courses were over. For example, Olivia said “[W]hat I wrote down is more for how I can apply what's in the course to the work that I'm doing in my job right now” and Nicole noted, “These two or three classes are pretty relevant to the information that I need for the Childcare Center [so] I plan on going back and reusing them.” Other learners wrote down health information, recipes, or simply pieces of information that “appealed to me … stuff that I wanted to remember myself” (Daniel). Consuming content All individuals participating in this study discussed factors that shaped the ways they consumed MOOC content, shedding light on the context surrounding their participation. Scholars in the learning sciences have long highlighted the critical role of the environment, arguing that learning must be understood as a sociocultural phenomenon situated in context and culture (Brown, Collins, & Duguid, 1989). Patterns of MOOC content consumption can be examined by clickstream data, but these contextual factors help explain why learners exhibit particular patterns of participation. The ways that learners consumed MOOC videos were guided by personal and environmental factors. For example, Daniel and Olivia usually watched lecture videos after work hours, while Joanne would watch videos at "any time of the day that it fits in...but never late at night." Though Olivia had a schedule for watching videos, that schedule was prone to shifts based on other events in her life. For example, the first week of one of her courses coincided with a vacation that she went with her husband, "so if it was raining and we couldn't easily get Digging Deeper into Learners’ Experiences in MOOCs 18 out, we would spend the morning on [the MOOC]." Ava and Mia described how motherhood impacted the ways they participated in MOOCs, and specifically how their participation in MOOCs was episodic and happened in dedicated bursts of time. Ava said, "I will just take one day while my son is napping and just watch all the materials, read everything, trying to do the quiz, I will just do it all in one shot." Mia made similar comments: "[I will try] to find an hour, or it was actually like 45 minutes, of concentrated time. So usually when my daughter was asleep, so after 8 PM, and I would just sit down and I would just go through all the videos." Luis also tried to find a block of time to watch videos “about one hour, two hours,” but in the cases when he wasn’t able to find a sizeable chunk of time, he tried to listen to the videos while engaging in other activities: “when I'm cooking I'm trying to get the time about ten minutes to see the video.” The design of each course also appeared to impact the ways learners’ consumed MOOC content. Numerous learners we interviewed described how their participation and activities varied as a result of course-related factors. For example, Sophia described how the quality of the transcript impacted the way she viewed the course videos: “So, for the two courses where I did watch video, Coursera had a pretty poor quality transcript available but it was available and then the class I’m in right now they had a very high quality edited transcript, fully edited transcript.” Olivia, on the other hand, described how in one particular course she sped up the videos because the presenters spoke “pretty slowly” and it was “easy to absorb everything they're saying at a higher speed” while in a second course she watched the videos in normal speed because “the videos were actually really fast, and they were very dense...they were just so packed, and...the subject matter wasn't as familiar to me as some of the other courses I've taken.” Discussion and Implications Digging Deeper into Learners’ Experiences in MOOCs 19 In the sections that follow we discuss the importance and implications of our findings, and provide pragmatic suggestions to improve open teaching and learning through social, pedagogical, and technological interventions. Engagement and Mediation The experiences and practices reported in this paper are relevant to learning processes in unique ways. First, it appears that time and modality underpin the ways that learners engage with content. With regards to time, we discovered that these learners engage with content in both concentrated ways (e.g., while a child is asleep) as well as in dispersed ways (e.g., daily, after work). With regards to modality, learners engage with content in multiple modes (e.g., video, video transcript), and they do so in unique ways based on affordances imbued in the different modalities (e.g., pausing and replaying videos, taking notes on printed transcripts). Second, and most significant for researchers attempting to make sense of learning in MOOCs, learners’ participation and experiences in these courses resist binary and monolithic interpretations as they appear to be mediated by a digital-analog continuum as well as a social-individual continuum. In other words, learning and participation in MOOCs are both distributed and individually-socially negotiated. Figure 1 illustrates these two continua on an XY graph and highlights the fact that learner activities and experiences are varied and mediated. Digging Deeper into Learners’ Experiences in MOOCs 20 Figure 1. MOOC learners’ experiences and practices on a digital-analog continuum and a socialindividual continuum Social networks Brought together by mutual interests, learners shared information and interacted with one another outside of the MOOC platform. While the specific ways that these off-platform Digging Deeper into Learners’ Experiences in MOOCs 21 experiences affect learning is an open question for research, theorists have long emphasized the central role of dialogue and social interaction for learning (e.g., Lave & Wenger, 1991; Vygotsky, 1978). While the interactions of participants may or may not have directly impacted the content knowledge they acquired from the course, learners consistently described these interactions as meaningful. Interacting across multiple platforms is a key tenet of the design of cMOOCs, but prior research has revealed that some learners are initially uncomfortable with navigating multiple learning platforms. Other learners, usually those with more experience in courses of this style, possess greater confidence and more fluency in the required literacies for navigating multiple learning platforms (e.g., Waite et al. 2013). This research makes the following contributions to the existing knowledge base: Irrespective of MOOC design, some learners appear to (a) exhibit the agency to engage with the course on platforms outside of the one provided by the course designers, and (b) share and discuss the course with individuals who are part of their broader social network. MOOCs may be designed to recognize and value these two findings in similar ways that brick-and-mortar universities recognize the value of socialization as part of educating the “whole person.” For example, on-campus student organizations are central for residential and commuter campuses alike (e.g. Chambliss, 2014; Matthews, 1998; Rosenbaum et al., 2007). MOOC educators and designers may use technological or instructional strategies to remind learners of the potential value that exists in their learning being situated in a broader social network . An instructor, for example, may provide direct support for offline social learning activities via discussion guides and study group support documents. Guided practice activities for using other platforms productively for the MOOC could also be offered. Digging Deeper into Learners’ Experiences in MOOCs 22 Acknowledging that learners are active in digital environments other than the central platform can also be a way of designing for practices that happen within the platform. For example, course-related interactions that are happening on social media could be syndicated on the centralized course platform - a practice of aggregation which was systematically used in decentralized MOOCs (e.g. Kop 2011) but only happens in xMOOCs on a case-by-case basis. Student-facing dashboards could aggregate this information and future research could examine how to efficiently and effectively (a) integrate such dashboards into the learning environment, and (b) support learners in navigating and using syndicated content. Based on these results, we also advise MOOC providers to foster and encourage sharing and traversal by supporting platform-platform interoperability (e.g., between FutureLearn and edX) and platform-social media interoperability (e.g., between edX and Twitter). For instance, MOOC providers could develop social media plugins to allow learners to find and connect with each other: an edX-Facebook plugin could connect to a students’ Facebook profile and, given appropriate permissions from everyone involved, could notify individuals that individuals from their social network are enrolled in the same course as they are. Such a design feature would support the findings of this study and the findings of related research that suggests that learners who enrolled with colleagues or friends in MOOCs were slightly more likely to be highly engaged with the course materials than those learners who enrolled alone (Kizilcec and Schneider, 2014). There are a number of productive avenues for future research on the relationship between social networks and MOOC participation. One open question is the impact of these relationships on learning the topics of the course. Future research could seek to identify the barriers and enablers that exist for learners to engage socially online in the ways described. What prior Digging Deeper into Learners’ Experiences in MOOCs 23 experiences do socially engaged learners tend to have? What literacies for online navigation and communication do they possess? How do learners develop their own support networks and how do they use various technologies to do so? Answers to these questions will help researchers better understand and designers better support social learning endeavors in open online learning environments. Notetaking Notetaking, whether enacted in online or face-to-face courses, is generally considered to be an essential study skill, as it allows learners to create a reference document and to deepen their understanding through personal articulation of topics (Kobayashi 2005). Notetaking, as well as other activities that support learning, such as reflective journals, can be supported in the context of MOOCs. For instance, notetaking tools could be integrated into MOOC platforms. Notetaking integration could take advantage of the technology in multiple ways. First, digital notetaking allows the notetaking process to be scaffolded by the platform. For example, templates from instructors can direct learners’ attention to types of information that they would benefit from writing down or to particular notetaking strategies that have proven beneficial in other contexts (e.g., graphic organizers, self-explanation of concepts, concept maps (e.g., Okada, et al., 2008). Incorporating established notetaking strategies into a MOOC would not only allow digital platform providers to implement valued pedagogical strategies, but would also allow researchers to examine the impact of such strategies on a large scale. Another potential advantage of integrated, digital notetaking is that notes could be written collaboratively or shared. Collaborative notetaking could be a powerful strategy for leveraging the collective knowledge and interest of a group of learners (Kam et al., 2005; Miyake & Masuakawa, 2013; Steimle et al. 2008), but collaborative notetaking may also threaten the Digging Deeper into Learners’ Experiences in MOOCs 24 learning benefits of notetaking, as the work of personally articulating complex concepts is centrally important for knowledge construction (Bransford, Brown, & Cocking, 1999; Kobayashi 2005). However, there is not yet an established design paradigm for such a system, particularly at a massive scale. Finally, a notetaking system integrated into a platform can go beyond the traditional linear organization required by paper or a text file (Schneider, 2014). Notes could be connected to particular moments in a video or assessment, either in the form of annotations - notes on the resources themselves - or through hyperlinks on a separate note text. Notes could be tagged (automatically or by the student) by the concepts they are covering, which would make retrieval easier for studying and later reference. As with the templates for notetaking, the effectiveness of each of these strategies could be examined empirically. Such research would contribute to scholarly conceptions of the impact of notetaking as a learning strategy, while simultaneously allowing the interface design of the notetaking tool to be refined over time. If designers of learning platforms were to follow our suggestion to implement digital notetaking systems, learners should be given ownership over not only the process of note creation but continued easy access to their notes after courses are over. Our participants reported sharing their notes with friends and referring to them to address situations outside the course and after the course was over. We recommend that MOOC platform providers design digital notebooks that (a) live outside of a particular course so that learners can use them for multiple related courses if they have the need or desire; (b) are based on an interoperable protocol so that learners can use them across multiple platforms; and (c) can be exported in multiple formats for easy sharing and retrieval. Such a system will support and empower learners to organize their Digging Deeper into Learners’ Experiences in MOOCs 25 notes to fit their needs, including making connections between information across multiple courses and platforms. Learning Analytics, clickstream data, and the need for diverse research methods By capturing and analyzing digital data, the field of learning analytics promises great value and potential in understanding and improving learning and teaching. The focus on big data, log file analyses, and clickstream analytics in MOOCs is reflective of a broader societal trend towards big data analytics (Eynon, 2013; Selwyn, 2014) and toward greater accountability and measurement of student learning in higher education (Leahy, 2013; Moe, 2014). As technology becomes integrated in all aspects of education, the use of digital data and computational analysis techniques in education research will increase. However, an over-reliance on log file analyses and clickstream data to understand learning leaves many learner activities and experiences invisible to researchers. While computational analyses are a powerful strategy for making a complex phenomenon tractable to human observation and interpretation, an overwhelming focus on any one methodology will fail to generate a complete understanding of individuals’ experiences, practices, and learning. The apparent over-reliance on MOOC platform clickstream data in the current literature poses a significant problem for understanding learning in and with MOOCs. Critics of big data in particular question what is missing from large data sets and what is privileged in the analyses of big data (e.g., boyd & Crawford, 2012). For instance, contextual factors such as economic forces, historical events, and politics are often excluded from clickstream data and analyses (Carr, 2014; Selwyn 2014). As a result, MOOC research frequently examines learning as an episodic and temporary event that is divorced from the context which surrounds it. While the observation of actions on digital learning environments Digging Deeper into Learners’ Experiences in MOOCs 26 allows researchers to report activities and behaviors, such reporting also needs an explanation as to why learners participate in MOOCs in the ways that they do. For example, in this research, participants reported that their participation in MOOCs varies according to the daily realities of their life and the context of the course. Learners’ descriptions of how these courses fit into their lives are a powerful reminder of the agency of each individual. To gain a deeper and more diverse understanding of the MOOC phenomenon, researchers need to use multiple research methods. While clickstream data generates insights on observable behaviors, interpretive research approaches (e.g., ethnography, phenomenology, discourse analysis) add context to them. For example, Guo, Kim, and Rubin (2014), analyzed a large data set of MOOC video-watching behaviors, found that the median length of time spent watching a video is six minutes, and recommended that “instructors should segment videos into short chunks, ideally less than 6 minutes.” While dividing content into chunks aligns with psychological theories of learning (Miller, 1956), this finding does not explain why the median length of time learners spent watching videos is six minutes. Qualitative data and approaches can equip researchers to investigate the reasons why learners engage in video-watching behaviors in the ways that they do. For example, the median watching length of time might be associated with learner attention spans. On the other hand, multiple participants in this study noted that they were fitting the videos in-between other activities in their lives - thus shorter videos might be desirable for practical reasons: because they fit in individuals’ busy lives. Different reasons might be uncovered that explain why learners seem to engage with videos for six minutes, leading to different design inspirations and directions. Because the MOOC phenomenon, and its associated practices, are still at a nascent stage, interpretive approaches are valuable as they allow researchers to generate a refined understanding of meaning and scope of MOOCs. At the same Digging Deeper into Learners’ Experiences in MOOCs 27 time, it is significant to remember that a wholly interpretive approach to understanding learning in MOOCs will be equally deficient. Combining methods and pursuing an understanding of the MOOC phenomenon from multiple angles, while keeping in mind the strengths and weaknesses of each method, is the most productive avenue for future research. A computational analysis and data science discourse is increasingly evident in educational technology research. This discourse posits that it is possible to tell a detailed and robust story about learning and teaching by relying on the depth and breadth of clickstream data. However, the findings in our research reveal meaningful learner activities and practices that evade data-capturing platforms and clickstream-based research. Off-platform experiences as described above (e.g., notetaking) call into question claims that can be made about learning that are limited to the activities that are observable on the MOOC platform. Further, the reasons that course content is consumed in the ways that it is exemplifies the opportunity to bring together multiple methodological approaches to researching online learning and participation. Limitations This study faces a number of limitations that should be considered in evaluating the results presented. First, activities and experiences are self-reported. Second, participants were individuals who completed at least 3 weeks of a MOOC and many of them completed multiple MOOCs. As a result, their activities and experiences may only be representative of individuals who share this characteristic. Future research can address these issues and further examine these results by using the methods, protocols and instruments used in this study with other populations, by employing different data sources (e.g., surveys), and by triangulating interviews with other data sources. Conclusion Digging Deeper into Learners’ Experiences in MOOCs 28 In this study we examined learners’ activities and experiences in MOOCs. We discovered three activities and experiences that have received little attention in prior MOOC literature: interactions in social networks outside of the MOOC platform, notetaking, and consuming content. We also noted that the availability of large-scale data sets appears to have shaped the research questions that are being asked about MOOCs. To gain a more comprehensive understanding of learning and participation in open courses, researchers need to examine learner experiences more deeply. In short, researchers need to dig deeper, and use an array of methodological tools to do so. Separately or together, each research method can lead to pragmatic suggestions to improve open teaching and learning through social, pedagogical, or technological approaches. Acknowledgements: We would like to thank the three anonymous reviewers who provided feedback to this paper. We are indebted to one of the reviewers who encouraged us to explore how our findings relate to mediation and engagement. References Adams, C., Yin, Y., Vargas Madriz, L.F., & Mullen, S. (2014). A phenomenology of learning large: The tutorial sphere of xMOOC video lectures. Distance Education, 35, 202-216. Anderson, T. (2003). Getting the mix right: An updated and theoretical rationale for interaction. International Review of Research in Open and Distance Learning, 4(2). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/149/708 Baker, S.E. & Edwards, R. (2012). How many qualitative interviews is enough? National Centre for Research Methods (NCRM): University of Southampton. Digging Deeper into Learners’ Experiences in MOOCs 29 Bayne, S., & Ross, J. (2014). The pedagogy of the Massive Open Online Course: the UK view. York, UK: The Higher Education Academy. Beaudoin, M. (2002). Learning or lurking? Tracking the “invisible” online student. Internet and Higher Education, 5(2), 147-155. boyd, d., & Crawford, K. (2012). Critical Questions for Big Data. Information, Communication and Society, 15(5), 662-679. Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn: Brain, mind, experience, and school. National Academy Press. Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Studying learning in the worldwide classroom: Research into edX’s first MOOC. Research & Practice in Assessment, 8, 13-25. Brinton, C., Chiang, M., Jain, S., Lam, H., Liu, Z., & Wong, F. (in press). Learning about social learning in MOOCs: From statistical analysis to generative model. IEEE Transactions on Learning Technologies. Retrieved on August 14, 2014 from: http://arxiv.org/abs/1312.2159 Brown, J., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-41. Buck, T. (2014). Campus tech 2014: Reinventing higher education. EdTech Magazine. Retrieved on August 6, 2014 from http://www.edtechmagazine.com/higher/article/2014/07/campustech-2014-reinventing-higher-education Carr, N. (2014). The limits of social engineering. MIT Technology Review. Retrieved from: http://www.technologyreview.com/review/526561/the-limits-of-social-engineering/ Digging Deeper into Learners’ Experiences in MOOCs 30 Caulfield, M., Collier, A., & Halawa, S. (2013). Rethinking online community in MOOCs used for blended learning. Educause Review Online. Retrieved January 29, 2015 from http://www.educause.edu/ero/article/rethinking-online-community-moocs-used-blendedlearning Chambliss, D. F. (2014). How College Works. Harvard University Press. Chu, J (2013). Data from edX’s first course offer preliminary insights into online learning. MIT News. Retrieved on August 10, 2014 from http://newsoffice.mit.edu/2013/6002x-dataoffer-insights-into-online-learning-0611 Conner-Simons, A. (2014). What 6.9 million clicks tell us about how to fix online education. MIT News. Retrieved on August 6, 2014 from http://newsoffice.mit.edu/2014/what-69million-clicks-tell-us-about-how-fix-online-education Cross, S. (2013). Evaluation of the OLDS MOOC curriculum design course: Participant perspectives, expectations and experiences. OLDS Research Project, Milton Keynes. Retrieved October 27, 2014 from http://oro.open.ac.uk/37836/1/EvaluationReport_OLDSMOOC_v1.0.pdf Crotty, M. (1998). The foundations of social research: Meaning and perspective in the research process. London, UK: Sage Publications. DeBoer, J., Ho, A.D., Stump, G.S., & Breslow, L. (2014). Changing ”course”: Reconceptualizing educational variables for massive open online courses. Educational Researcher, 43(2), 78-84. Dede, C. (1996). Emerging technologies and distributed learning. American Journal of Distance Education, 10(2), 4–36. Digging Deeper into Learners’ Experiences in MOOCs 31 Dernoncourt, F., Do, C., Halawa, S., O’Reilly, U., Taylor, C., Veeramachaneni, K, & Wu, S. (2013). MOOCVIZ: A large scale, open access, collaborative, data analytics platform for MOOCs. Paper presented at the 2013 NIPS Workshop on Data Driven Education at Lake Tahoe, Nevada. Retrieved on August 10, 2014 from http://groups.csail.mit.edu/EVODesignOpt/groupWebSite/uploads/Site/MoocViz.pdf Ebben, M., & Murphy, J. S. (2014). Unpacking MOOC scholarly discourse: a review of nascent MOOC scholarship. Learning, Media and Technology, 39(3), 328. Eynon, R. (2013). The rise of Big Data: What does it mean for education, technology, and media research. Learning, Media, and Technology, 38(3), 237-240. Gasevic, D., Kovanovic, V., Joksimovic, S., & Siemens, G. (2014). Where is research on Massive Open Online Courses headed? A data analysis of the MOOC Research Initiative. The International Review Of Research In Open And Distance Learning, 15(5). Glaser, B., & Strauss, A. (1967). The discovery of grounded theory. Chicago: Aldine Publishing. Guo, P. J., Kim, J., & Rubin, R. (2014, March). How video production affects student engagement: An empirical study of mooc videos. In Proceedings of the 1st ACM conference on Learning@ scale conference (pp. 41-50). ACM. Harmon, K. (2014). Who’s writing the Rhizo14 ethnography?: The problem of authorship. Communication and Society blog. Retrieved August 12, 2014 from http://idst2215.blogspot.com/2014/07/whos-writing-rhizo14-ethnography.html Huang, J., Dasgupta, A., Ghosh, A., Manning, J., & Sanders, M. (2014). Superposter behavior in MOOC forums. L@S ‘14 Proceedings of the First ACM Conference on Learning at Scale, 117-126. Digging Deeper into Learners’ Experiences in MOOCs 32 Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. The International Review Of Research In Open And Distance Learning, 15(1). Kam, M., Wang, J., Iles, A., Tse, E., Chiu, J., Glaser, D., ... & Canny, J. (2005, April). Livenotes: a system for cooperative and augmented note-taking in lectures. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 531-540). ACM. Kay, J., Reimann, P., Diebold, E., & Kummerfeld, B. (2013). MOOCs: So many learners, so much potential. IEEE Intelligent Systems, 28(3), 70-77. Kizilcec, R.F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing learner subpopulations in Massive Open Online Courses. In Proceedings of the Third International Conference on Learning Analytics and Knowledge, ACM, 170–179. Kobayashi, K. (2005). What limits the encoding effect of note-taking? A meta-analytic examination. Contemporary Educational Psychology, 30(2), 242-262. Koedinger, K. R., McLaughlin, E. A., & Stamper, J. C. (2014). MOOCs and technology to advance learning and learning research: Data-driven learner modeling to understand and improve online learning. Ubiquity, (May, 2014), 1-13. Koller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and intention in massive open online courses: In depth. Educause Review, 48(3). Kop, R. (2011). The challenges of connectivist learning on open online networks: Learning experiences during a Massive Open Online Course. International Review of Research in Open and Distance Learning, 12(3), 19-38. Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant support on massive open online courses. The International Review of Research in Open and Distributed Learning, 12(7), 74-93 Digging Deeper into Learners’ Experiences in MOOCs 33 Koutropoulus, A., et al. (2012). Emotive vocabulary in MOOCs: Context and participant retention. European Journal of Open, Distance, and E-Learning, 2012(1). Retrieved October 28, 2014 from http://www.eurodl.org/materials/contrib/2012/Koutropoulos_et_al.pdf. Leahy, C. (2013). Catch-22 and the paradox of teaching in the age of accountability. Critical Education, 4(6), 1-19. Lee, S. J., Srinivasan, S., Trail, T., Lewis, D., & Lopez, S. (2011). Examining the relationship among student perception of support, course satisfaction, and learning outcomes in online learning. The Internet and Higher Education, 14(3), 158-163. Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the published literature 2008-2012. International Review of Research in Open and Distance Learning, 14(3), 202-227. Mackness, J., & Bell, F. (2015). Rhizo14: A Rhizomatic Learning cMOOC in Sunlight and in Shade. Open Praxis, 7(1), 25–38. Mackness, J., Mak, S.F.J., & Williams, R. (2010). The ideals and reality of participating in a MOOC. Proceedings of the 7th International Conference on Networked Learning. Retrieved January 29, 2015 from http://eprints.port.ac.uk/5605/1/The_Ideals_and_Realilty_of_Participating_in_a_MOOC.p df Mandernach, B.J., Donnelli, E., & Dailey-Hebert, A. (2006). Learner attribute research juxtaposed with online instructor experience: Predictors of success in the accelerated online classroom. The Journal of Educators Online, 3(2), 1-17. Digging Deeper into Learners’ Experiences in MOOCs 34 Matthews, A. (1998). Bright College Years: Inside the American College Today. University of Chicago Press. May, K. T. (2012). Completely free online classes? Coursera.org now offering courses from 16 top colleges. Retrieved from: http://blog.ted.com/2012/07/18/completely-free-onlineclasses-coursera-org-now-offering-courses-from-14-top-colleges/ Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Teachers College Record, 115(3), p. 1-47. Merriam, S. (1995). What can you tell from an N of 1?: issues of validity and reliability in qualitative research. PAACE Journal of Lifelong Learning, 4, 51–60. Merriam, S. (2002). Qualitative Research in Practice: Examples for Discussion and Analysis. San Francisco: Jossey-Bass. Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological review, 63(2), 81-97 Milligan, C., Littlejohn, A., & Margaryan, A. (2013). Patterns of engagement in connectivist MOOCs. Journal of Online Learning and Teaching, 9(2). Miyake, N., & Masukawa, H. (2013, April). Relation-making to sense-making: Supporting college students' constructive understanding with an enriched collaborative note-sharing system. In Proc. of 4th international Conference of the Learning Science (pp. 41-47). Moe, R. (2014). The evolution and impact of the massive open online course. ProQuest, UMI Dissertations Publishing. Okada, A., Shum, S. J. B., & Sherborne, T. (2008). Knowledge Cartography: software tools and mapping techniques. Springer. Digging Deeper into Learners’ Experiences in MOOCs 35 Paechter, M., Maier, B., & Macher, D. (2010). Students’ expectations of, and experiences in elearning: Their relation to learning achievements and course satisfaction. Computers & Education, 54(1), 222-229. Picciano, A. (2012). The evolution of big data and learning analytics in American higher education. Journal of Asynchronous Learning Networks, 16, 9-20. Radford, A. W., Robles, J., Cataylo, S., Horn, L., Thornton, J., & Whitfield, K. E. (2014). The employer potential of MOOCs: A mixed-methods study of human resource professionals’ thinking on MOOCs. International Review of Research in Open and Distance Learning, 15(5). Ramesh, A., Goldwasser, D., Huang, B., et al. (2014). Learning latent engagement patterns of students in online courses. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence. Retrieved August 11, 2014 from: http://linqs.cs.umd.edu/basilic/web/Publications/2014/ramesh:aaai14/ Reich, J., et al. (2014, January). PH207x: Health in numbers & PH278x: Human health and global environmental change - 2012-2013 course report. HarvardX Working Paper Series No. 2. Retrieved August 11, 2014 from: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2382242 Rodriguez, C. (2012). MOOCs and the AI-Stanford like courses: Two successful and distinct course formats for massive open online courses. European Journal of Open, Distance and E-Learning, 15(2). Rosenbaum, J. E., Deil-Amen, R., & Person, A. E. (2007). After admission: From college access to college success. Russell Sage Foundation. Digging Deeper into Learners’ Experiences in MOOCs 36 Schneider, E. (2014). Designing a hyperlearning annotation tool. Proceedings of the Learning Innovations at Scale Workshop, CHI. Retrieved August 11, 2014 from: http://lytics.stanford.edu/wp-content/uploads/2014/08/chi-annotation-short-paper.pdf Seaton, D.T., Bergner, Y., Chuang, I., Mitros, P., & Pritchard, D.E. (2014). Who does what in a massive open online course? Communications of the ACM, 57(4), 58–65. Selwyn, N. (in press). Data entry: towards the critical study of digital data and education. Learning, Media and Technology. Simonson, M., Schlosser, C., & Orellana, A. (2011). Distance education research: a review of the literature. Journal of Computing in Higher Education, 23(2-3), 124–142. Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: Student perceptions of useful and challenging characteristics. The Internet and Higher Education, 7(1), 59-70. Steimle, J., Brdiczka, O., & Muhlhauser, M. (2008, July). Coscribe: Using paper for collaborative annotations in lectures. In Advanced Learning Technologies, 2008. ICALT'08. Eighth IEEE International Conference on (pp. 306-310). IEEE. Veletsianos, G. (2013). Learner Experiences with MOOCs and Open Online Learning. Madison, WI: Hybrid Pedagogy Publications. Retrieved from http://learnerexperiences.hybridpedagogy.com. Vonderwell, S. (2003). An examination of asynchronous communication experiences and perspectives of students in an online course: A case study.The Internet and higher education, 6(1), 77-90. Vygotsky, L.S. (1978) Mind in Society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Digging Deeper into Learners’ Experiences in MOOCs 37 Waite, M., Mackness, J., Roberts, G., & Lovegrove, E. (2013). Liminal participants and skilled orienteers: Learner participation in a MOOC for new lecturers. Journal of Online Learning and Teaching, 9(2). Wen, M., Yang, D. & Rose, C. P. (2014). Linguistic reflections of student engagement in Massive Open Online Courses. Association for the Advancement of Artificial Intelligence. Retrieved August 11, 2014 from http://www.cs.cmu.edu/~mwen/papers/icwsm2014camera-ready.pdf Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, H. S. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Teachers College Record, 107(8), 1836–1884. Digging Deeper into Learners’ Experiences in MOOCs 38 Appendix A: Semi-structured interview protocol • Tell me a little bit about you. What do you do? What are your interests? • Which courses did you take and on what platforms? • Tell me about how you found out about [courses]. • Tell me about how you decided to enroll in [courses]. • Think back to an open online course in which you participated. Recall a vivid event or experience in that course. Tell me what you remember about that experience in detail. • Recall a second vivid event or experience in a course. Tell me what you remember about that experience in detail. • What do you remember about the instructor? • Tell me about your experience with the instructor of the course. • Think back to an interaction that you had with another person in a particular course. Recall a particular interaction. It could be a conversation on the class forum, a discussion on a digital space outside the platform, a chat with a face-to-face person that was also taking the class, or just reading a forum posting. [pause] Tell me what you remember about that experience in detail. • Tell me more about other people in the course o Do you think about them? o What do you remember about them? • Tell me about what [this course] added to your life. How is your life different as a result of [this course]? • Is there anything else you wanted to tell me about your experiences or anything related to MOOCS?