# Learner Use of Online Educational Resources for Learning (LUOERL)–Final report

 Bacsich, P., Phillips, B., & Bristow, S. F. (2011). Learner Use of Online Educational Resources for Learning (LUOERL)–Final report.

BibTex

BibTex

BibTex

`@article{bacsich2011learner,
author = {Bacsich, P. and Phillips, B. and Bristow, S. F.},
date-added = {2012-04-07 02:42:58 +0000},
date-modified = {2013-01-14 21:01:49 +0000},
date-read = {2012-04-07 18:10:24 -0400},
keywords = {1p2pu},
oa-url = {http://poerup.referata.com/w/images/LUOERL.pdf},
title = {Learner Use of Online Educational Resources for Learning (LUOERL)–Final report},
year = {2011},

}

# Key ideas

Large review document on learning with OER, partially focused on UK. Complains of disconnect between “e-learning elite researchers” and OER. Also describes difficulties with using Mendeley for large literature reviews. Mentions P2PU.

## Gaps in literature

The literature on learner use of online educational resources is very immature, with a lack of meta-reviews. The overwhelming majority of published studies do not generalise beyond their particular contexts of study. There is no consistent methodology.

There are significant gaps in the literature: there are almost no meso-level studies, no international comparisons, and very little on learners other than university undergraduates.

## Areas of interest

• learners’ rationale for searching for online resources
• types of online resources being sought
• complexity/granularity of resources being sought
• how resources found are used
• whether learners in some subject areas appear to conduct more searches for online resources than others
• educational level of resources being sought
• location of resources
• extent to which resources are the principal or a supplementary source of learning materials
• whether or not learners are in formal education
• enablers and barriers to use of online resources
• how learners retain access to the resources
• provenance information

## Disconnect

Judged from the perspective of learner use, there is a significant disconnect between the OER community and the elite of the e-learning research community – with a few notable exceptions. We have no reason to believe that this would be different in other areas of OER scholarship/research.

We have the sense that numerous learner use authors do not consider themselves part of (or connected to) the more active OER community. Their publications, therefore, often stand alone, lacking bibliographies of great scale or relevance to a researcher.

It was also interesting that a systematic check of both the leading researchers’ CVs in TELEurope (the STELLAR Network of Excellence) and their associated literature database revealed almost no interest in OER or papers published on it. Indeed, there is an issue both in the UK and the EU of a disconnect between the OER community and the ‘elite’ (perceived or self-styled) of e-learning research – with the exception of The Open University in particular and some key individuals there and at a few other locations. It may be just an artefact of our perspective on them but this seems a little less the case in the Netherlands and Canada (though in some other English-speaking countries the communities seem as disconnected as the UK).

## No harm done

The overwhelmingly positive feedback to the resources across both programmes is encouraging but as stated above rarely proves (from the student perspective) the additional merit of OER. Much of the feedback could apply to closed and/or proprietary online resources. That is to say, much of the enthusiasm is generated either by the ability to access materials away from the classroom/campus or pre-/without enrolment. However, the feedback and other achievement and retention data indicate at the very least ‘no harm done’.

## Interesting papers

‘From Open Content to Open Course Models: Increasing Access and Enabling Global Participation in Higher Education’ (Morgan & Carey 2009) examines implementation of the [open course] model in three studies, relying on data collected from student interviews, instructor observations and reflections, discussion forum transcripts and more. Implementation with undergraduate students in Canada, Mexico and Russia is described. [International]

‘Open Educational Resources Plus Social Software: Threat or opportunity for Canadian Higher Education?’ (Anderson 2008) provides an abstract but useful (and well-cited) overview of OER use in the student context. [Canada]

‘Towards a theory of online learning’ (Anderson 2008) is a highly theoretical approach to understanding online learning. Looks at learner-, knowledge-, community and assessmentcentred contexts, in light of the affordances of the world wide web and Semantic Web. [Canada]

Using student-generated notes as an interface to a digital repository (Harvel 2005) is a dissertation which examines the concept of student notetaking as an interface to a digital repository. [US]

‘From Microtraining and Open Educational Resources (OER) to Master Courses on Geographical Information Systems (GIS)’ summarised at http://www.lunduniversity.lu.se/o.o.i.s?id=12683&postid=1482680

## Highlights (7%)

Findings p. 3

The literature on learner use of online educational resources is very immature, with a lack of meta-reviews. The overwhelming majority of published studies do not generalise beyond their particular contexts of study. There is no consistent methodology. [O.21, O.25.c/h, O.26] p. 3

There are significant gaps in the literature: there are almost no meso-level studies, no international comparisons, and very little on learners other than university undergraduates. [O.19 and O.25.h; O.25.d/c/g; O.18, O.35] p. 3

Only one key study could be found that demonstrated OER having an impact on student attainment. [O.42, 5.4.2.C] p. 3

3. An Open Educational Bibliography of OER: A comprehensive open and editable bibliography of papers and other literature on all aspects of OER should be generated. (The current project can be regarded as a pilot of this.) [O.49] p. 4

Section 1: Background p. 5

1. Sero Consulting Ltd was asked by the Higher Education Academy to undertake a literature review “to provide a greater understanding of the ways in which learners, whether or not in formal education, use online resources to aid their learning experiences and the factors which influence the selection of resources”. It was anticipated that “collectively this work will enable practitioners, policy makers and researchers to adopt more effective evidence-informed or research-informed approaches to their decision-making, research and practice on matters relating to the use of open-educational resources in learning and teaching”. p. 5

2. Twelve areas of interest for the research were proposed by the HEA: learners’ rationale for searching for online resources; types of online resources being sought; complexity/granularity of resources being sought; how resources found are used; whether learners in some subject areas appear to conduct more searches for online resources than others; educational level of resources being sought; location of resources; extent to which resources are the principal or a supplementary source of learning materials; whether or not learners are in formal education; enablers and barriers to use of online resources; how learners retain access to the resources; and provenance information and copyright status of resources being used. p. 5

3. There were two key outputs of which the first was to be a literature review, “to contain an executive summary; outline of methodological approach; identification, selection and analysis of the literature; conceptual perspectives; findings; conclusions, implications and recommendations; and references”. That is this document. p. 5

4. The second key output was to be “a database of literature which is relevant to the review topic and which can be made available by the HEA and JISC to the sector as a searchable resource to facilitate the identification of literature by researchers and practitioners in the future”. After further discussion it was agreed to use Mendeley, a newish low-cost reference management and social networking system developed in the UK. p. 5

12. Two Mendeley groups were set up core to the project: Learner Use of OER and Learner Use of non-OER Online Resources. Other groups were set up round this for less relevant OER material and to support community building. p. 6

15. A long list of search keywords was evolved – OER; OERs; open educational resources; open educational; open resources; open content; education; digital textbooks; (university) digital library users; open/free digital textbooks; (digital) information-seeking behaviour; information seeking; information behaviour; digital information literacy; digital information behaviour; information; research; online; online research; internet; internet research; e-books; e-textbooks; e-journals; student(s); learner(s); user(s); end users; use; student use; student experience; student perceptions; Wikipedia; google; merlot; iTunes U; RLO; RLOs; digital; digital library; digital/information literacy; impact; evidence; scaffolding; digital natives/immigrants; millennials – and variants to cover US spellings. p. 6

18. Thanks to the tagging system a number of contextual variables were captured – country of relevance, educational level, subject etc. There was a preponderance of material about university students and even many of the informal learning examples were about informal learning of individuals who were or wanted to be or had been university students. p. 7

21. Judged from the perspective of learner use, there is a significant disconnect between the OER community and the elite of the e-learning research community – with a few notable exceptions. We have no reason to believe that this would be different in other areas of OER scholarship/research. p. 7

23. The first section is a meta-analysis of all the JISC/HEA OER Programme Pilot Project reports and associated analyses, but it was felt convenient to include OpenLearn also. There are 29 OER Pilot projects and also 201 RePRODUCE projects, making 49 in all. These entries are all in the Learner Use of OER group, but tagged to identify them separately. OTTER was particularly notable in having done a student survey. p. 7

24. The second section is in essence an annotated bibliography of 80 key OER papers, analysed over the five levels of relevance in appropriate detail. Only levels 3, 4, and 5 are in this section – the lower levels are devolved to Appendix 4. p. 7

c) We have the sense that numerous learner use authors do not consider themselves part of (or connected to) the more active OER community. Their publications, therefore, often stand alone, lacking bibliographies of great scale or relevance to a researcher. p. 8

h) Studies involving human subjects tend to focus on the experiences of either small clusters of three to 100 students (typically under 50), or broad swathes of users (upwards of 3,000). Data may be collected through a combination of interviews; surveys; observation; focus groups; assessment; metrics; user logs; emails; and other means. There is little uniformity in methodology here, and so it is difficult to compare these studies and their results. p. 8

27. The third and final section looks at learner use of non-OER online resources. Again it is in essence an annotated bibliography of 153 key non-OER papers, analysed over the five levels of relevance in appropriate detail. Only levels 3, 4, and 5 are in the section – the lower levels are devolved to Appendix 4. p. 9

30. Learners’ rationale for searching for online resources: The OER literature is dominated by the large open university2 and MIT studies. It is debatable how applicable these are to the generality of UK universities and their students. The non-OER literature typically addresses this issue from the standpoint of assessment-driven student behaviour. There is clear room for studies looking at the middle ground. p. 9

32. Complexity/granularity of resources being sought: OER studies tended to confirm the tension between specificity and potential for reuse (seen since the early days of RLOs). Also, students want narrative structure in, or above, the resources they use. The non-OER literature seems to focus more on students typically seeking a single item per search and hints at the assessmentdriven paradigm again – or filling in gaps in an existing narrative not creating their own. It is tempting to draw the conclusion that the two types of study are in fact addressing two different student populations. Again there is clear room for studies looking at the middle ground. p. 9

LUOERL final report 39. Enablers and barriers to use of online resources: It remains true across the wider research that most of the barriers to the use of OER are the same as/or a consequence of more generic barriers to accessing and using technologies for learning. However, the issues of designing learning for the unknown user and the tensions between granularity and the need for scaffolding permeate much of the research. Esslemont (2007) puts it pithily: “There are several interlinked issues related to completeness of content, granularity, copyright, offline access, use, etc., that sometimes limit the effectiveness of material provided. Therefore in order to support the learner we need to understand and support … the learner’s limitations in terms of content selection, access, use and management of their personal knowledge silos on their desktop.” p. 11

8 There is a work-in-progress bibliography at http://www.mendeley.com/groups/1216571/learner-generatedcontent/papers/. p. 16

http://www.mendeley.com/groups/1223261/staff-and-institution-aspects-of-oer/) and National Policies for OER Uptake (http://www.mendeley.com/groups/1075011/nationalpolicies-for-oer-uptake/). p. 23

Open Educational Resources (OER) – http://www.mendeley.com/groups/1003421/openeducational-resources-oer/;

In the non-OER area one of the biggest issues was that very few papers dealt with the learner use of content27. Researchers seem far more comfortable dealing with learner interactions with systems, other students or in some cases tutors. Content seems rather passé to researchers, as if constructivism had won when it clearly has not – and even less so at pre-tertiary level. p. 30

Thus when reviewing the TALL Mindmap it was obvious that the ‘learner use’ aspect is just a small spot of a big network28. p. 30

Students seem to have a very operational view of OER as ‘stuff I can access’. p. 30

4.2 Key contextual variables p. 32

By contextual variable we mean a variable in the environment surrounding the learner or a variable aspect of the learner known prior to their engagement with a mode of learning – such as age, sex, ethnicity, previous educational level, etc. p. 32

There is a fascinating study by KERIS that looks at contextual variables for national OER uptake32. p. 32

• previous educational level;
• subject wished to be studied;
• level of prior information literacy (along some dimension of ‘digital native’). p. 33

There are many examples of studies looking at these in section 5. However, and interestingly, we do not find many studies looking at age, sex, language (or language competence) or ethnicity, in the way one would find in many broader studies of ICT uptake – and would have found to an even greater extent even decades ago. p. 33

There were, however, several notable exceptions among which were: p. 35

• the OTTER (Leicester) project, where 71 learners responded to a learner use survey; p. 35
• ChemistryFM (Lincoln), BERLiN (Nottingham) and OER ADM (Brighton), where students were involved in content creation; p. 35
• Open Space (University College Falmouth), where students were encouraged to collaborate and manipulate resources. p. 35

the OpenSpace project: … found that independent informal learners did not want to be strongly guided through materials, and implemented a more flexible navigation structure that enabled easy movement back and forth38… p. 36

They have also noted that many independent learners want to dip in, just to resources that are of immediate relevance to them, rather than to follow a set path through a p. 36

LUOERL final report unit of study. Projects see this as a more learner-centric approach, that has implications for the way they structure their OERs39. p. 37

The overwhelmingly positive feedback to the resources across both programmes is encouraging but as stated above rarely proves (from the student perspective) the additional merit of OER. Much of the feedback could apply to closed and/or proprietary online resources. That is to say, much of the enthusiasm is generated either by the ability to access materials away from the classroom/campus or pre-/without enrolment. However, the feedback and other achievement and retention data indicate at the very least ‘no harm done’. Taken in the context of the potential of OER to expand choice through tailored offers this looks significant. p. 42

Their research report Open Transferable Technology-enabled Educational Resources (OTTER) project: Stakeholder views on open educational resources by Dr. Samuel Nikoi in June 201064 p. 43

1. DESIGNING FOR INNOVATION AROUND OER (IN JIME SPECIAL), by Andy Lane66 p. 44

Abstract: This paper argues that designing collections of ‘closed’ educational resources (content and technologies) for use by specific student cohorts and collections of open educational resources for use by any ‘learner’ require different design approaches … there appears to be a paradox in that learning design assumes a reasonably well-known and well-defined student audience with presumed learning needs and mediating technologies while OER are exposed to a multitude of potential learners, both formal and informal, with unknown learning needs and using diverse technologies … it is necessary to design for greater flexibility so as to allow the users to adapt their use of the innovative solution for their own requirements once it has been deployed. The use of such an ‘innofusion’ approach for OER is highlighted using the case study of OpenLearn. p. 44

The scale of the impact of these six trends on education is difficult to predict as they have emerged at a fairly rapid rate and without any end in sight to such developments. Will many people actually construct their own, personal learning experiences? Will many modify what others have produced? However in terms of enabling innovation these different factors do allow learners to make such choices if they wish. They are not as constrained by pre-defined products and processes and can experiment as much as they might want to. p. 44

66 Lane (2010) see http://jime.open.ac.uk/article/2010-2/pdf. For more on the “six trends” see David Wiley and John Hilton, Openness, Dynamic Specialization, and the Disaggregated Future of Higher Education, IRRODL, Vol. 10 (5), 2009, http://www.irrodl.org/index.php/irrodl/article/view/768/1414. p. 44

In other words, when we expand our literature search behind the scope of JISC/HEA OER, we find that – although there have been several thousand69 papers published that address some aspect of OER – there is a surprisingly limited selection of studies and publications directly (and primarily) addressing the topic of ‘learner use’. p. 46

We have found only about 80 additional publications deemed as directly relevant to the topic. (For more details see section 3.) That said, these items examine the experiences of individuals exposed to a fairly broad and representative range of open content initiatives and programmes, and we have been able to extract some healthy conclusions in section 5.2.5 below. p. 46

Here we see OERs that are largely programme-specific, e.g. they have been produced in conjunction with a particular university course; studies tend to stay close to home, i.e. to examine a local user population. p. 47

5. ‘Open Content: when is it effective educationally?’ (Lane 2007) delivers a well-researched and well-cited general analysis of learner-content interaction (and other topics). He concludes that OERs should be presented in an environment that allows learners and creators to communicate with each other, adding a sense-making layer to the original material. [UK] p. 47

6. ‘The networking effects of OER’ (Lane et al. 2009) reviews various ways of analysing student use of online materials via the OU’s OpenLearn (and several other communities). [UK] p. 48

7. ‘Impact of Open Educational Resources in The Netherlands’ (Schuwer et al. 2007) analyses via survey which students choose to use OER and how they do so. [Netherlands] p. 48

11. ‘Revolutionizing Education through Innovation: Can Openness Transform Teaching and Learning?’ (Casserly & Smith 2008) provides descriptions of select OERs and how students interact with them. [US] p. 48

3. ‘Evidence Hub for Open Education Resources’ website (anon 2011) offers a fledging evidence base on learner use of OER, addressing all the right topics for this critical area (albeit in preliminary alpha format). [UK] p. 49

5. ‘Exploring user types and what users seek in an open content based educational resource’ (Godwin & McAndrew 2008) provides an overview of the online activity of approximately 65,000 OpenLearn users registered with the OER site: select users identify how they view OpenLearn content and its relationship to their learning. [UK] p. 49

6. ‘Repurposing with a purpose: A story with a happy ending’ (Greaves et al. 2010) equates reusable learning objects (RLOs) with OER, considering its impact in helping to improve the student learning experience and student success rates. Student achievement in a first-year science module before and after the introduction of an OER is explored. [UK] p. 49

8. ‘Open Learning Network: the evidence of OER impact’ (McAndrew & Cropper 2010) directly addresses the need to study the impact of OER on learners. [UK] p. 49

9. OpenLearn: Research Report 2006-2008 (McAndrew et al. 2008) takes advantage of automated collection of data through tracking of Moodle logs, cross-correlation of data with other Open University sites, and Google Analytics. The report identifies patterns in learner experience, drawing on OpenLearn participants by studying their actions, carrying out surveys and analysing responses – in the context of understanding how open content works, and how its learners see themselves. [UK] p. 49

11. ‘The feasibility of capturing learner interactions based on logs informed by eye-tracking and remote observation studies’ (San Diego & McAndrew 2009) attempts to illustrate the feasibility of examining, identifying and observing ‘learning’ and ‘browsing’ actions based on OU OpenLearn user logs. [UK] p. 49

12. The OER Use and Reuse Landscape (TALL, 2011) mindmap divides a linear bibliography into sections by stakeholder, e.g. “what attributes of OERs determine their usefulness (evidencebased)”. Learner information is sparse but present and well organised for review. [UK] p. 49

13. ‘Who is using Open Educational Resources?’ (White 2010) is a slight but relevant blog posting, offering links of interest. [UK] p. 49

LUOERL final report 14. ‘Sharing and reuse in OER: experiences gained from open reusable learning objects in health’ (Windle et al. 2010) offers a good amount of student focus and review of student feedback, highlighting the importance of OER community, ownership and empowerment. Published figures portray learner OER usage data. [UK] p. 50

15. ‘The Advancement of Lifelong Learning Through Open Educational Resources in an Open and Flexible (Self) Learning Context’ (Mulder 2007) surveys 35 students about their OER usage, attitudes and preferences. Issues of quality and appeal are briefly addressed. [Netherlands] p. 50

16. ‘Why Understanding the Use and Users of Open Education Matters’ (Harley 2008) probes the importance of gathering data on student demand for OER, placing this in the context of student experiences. [US] p. 50

18. ‘Incentives and Disincentives for the Use of OpenCourseWare’ (Arendt & Shelton 2009) provides a fascinating glimpse into Utah residents’ general attitude towards and (limited) past experience of OER, as determined by a state-wide survey. [US] p. 50

20. ‘Using Open Educational Resources To Help Students Understand The Sub-Prime Lending Crisis’ (McDowell 2010) examines classroom use of OERs from the Kahn Academy. Feedback from students explores learner confidence, and reactions to and attitudes towards OER. [US] p. 50

2. ‘Am I good enough? The mediated use of open educational resources to empower learners in excluded communities’ (Lane 2008) proffers third-party narratives, discussing four cases in which mediated OER have been delivered. [UK] p. 50

3. ‘How long will it take me? Assessing appropriate study times for open educational resources’ (Lane 2007) reviews early evidence of how long OpenLearn self-learners spend studying. Research questionnaires are administered but not discussed in depth. [UK] ￼￼ p. 50

LUOERL final report 4. ‘Re-invigorating openness at The Open University: the role of Open Educational Resources’ (Lane & Gourley 2009) details the use of OpenLearn, e.g. via select data from surveys of registered users. [UK] p. 51

5. OpenLearn: Researching open content in education (McAndrew & Watts 2007) conference proceedings contain a number of relevant ‘user experience’ talks. [UK] p. 51

8. Oriole (Oriole 2011) project website reveals a focus on investigating the influence of digital/open educational resources in teaching and learning. [UK] p. 51

10. ‘From Open Content to Open Course Models: Increasing Access and Enabling Global Participation in Higher Education’ (Morgan & Carey 2009) examines implementation of the [open course] model in three studies, relying on data collected from student interviews, instructor observations and reflections, discussion forum transcripts and more. Implementation with undergraduate students in Canada, Mexico and Russia is described. [International] p. 51

12. ‘Open Educational Resources Plus Social Software: Threat or opportunity for Canadian Higher Education?’ (Anderson 2008) provides an abstract but useful (and well-cited) overview of OER use in the student context. [Canada] p. 51

LUOERL final report 15. ‘Open Educational Resources: New Possibilities for Change and Sustainability’ (Friesen 2009) analyses how MIT OpenCourseWare is used by learners worldwide, and may serve as a model. [US] p. 52

1. ‘Student digital information-seeking behaviour in context’ (Nicholas et al. 2009) provides an excellent overview and bibliography in this area. To analyse student behaviour the team uses logs from two digital journal libraries, Blackwell Synergy and OhioLINK, and one e-book collection (Oxford Scholarship Online). The study shows a distinctive form of informationseeking behaviour associated with students as compared to other academics: students ￼￼ p. 54

constituted the biggest users in relation to sessions and pages viewed, and were more likely to undertake longer online sessions. Undergraduates and postgraduates were moreover the most likely users of university library links to access scholarly databases, suggesting an important ‘hot link’ role for libraries. [UK] p. 55

9. ‘Towards a theory of online learning’ (Anderson 2008) is a highly theoretical approach to understanding online learning. Looks at learner-, knowledge-, communityand assessmentcentred contexts, in light of the affordances of the world wide web and Semantic Web. [Canada] p. 58

22. Using student-generated notes as an interface to a digital repository (Harvel 2005) is a dissertation which examines the concept of student notetaking as an interface to a digital repository. [US] p. 66

1. Learners’ rationale for searching for online resources p. 67

• Of students using OER, 44% said it was to enhance personal knowledge, 39% said it was to complement a course and 12% said it was to plan a course of study. p. 67
• Of self-learners using OER, 41% said it was to explore interests outside of the professional field, 20% said it was to plan future study, 17% said it was to review basic concepts in their field and 11% said it was to remain current in their field. (Carson 2009) p. 67

LUOERL final report The Open Universiteit Nederland (OUNL) OpenER project was aimed specifically at those who had not “successfully attended higher education” (Schuwer et al. 2007). Reasons given (respondents were allowed more than one choice) for visiting the site were:

• to follow a free course: 64%;
• to test if they are able to cope with university courses: 23%;
• to try out different domains: 38%;
• to try out a study at the Open Universiteit Nederland: 43%;
• to use the material in their own courses: 5%. (Schuwer et al. 2007) p. 68

Wilson surveyed students prior to OpenLearn starting, with responses revealing a hunger for assessment (90%), qualifications (89%) and tutorials (64%) (Wilson 2008). p. 72

Non-OER: The research literature we have analysed seems silent on this topic but it is known that for at least ten years there have been online courses (and not only at open universities) where online resources form the principal source of learning materials74. p. 75

‘From Microtraining and Open Educational Resources (OER) to Master Courses on Geographical Information Systems (GIS)’ summarised at http://www.lunduniversity.lu.se/o.o.i.s?id=12683&postid=1482680. p. 75

74 For one contemporary example, see P2PU (http://p2pu.org/en/). p. 75

It was also interesting that a systematic check of both the leading researchers’ CVs in TELEurope (the STELLAR Network of Excellence) and their associated literature database revealed almost no interest in OER or papers published on it. Indeed, there is an issue both in the UK and the EU of a disconnect between the OER community and the ‘elite’ (perceived or self-styled84) of e-learning research – with the exception of The Open University in particular and some key individuals there and at a few other locations. It may be just an artefact of our perspective on them but this seems a little less the case in the Netherlands and Canada (though in some other English-speaking countries the communities seem as disconnected as the UK). p. 82

7.1 Bibliography on learner use of OER p. 88

7.2 Bibliography on learner use of non-OER online resources p. 100