Research in Learning Technology

ALT's Open Access Journal

Policy implementation staircase

The implementation staircase (Reynolds and Saunders 1987) is one useful conceptualisation employed by policy implementation studies (Trowler 2008).

The staircase expresses the stages of mediation that policy implementation goes through (Figure 1). Each level receives and re-interprets policy from actors in the level above, and communicates these adapted ideas to the level below. As mentioned above, studies on minimum VLE standards tend to explore academics’ VLE practice, or their perceptions of it (Ahmed and Morley 2010; Reed 2014), which means a focus on the lower levels of the implementation staircase. However, little has been written about the work occurring at the top, institutional level of the implementation staircase. One exception is the aforementioned work of Fresen, Hill, and Geng (2014) who focus on the mediating role of the development team whose remit was to design and develop a VLE template that embodied and communicated the institutional VLE standard policy to further levels in the staircase.

[Figure ID: F0001] Figure 1. Implementation of the QEF policy.

A VLE template is a virtual artefact, a set of webpages that appear automatically in each VLE module as an empty shell ready for customisation by the teaching staff. Figure 2 shows such a template from a module in music in a Blackboard VLE. At the case study university and at the time of this paper's publication (2016/17), the stipulated minimum requirement for each VLE module was six core elements. They are (1) module staff details, (2) module overview, (3) learning resources, (4) assessment, (5) exam resources, and (6) coursework & exam feedback at the cohort level. As Figure 2 shows, the default module menu structure on the left prompts staff to include these core elements. The provision of ‘streamed lectures’ was optional at this time. Each module menu is either a link to a content area or a tool within Blackboard (e.g. staff contacts).

[Figure ID: F0002] Figure 2. A sample VLE template – as it appears blank, ready for customisation.

Academics can choose to customise this blank module space. They can amend or add to the module structure, add other tools or branding/visual elements to customise their module, and also copy course content over from previous years. In other words, they re-interpret this central policy requirement to their own teaching context. Some contents are automatically pre-populated from other university systems (e.g. the Module Overview page). As Fresen, Hill, and Gang (2014) posited, such a template guides staff towards providing the identified set of elements. Different middle-level managers (Departments or Schools) may decide to implement their own version of the VLE template. Academics can choose to ignore the template, too.

What the above scenarios suggest is that the template can directly impact on both higher and lower levels in the chain; the latter by reaching academics directly. This direct link between VLE Baseline Group and Staff is depicted with blue arrows (Figure 3). The Group's collective human activity is, therefore, a tangible artefact of institutional vision (Gunn 2010). As opposed to traditional educational policy implementations, this feature seems particular to TEL. In the latter, each level's understanding of policy is mediated via a higher level (Saunders, Dim, and Dempster 2015). Since it is likely that there will be teams with a similar mediating role in charge of developing artefacts that encapsulate institutional TEL policy, the nature of their work is worthy of exploration.

[Figure ID: F0003] Figure 3. Implementation staircase of the VLE Baseline development.

Activity theory

Activity theory was chosen to explore the nature of this work, because it focuses on human agency as they act with technology in a socio-historical/cultural context (Engeström 1987), unfolding over time (Kaptelinin and Nardi 2006). Its derived methodology, activity system analysis (Yamagata-Lynch 2010), has informed various evaluation studies of educational technology interventions (Abdullah 2014; Bligh and Coyle 2013; Kaptelinin and Nardi 2006).

An activity system is characterised by specific division of labour, tools, rules and community (Bligh and Coyle 2013; Engeström 1987). Activity theory enables understanding systemic tensions (Yamagata-Lynch 2010). By focusing on such systemic contradictions, transformation or development can be encouraged (Engeström 1999). This is useful for universities who want to gain insights into their TEL practices at a systemic level.

Both the implementation staircase and the activity system analysis have mediation as a central concept. In an activity system, there are mediating artefacts that enable people as subjects in the world ‘constructing and instantiating their intentions and desires as objects’ (Bennett 2010, p. 10). Accordingly, Figure 4 represents the chain in which each object of the previous level's activity system can become the mediating artefact of the system in the next level below it. The conclusion will discuss the strengths and weaknesses of both frameworks, and why their combination worked well for this study.

[Figure ID: F0004] Figure 4. The implementation staircase of activity systems.

Case study

The VLE Baseline Group was an example of a small, unique and bounded population at the UK study university. These features made case study methodology a natural choice (Cohen, Manion, and Morrison 2007; Denzin and Lincoln 2011). A strength of this methodology is that it could help understand an in-depth situation in detail (Yin 2009) which suited the qualitatively oriented research question (Prescott 2013 quoting Tellis 1997) aimed at exploring the nature of the work of the Baseline Group.

The five components of case study research design are: the study questions, its propositions, the unit of analysis, logic linking data to propositions and criteria for interpreting findings (Yin 2009). The unit of analysis of this study is the nature of the work of the Baseline Group. The research questions have been discussed above, premised on the proposition that the impact of the Group's work, because of its importance in the implementation staircase will have influence on policy implementation. The logic that links data to this proposition is drawn on from an activity system analysis as discussed above, which is the theoretical underpinning for this exploratory case study (Yin 2009). Another feature of case study methodology is its in-depth data collection using multiple sources of data (Creswell 2013), which are discussed next.

Data collection

The author was part of the wider TEL Strategy Group: attended the meetings where the Baseline Group's work was reported, and collected observational data via team meetings, consult documents and reports produced by the Group. The theoretical framework and the chosen methodology aligned themselves well to qualitative semi-structured interviews. Five one-to-one interviews were conducted to ensure representation of the perspective of each professional role: learning technologist, programmer, academic and two senior managers. Ethical approval was granted by Lancaster University. The 50- to 70-minute interviews were transcribed and sent to participants for approval. Two paragraphs were removed in one transcript.

Data analysis

Data analysis was carried out via activity system analysis. Transcripts were coded for each activity system element in ATLAS.ti: the Subjects, the Object, the Outcome, and the environment: the Rules, Community and Division of labour (Engeström 1999). Different kinds of mediating artefacts were also identified, including language, policies, technology systems and expertise. As activity system analysis pays particular attention to contradictions and transformations (Abdullah 2014; Bligh and Coyle 2013), these were also coded.

Limitations

Potential bias arising from being an insider researcher (Ludvigsen and Digernes 2009; Mercer 2007) need to be acknowledged. Critiques of case study methodology (see Yin 2009), and of activity theory (see Yamagata-Lynch 2010), comment on their lack of generalisability. The author would concur with Bligh and Coyle (2013) who do not view the study's contextual nature as a limitation.

Success factors

Rules and division of labour

The two key tasks of the group were to agree what elements would be required in the VLE Baseline and what could be automated. Group members considered the activity successful in implementing an institutional template for customisation by staff, although also reported frustrations with technological limitations that prevented some elements to be automated.

As members reflected, rules and division of labour (Figure 6), the clarity of the task to be completed within a given timeframe contributed to this achievement, which was the Easter VLE module roll-over date. Another success factor, which hints at the community aspect (Engeström 1987), was given, concerned the nature of the project:

[The VLE Baseline] is relatively easy to sell to people. It is self-evident that we should be giving all students the same. Most people aren't going to argue with that. So there wasn't a big hearts and minds … it was quite a logical project. (I3)

[Figure ID: F0006] Figure 6. Success factor 1.

Despite the fact that the institutional environment had undergone significant changes as the project progressed, the project was ‘this wonderfully calm piece of work going on, when there was this storm: anything that could change, has changed around it (I3)’. Clarity of project remit and schedule seem to overcome even major environmental changes.

Mediating artefacts

Mediating artefacts contributed to the effective working of the Baseline Group in two ways.

First, the Group's collective activity benefited from each member drawing from their own different mediating artefacts in achieving their collective objective (Figure 7). The Group comprised a range of professionals, which meant that every member contributed their own different area of expertise. The way their different areas of expertise contributed to the VLE template can be illustrated by the design of the module overview page. This was an automated page displaying selected module information to students from the module specification system. The Group had decided on what information would be useful for students to see within the given technological possibilities and constraints. It was clear that such decisions were possible because some Group members had access to their own different mediating artefacts and tools. Naturally, the programmer had technical expertise and a deep knowledge of the institutional systems, which was necessary to see what was and what was not possible to achieve, and what could be automated and what could not. Whilst knowledge of ‘common’ academic practice by academics and the learning technologist helped differentiating useful and less useful areas of information to include:

I might think, ‘that particular field is probably not necessary’; whereas [Academic1] might say, actually, I'd quite like to see that in there. (I2)

[Figure ID: F0007] Figure 7. Success factor 2.

Second, the Group members drew on shared mediating artefacts in their discussions and decisions. The Group's objective was to develop a VLE Baseline specifying the core elements to be present in each VLE module space, for example, module information and staff contacts. The approach of the Group was to make it as easy as possible for staff to provide the required information by automatically pre-populating as many of these core elements as possible. In the case of the module overview page, all members needed to be aware of, or understand, explanations of VLE and other systems interoperated, and what module information was stored in the module specification system. This shared mediating tool was needed for all members of the Group. The learning technologist had a special role in translating so that all Group members understood both the pedagogic and the technological implications. Technological understanding of the academic representative was important so that the requirements represented on behalf of the academic community remained within technologically feasible bounds. This technical awareness also acted as a smoothing conduit between the wider academic community and computing services. It helped the academics translate to colleagues what was possible to achieve and what was not. It showed to other academics that computing services could be just as frustrated by the technological limitations as they were:

If [academics] can ask for the right kind of sensible things that we can do technically. And also understand where and why we can't do certain things. Then that would be great. Because they can feed back our frustrations to their department to people who aren't necessarily as technical. (I2)

In summary, the collective agency of the Group was effective when it comprised both individual expertise in one or more of these areas (depth) and shared expertise in others (breadth), in a way that all of the identified knowledge areas were covered:

• Technology, including a knowledge of institutional systems.
• Pedagogy, a knowledge of learning and teaching.
• Academic Practice, forms of research and scholarship, teaching and academic administration and management within a university context.

Programmers needed an in-depth knowledge of Technology, including the VLE and other institutional systems. Academics contributed knowledge of Academic Practice, that is, specifically, how they and colleagues used the VLE for student learning. Their technological know-how was key in successfully engaging programmers in requiring features of the VLE template that were feasible to implement and were able to communicate limitations in accessible language to their own colleagues. Both academics and learning technologists had the required knowledge of Pedagogy to influence technological decisions in the right direction. They were also users of some of the other institutional systems concerned (reading list and module specification). The more localised knowledge of VLE practice of academics was complemented by the breadth of knowledge of learning technologists of existing institutional VLE Practice. Learning technologists had an in-depth knowledge of the pedagogical uses of the VLE, pedagogy, technology and other institutions’ Baseline requirements as a result of national benchmarking. They also acted in a bridging role between the academics and the programmers through the mediation of language. As one interviewee commented, one success factor of this project was the good working relationship and communication between the learning technologist and the programmer.

Contradiction in object: student expectations versus pedagogical rationale

During the development of the VLE Baseline, one contradiction that emerged was the difference between the values of two intersecting systems, the VLE Baseline Group's activity and the student expectations of VLE provision (Figure 8).

[Figure ID: F0008] Figure 8. Contradiction 1: difference in value systems.

One example of this tension was, for instance, students’ requests for exam papers. Whilst certain academics did not have a problem providing exam papers on the VLE, others had good pedagogical reasons for not doing so. This confirms previous studies which found students’ digital expectations to be access or content-driven (Beetham and White 2013), and do not necessarily value, or even expect, what pedagogically may be beneficial for them, and why VLE minimum standards tend to focus on content provision (Varga-Atkins and Roberts 2016). To work with this contradiction, the Group decided to include a module area called ‘exam resources’ in the default VLE template. Whilst ‘exam resources’ did not strictly prescribe the provision of exam papers to be uploaded to the VLE, its mere presence was meant to encourage staff to populate it with useful resources. This solution hoped to satisfy student expectations in a way that did not jeopardise pedagogical creativity. This example also demonstrates how the Group developed the template with pedagogical value as its underpinning principle (Hill, Fresen, and Geng 2012) to influence re-interpretation of policy.

Secondary and quaternary contradiction: reclaiming agency

Although the Group's task had clarity, there were some tensions reported in terms of agency; For instance, concerning the origins of the project Brief (Figure 9).

[Figure ID: F0009] Figure 9. Contradiction 2: reclaiming agency.

The project Brief was an important mediating artefact for the Group to achieve their object. All agreed that the Student Report was to some extent the project Brief, which contained students’ wish-list for the VLE Baseline. Reasons for this uncertainty could have been because Group members were not able to easily recall events that happened 2–3 years prior on-the-spot during the interview, or owing to the lack of formal project documentation. Whether they were given this list to implement without the opportunity to debate, or whether they had agency in terms of going beyond the wish-list was unclear. The Group's individual accounts slightly differed on this point. One view was that there was some agency around the Brief:

I think there was a question of what we could do, as well as what we would do. (I4)

Another reported that they felt the Brief was not negotiable:

what you had was a ready-made thing [students’ wish-list] handed over, to then go and do something about without not much rationale attached to it. (I1)

This lack of agency was professionally challenging for this latter Group member who would have liked to have more agency to scope and shape this brief since they had the necessary institutional and national knowledge of VLE standards. The uncritical acceptance of student views with regard to the reported VLE inconsistency caused some frustration:

Is it a problem? Is it a problem of [student] perception as opposed to something real? (I1)

This combined with a quaternary contradiction between elements of neighbouring activity systems was a source of transformation (Engeström 1987), in other words opened window on a solution within this context.

The intended objective of the Baseline development would have been to arrive at a Baseline template containing the students’ wish-list, which would have led staff towards a tick-box approach in their take-up of the Baseline (see blue dotted line in Figure 10). This was at odds with Interviewee 1's motivation using the Baseline as a mediating artefact for staff to pedagogically re-think their VLE use, similarly to the course templates designed by Fresen, Hill, and Geng (2014) – see this represented in Figure 10 (activity system of Interviewee 1). This was a quaternary contradiction between two intersecting activity systems: in the VLE Baseline development, the VLE Baseline template was the object; and in the activity system of Interviewee 1, the template acted as a mediating artefact.

[Figure ID: F0010] Figure 10. Transformation arising from contradiction 2.

To turn this contradiction into reclaimed agency transforming systemic contradictions to match their own goals, Interviewee 1 paid careful attention to the wording of the Baseline instructions so that it would raise staff aspirations to reconsider their VLE use from a pedagogical perspective:

[I would] tweak the agenda to move the language away from things to do with compulsion: ‘You must’, to move it more towards aspiration. Because there is that tension built into something like a Baseline. (I1)

Interviewee 1 purposefully used language as a mediating tool to match their own objective in texts such as instructions, descriptions and the Baseline template. This also highlighted that the linguistic discourse contained within Baseline artefacts acted as powerful mediators, influencing policy re-interpretation lower down the chain of the TEL implementation staircase (Figure 10) (Reynolds and Saunders 1987).

Implications to practice

Success factors demonstrated the importance of good project management principles, establishing clear rules and division of labour for such an institutional TEL development group. The finding that Group members needed to draw on both different and shared mediating artefacts supports the conclusion that the composition of the group and the expertise of its members are crucial for project success. Situated and combined knowledge of technology, including a knowledge of institutional systems, pedagogy and academic practice, are essential to be covered by Group members to different levels of expertise.

However, some level of technical awareness and understanding was crucial for the effective working of this institutional group in order to best represent academic and pedagogic concerns. The lead learning technologist had a key role in bridging and translating between these areas. Based on this, similar institutional TEL implementations would benefit from being led by learning technologists, rather than academic or computing staff. Tensions around the lack of clarity of the project Brief suggested that documenting the project brief and key decisions taken along its development are highly recommended for such projects.

Implications to theory

One theoretical contribution of this paper is an enhanced representation of a TEL policy implementation staircase (Figure 11). This paper demonstrated that in the case of a TEL policy implementation process, there is another level that could be considered in addition to those identified in previous studies.

[Figure ID: F0011] Figure 11. Adapted implementation staircase of a TEL policy.

This additional stage concerns the work of the VLE Baseline Group. Although its members were not necessarily situated at the more senior, top end of the institutional hierarchy, their work has significant, top-level impact on how the policy gets interpreted by the respective lower levels in the staircase. They are what Engeström (2010) calls ‘knotworkers’ coming together temporarily for a particular project as a knot gets tied on a rope. The outcome of their mediatory work is an institutional VLE template (Fresen, Hill, and Geng 2014). The potential reach of their impact is pictured in blue in Figure 11, paving way to further customisation and re-interpretation of policy in the form of Departmental/School level VLE templates (Figure 12).

[Figure ID: F0012] Figure 12. A School-level customisation of the central VLE Baseline template.

Representations of TEL policy implementations, therefore, need to consider the presence of such TEL development groups. This study also highlighted that in the current HE context, students have a much stronger voice in influencing policy. Instead of being represented at the bottom (Figure 1), students now also occupy a place at the top of an educational/TEL policy implementation staircase (Figure 11).

Using activity system analysis allowed insights into how such a significant activity could work effectively and how the group dynamic, shared and different expertise and individual agency influenced the emerging the VLE Baseline template itself. Results pointed to a necessary combination of Group members’ situated knowledge. Linguistic discourse was one powerful mediating tool in the hand of human actors to both work effectively as a group as well as transform systemic contradictions when needed. These findings resonate with the Technology, Pedagogy and Content Knowledge model (Mishra and Koehler 2006), which describes the overlapping and situated knowledge areas TEL practitioners need to draw on. Future research could test the relevance of this model in other institutional TEL innovation projects.

The above implications to theory demonstrate the way that the strengths and weaknesses of the two theories combined into enhancing this study. The implementation staircase model helped to situate the work of the Baseline Group at macro-level, within an institutional context of policy implementation. Whilst using activity theory, it was possible to analyse the nature of the Group's work at a micro-level, in terms of factors or success and how the Group members overcome contradictions through expansive learning to achieve their brief of developing an institutional VLE template. Finally, although the study was carried out in a specific context, it has potential relevance to other institutional TEL implementation projects that draw on a cross-institutional team.