Research in Learning Technology

ALT's Open Access Journal

Familiarisation

The familiarisation process began with two members of the team independently immersing themselves in the data and reading for items of interest. This familiarisation phase involved ‘text segmentation’ (see Guest, Macqueen, and Namey 2012). This is an iterative process of defining text boundaries around identified features of interest. This process involved the analysts continually questioning where meaning begins, ends, intersects and overlaps in the text (see Guest, Macqueen, and Namey 2012). Whilst this phase is presented as the first sequentially, this process of familiarisation permeates all subsequent sequences as the analysts continually question and improve the themes identified refining their application of them. This allowed the researchers to compare bounded, segmented text fragments (Guest, Macqueen, and Namey 2012) in order to:

1. Evaluate and document the overall utility of the data
2. Aid the exploration of thematic elements in terms of similarity, dissimilarity and relationships

Thematic framework

The thematic framework was developed through the two independent analysts from the familiarisation phase comparing segmented text to tag arising themes or codes. At the most arbitrary level, this involved the application of what Guest, Macqueen, and Namey (2012), termed ‘key-word-in-context’ (KWIC). This was the identification of a word as the locus for the theme or concept being tagged. Saldaña defines it as ‘a phrase or sentence that identifies what a unit of data is about and/or what it means’ (Saldaña 2009, p. 139). Metaphorically this is the placing of the pin in a map. The benefit of using KWIC in a large data set is that it allows rapid searching as you can quickly search for these KWIC words. Also the analyst is able to quickly search for synonyms allowing rapid tagging of schema and development of a codebook for application. By embedding the synonym searching in the method, this strengthens the process expanding a literal search into a broader concept search. The process culminated in the production of the codebooks in Tables 1 and 2.

[Figure ID: F0001] Figure 1. Institutional overview of ‘best’ themes.

[Figure ID: F0002] Figure 2. Institutional overview themes ‘to improve’.

Indexing

A team of eight TELAs analysed the data using the codebooks derived from the framework setting stage. During the indexing phase several analysts revisited the familiarisation phase clarifying and improving the coding which could be more formally built into the method. MacQueen et al. (2008) present an exemplified codebook template which contains a code label which is a short descriptive mnemonic. This is further expanded by a short definition. The codebook utilised here in the indexing phase comprised a hybrid version encapsulating a blend of the code label and short definition. However, the scale of the thematic analysis here perhaps justifies the condensing of the codebook.

Charting

The charting phase involved a higher level of abstraction and organisation. In searching for connective meanings and interpretation, the analytical methodology introduced categorisation. This involved the introduction of structure onto the codes or themes through the use of meta-themes. The grouping of themes into linked categories involved the application of a higher level of meaning not directly visible in the data. The meta-themes were derived from the original codebooks and are shown in Table 3.

[Figure ID: F0003] Figure 3. Departmental hotspot analysis – areas of best themes – horizontal correlation demonstrates persistently positive reporting of a specific theme across multiple departments. Vertical correlation demonstrates persistently positive reporting of multiple themes within a single department.

This codification is demonstrated in adaptations of Tables 1 and 2 to produce Tables 4 and 5, respectively.

[Figure ID: F0004] Figure 4. Institutional hotspot analysis areas for improvement – horizontal correlation demonstrates persistently negative reporting of a specific theme across multiple departments. Vertical correlation demonstrates persistently negative reporting of multiple themes within a single department.

[Figure ID: F0005] Figure 5. Mirrored meta-themes.

The revised codes were charted to produce initial visualisations of the data at an institutional and faculty level. Initially, these were bar charts as shown in Figures 1 and 2. Here the occurrence of the codes was plotted across the meta-themes and themes. Firstly the ‘best’ elements of the programme/unit were plotted.

Secondly the areas for ‘improvement’ were plotted as shown in Figure 2.

Interpretation of these initial visualisations indicated a degree of ‘mirroring’ of certain meta-themes and themes. For example, ‘Communication’ was reported a ‘positive’ facet with 127 reports of ‘Communication’ as a strength of the unit. Yet it was also reported as a ‘negative’ with 155 incidents where students indicated perhaps a quicker response or clearer communications were required.

This ‘mirroring’ was not a contrary report by the students but indicated an inconsistent experience across programmes. Used as a comparator, ‘Communication’ could be reported positively in one unit that was utilised as a benchmark and to report a ‘poor’ experience in a different unit. So students are not commenting differently about the same unit but are highlighting that communications in some units/programmes are perceived as effective and in others are not. Students are positive about a fast response to questions and highly frustrated where this is not the case.

In investigating this concept of ‘mirroring’ further, subsequent visualisations were produced using Tableau as an image medium. These included hotspot analysis to drill into the data at a departmental level to inform the faculty reports that were produced. Also ‘mirrored’ charts were visualised to investigate the mirroring concept further.

Mapping and interpreting

The mapping of hotspots across the themes and departments generated visualisations that gave a clear perception of where issues or strengths lie. They also clearly indicate departments that can be exercised for sharing good practice and for identifying ones that perhaps need greater levels of support. In Figure 3, well performing departments can be seen by strong vertical colour bands that indicate departments performing well across numerous facets and could be leveraged by faculties to lead in good practice sharing. Strong horizontal banding shows strong performance in a particular theme across numerous areas, which perhaps is more pertinent for institutional level attention.

Similarly, the representation of areas for improvement indicates areas of practice that need development at an institutional level by strong colour banding in horizontal themes. These may indicate areas of focus for institutional concern. Strong vertical colour banding indicates particular departments that may need further support or interventions at a faculty level to support development. This can be seen in Figure 4, here several themes show particularly high colour strength indicating a high degree of student comment. For example, areas of concern in Figure 4 are Communications and the use of PowerPoint slides in the VLE.

The researchers wished to avoid alienating cooperative academic colleagues and the reports produced utilising the hot spot analysis where produced on a per faculty basis with anonymised results shown for other faculties. This allowed the institutional view but moved away from departments being able to compare with each other outside of their own faculties. The hotspot analysis formulated part of the reports they were sent to Faculty Executive Groups for action and comment. Some of the results of which are discussed later.

Further mappings of the ‘mirroring’ effect were also produced to investigate further which themes were prevalent. The early visualisations as displayed in Figures 1 and 2 were considered too visually busy to be easily interpreted. The categories were compared in an adapted area graph with positive results displayed on the top and negative results presented underneath. As seen from Table 3, the meta-themes across both the ‘best’ and ‘to improve’ elements were not uniform, so outlier elements were removed from the analysis. The plot shown in Figure 5 shows the resulting comparison between the meta-themes that remained for comparison.

The resulting visualisation clearly identified five significant areas that the students commented upon in terms of their experience. The distinct yellow plot referred to ‘general positive comments about Moodle’ which the analysts discarded as it appeared a catchall rather than a meaningful research proposition. The remaining four characteristics that received significant student comment focused on:

1. Effective communications
2. Moodle organisation and high quality materials
3. Use of audio visual resources
4. Desire for ‘interactive’ lectures.
5. Moodle a ‘good’ thing (yellow area discounted as discussed above)

The subsequent phases sought to explore the staff perception in relation to the outcomes in order to attempt to determine if there was awareness of these issues amongst academic colleagues and to suggest possible interventions.

At this juncture, the role of the TELAs was perhaps seen as being a lens through which the data could be interpreted on behalf of academic colleagues. It was felt by the researchers that the expressions of the student voice centred on some key concepts with regard to the student experience within the environs of technology.

For example, ‘Effective Communications’: There is a plethora of possibilities with regard to what these means on practical level. Does it refer to speedy response to email queries? Does it refer to meaningful communication rather than being time specific? Does it refer to effective contextualising of resources made available in the VLE for students to use by using effective and efficient instructions? There are a variety of possibilities. As such the researchers felt that there may be a role for them in positioning the student voice feedback. Where this interpretation should appear in the sequence of the research is still a moot point. In this study, it was positioned following staff investigations but perhaps in subsequent iterations of this research the action research cycle might position this interpretive phase between student data analysis and the staff data collection. These results highlighted target areas for intervention. From a TEL perspective, interpretation of these results pointed towards clear intervention strategies.

Effective Communications was interpreted as a need to develop a mutual expectation among staff and students around clarity of communication and setting expectations around the immediacy of communication. Also, it pointed towards strategies for minimising communication need by providing greater context and instructional notes round resources made available. Much of the perceived need for communication stemmed from assessment materials and deadlines that emphasised the need for academic colleagues to focus more attention on to the provision of meaningful communication in relation to assessment.

In a similar vein, feedback pertaining to Moodle use, emphasised the need for academic colleagues to provide greater contextualisation and instructional direction to resources in the VLE to drive the usage and utility value of them.

Students cited further, and more targeted use of audio–visual resources as a desirable outcome. Often the use of these resources were sporadic and seemingly unrelated to content either in the sessions or in the VLE, echoing the earlier concern around setting the context and providing instructional guidance.

Yet these interpretations derived from the TELAs were often not expressed by academic colleagues. Indeed, it became apparent through anecdotal evidence that the interpretations of these student results by academic colleagues deviated dramatically from the TELAs. Why and how this was the case became the next area of focus. The data derived from the student data analysis was utilised as foci for questions to be put to academic colleagues in investigating this disconnect.

Communications

Some examples as shown in Table 6 demonstrate the varying opinions around communications.

Comparing these insights underlines the incongruence between student expectation and staff perception. Staff appear to recognise the expectation for a ‘quick accurate response’ but that ‘responding at different times…creates an expectation that colleagues will do this as well’. In contrast, students indicate some staff refuse to engage at all in email communication and some staff seemingly engage as a useful way of ‘covering yourself’. These typologies of response expand the sense of disconnection ostensibly strengthening the view expressed earlier that TELAs are perhaps critical filters in determining meaningful interpretations of the student data.

Audio–visual

The use of audio–visual resources demonstrated a disparity of views. Students emphasised the positive use that enhanced learning by allowing them to ‘gain a full understanding of topics’ but also criticised practices that were poor when they were ‘rather pointless videos’. In contrast, staff perception of audio–visual resources suggested a fear of being replaced (see Table 7). Also further recognition that colleagues using these resources would then create an expectation among students and potentially create a mismatch. Therefore, a programme approach should be taken either all in or all out (see Table 7). Also prevalent were performance fears whereby technical difficulty or unfamiliarity could change ‘the dynamic, you have to regain your dignity unfortunately’ (see Table 7).

Interactive lectures

Performance issues permeated the staff psyche in relation to interactive lectures. Here staff perceived a sense of a ‘West end show’ and difficulty for staff to ‘live up to that expectation’ (see Table 8).

Further staff concerns centred around control whereby enabling students to use technology led to a sense of ‘I've lost them and I need to try and get them, back in’ (see Table 8). Conversely students viewed this approach as enabling for them with the ability to ‘engage more verbally in class through the texting app’ (see Table 8). The absence of interaction was seen as an impediment to learning as students found it ‘hard to concentrate when the lecturer is simply reading’ (see Table 8).

Moodle use

The Moodle usage comments from staff appeared to emphasise the quantity of material and the ‘temptation to put loads on there’ (see Table 9). The predominant feeling from students was that material was ‘hard to understand when it was out of context’.