Using Learning Design to Unleash the Power of Learning Analytics

December 4, 2015

12274589_10153905502716282_5140343966725134468_n.jpg

Atkinson, S.P. (2015). Using Learning Design to Unleash the Power of Learning Analytics. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. 358-364 / CP:6-CP:10).


 

A very enjoyable presentation made this week at ascilite 2015 in Perth, Australia. Wonderful to engage with this vibrant and hospitable community. Amongst some fascinating presentations exploring the theoretical and information management dimension of learning analytics and academic analytics, my very foundational work on constructively aligned curricula and transparency in design was I believe welcomed.

I said in my paper that I believed “New learning technologies require designers and faculty to take a fresh approach to the design of the learner experience. Adaptive learning, and responsive and predicative learning systems, are emerging with advances in learning analytics. This process of collecting, measuring, analysing and reporting data has the intention of optimising the student learning experience itself and/or the environment in which the experience of learning occurs… it is suggested here that no matter how sophisticated the learning analytics platforms, algorithms and user interfaces may become, it is the fundamentals of the learning design, exercised by individual learning designers and faculty, that will ensure that technology solutions will deliver significant and sustainable benefits. This paper argues that effective learning analytics is contingent on well structured and effectively mapped learning designs.

12316100_10153905502676282_6800926538129710150_n


Graduate Competencies, Employability and Educational Taxonomies: Critique of Intended Learning Outcomes

July 18, 2015

We hear much about the changing world of work and how slow higher and professional education is to respond. So in an increasingly competitive global market of Higher Apprenticeships and work-based learning provision I began to take a particular interest in students’ ‘graduateness’. What had begun as an exploratory look for examples of intended learning outcomes (ILO) with ’employability’ in mind ended up as this critical review published in an article entitled ‘Graduate Competencies, Employability and Educational Taxonomies: Critique of Intended Learning Outcomes’ in the journal called  Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education.

I randomly identified 20 UK institutions, 80 undergraduate modules and examined their ILOs. This resulted in 435 individual ILOs being taken by students in current modules (academic year 2014-2015) across different stages of their undergraduate journey (ordinarily in the UK this takes place over three years through Levels 4,5 and 6). This research reveals the lack of specificity of ILOs in terms of skills, literacies and graduates attributes that employers consistently say they want from graduates

The data in the table below from the full paper which describes the post-analysis attribution of ILOs to domains of educational objectives (see paper for methodology) which I found rather surprising. The first surprise was the significant percentage of ILOs which are poorly structured, given the weight of existing practice guidance and encouragement for learning designers and validators (notably from the UK Higher Education Academy and the UK Quality Assurance Agency). Some 94 individual ILOs (21.6%) had no discernible active verbs in their construction.  64 ILOs (14.7%) did not contain any meaningful verbs so could not be mapped to any educational domain. This included the infamous ‘to understand’ and ‘to be aware of’. So as a result only 276 ILOs (64%) were deemed ‘well-structured’ and were then mapped against four domains of educational objectives.

Table 8.          Post-analysis attribution of ILOs to Domains of Educational Objectives

  Level 4 Level 5 Level 6 Total
Knowledge(Subject Knowledge) 14 5 11 30
Cognitive (Intellectual Skills) 46 91 61 198
Affective(Professional Skills) 1 4 1 6
Psychomotor(Practical/Transferable Skills) 12 18 13 43
No Verbs 35 32 27 94
Not classifiable 23 30 11 64
Totals 131 180 125 435

Remember what I had been originally looking for were examples of ILOs that represented skills that the literature on employability and capabilities suggested should be there. These could have been anticipated to be those in the affective or psychomotor domains.

So it was rather surprising to see that of the 64% of the full sample that was codeable,  sizeable percentage were cognitive (45.4%), a relatively small percentage fell into the psychomotor domain (9.8%), even less into the knowledge domain (6.8%) and a remarkably small number could be deemed affective (1.4%).

I say remarkable because the affective domain, sometimes detailed as personal and professional skills, are very much the skills that employers (and most graduates) prize above all else. These refer to the development of values and the perception of values, including professionalism, inter-personal awareness, timeliness, ethics, inter-cultural sensitivity, and diversity and inclusivity issues.

Apparently despite all the sterling work going on in our libraries and career services, employment-ready priorities within programmes and modules in higher education, are not integrated with teaching and learning practices. I suggest that as a consequence, this makes it difficult for students to extract, from their learning experience within modules, the tangible skill development required of them as future employees.

There is an evident reliance by module designers on the cognitive domain most commonly associated at a lower level with ‘knowing and understanding’ and at a higher level as ‘thinking and intellectual skills’. The old favourite ‘to critically evaluate’ and ‘to critically analyse’ are perennial favourites.

There is much more to the picture than this single study attempts to represent but I think it is remarkable not more attention is being paid to the affective and psychomotor domains in module creation.

More analysis, and further data collection will be done, to explore the issue at programme level and stage outcomes (is it plausible that module ILOs are simply not mapped and unrelated and all is well at programme level). I would also be interested to explore the mapping of module and programme ILOs to specified graduate attributes that many institutions make public.

I go on in the full paper about the relative balance of different ILOs in each of the domains depending on the nature of the learning, whether it is a clinical laboratory module or a fieldwork module or a literature-based module.

The reason I think this is important, and I have written here before, that this important (it is about semantics!), is that students are increasingly demanding control over their choices, their options, the shape of their portfolios, their ‘graduateness’, and they need to be able to identify their own strengths and weaknesses and make meaningful modules choices to modify the balance of the skills acquired in a ’practical’ module compared with those in a ‘cerebral’ one. I conclude that the ability to consciously build a ‘skills profile’ is a useful graduate attribute in itself…. which incidentally would be an affective ILO.

You can download the full paper here LINK.

Also available on ResearchGate and Academia.edu

Full citation:
Atkinson, S. 2015 Jul 9. Graduate Competencies, Employability and Educational Taxonomies: Critique of Intended Learning Outcomes. Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education [Online] 10:2. Available: http://community.dur.ac.uk/pestlhe.learning/index.php/pestlhe/article/view/194/281


Enhancements to the SOLE Tookit – now version 3.5

May 31, 2015

I have no idea what the protocol is for naming versions of things. I imagine, like me, someone has an idea of what the stages are going to look like, when a truly fresh new is going to happen. For me I have a sense that version 4.0 of the SOLE Toolkit will incorporate what I am currently learning about assessment and ‘badges’, self-certification and team marking. But for now I’m not there yet and am building on what I have learnt about student digital literacies so I will settle for Version 3.5.

This version of the SOLE Toolkit, 3.5, remains a completely free, unprotected and macro-free Excel workbook with rich functionality to serve the learning designer. In version 3.0 I added more opportunities for the student to use the toolkit as an advanced organiser offering ways to record their engagement with their learning. It also added in some ability to sequence learning so that students could plan better their learning although I maintained this was guidance only and should allow students to determine their own pathways for learning.

Version 3.5 has two significant enhancements. Firstly, it introduces a new dimension, providing a rich visualization of the learning spaces and tools that students are to engage with in their learning. This provides an alternative, fine-grain, view of the students modes of engagement in their learning. It permits the designer to plan not only for a balance of learning engagement but also a balance of environments and tools. This should allow designers to identify where ‘tool-boredom’ or ‘tool-weariness’ is possibly a danger to learner motivation and to ensure that a range of tools and environments allow students to develop based on their own learning preferences.

Secondly, it allows for a greater degree of estimation of staff workload, part of the original purpose of the SOLE Model and Toolkit project back in 2009. This faculty-time calculations in design and facilitating are based on the learning spaces and tools to be used. This function allows programme designers and administrators, as well as designers themselves, to calculate the amount of time they are likely to need to design materials and facilitate learning around those materials.

I invite you to explore the SOLE Toolkit on the dedicated website for the project and would welcome any comments of feedback you might have.


Why ‘learning analytics’ is like a sewer

May 2, 2015

Back in the late northern hemisphere summer of 2013 I drafted a background paper on the differences between Educational Data Mining, Academic Analytics and Learning Analytics. Entitled ‘Adaptive Learning and Learning Analytics: a new design paradigm‘, It was intended to ‘get everyone on the same page‘ as many people at my University, from very different roles, responsibilities and perspectives, had something to say about ‘analytics’. Unfortunately for me I then had nearly a years absence through ill-health and I came back to an equally obfuscated landscape of debate and deliberation. So I opted to finish the paper.

I don’t claim to be an expert on learning analytics, but I do know something about learning design, about teaching on-line and about adapting learning delivery and contexts to suit different individual needs. The paper outlines some of the social implications of big data collection. It looks to find useful definitions for the various fields of enquiry concerned with collecting and making something useful with learner data to enrich the learning process. It then suggest some of the challenges that such data collection involves (decontextualisation and privacy) and the opportunity it represents (self-directed learning and the SOLE Model). Finally it explores the impact of learning analytics on learning design and suggests why we need to re-examine the granularity of our learning designs.

I conclude;

Learning Analytics Cover“The influences on the learner that lay beyond the control of the learning provider, employer or indeed the individual themselves, are extremely diverse. Behaviours in social media may not be reflected in work contexts, and patterns of learning in one discipline or field of experience may not be effective in another. The only possible solution to the fragmentation and intricacy of our identities is to have more, and more interconnected, data and that poses a significant problem.

Privacy issues are likely to provide a natural break on the innovation of learning analytics. Individuals may not feel that there is sufficient value to them personally to reveal significant information about themselves to data collectors outside the immediate learning experience and that information may simply be inadequate to make effective adaptive decisions. Indeed, the value of the personal data associated with the learning analytics platforms emerging may soon see a two tier pricing arrangement whereby a student pays a lower fee if they engage fully in the data gathering process, providing the learning provider with social and personal data, as well as their learning activity, and higher fees for those that wish to opt-out of the ‘data immersion’.

However sophisticated the learning analytics platforms, algorithms and user interfaces become in the next few years, it is the fundamentals of the learning design process which will ensure that learning providers do not need to ‘re-tool’ every 12 months as technology advances and that the optimum benefit for the learner is achieved. Much of the current commercial effort, informed by ‘big data’ and ‘every-click-counts’ models of Internet application development, is largely devoid of any educational understanding. There are rich veins of academic traditional and practice in anthropology, sociology and psychology, in particular, that can usefully inform enquiries into discourse analysis, social network analysis, motivation, empathy and sentiment study, predictive modelling and visualisation and engagement and adaptive uses of semantic content (Siemens, 2012). It is the scholarship and research informed learning design itself, grounded in meaningful pedagogical and andragogical theories of learning that will ensure that technology solutions deliver significant and sustainable benefits.

To consciously misparaphrase American satirist Tom Lehrer, learning analytics and adaptive learning platforms are “like sewers, you only get out of them, what you put into them’.”

Download the paper here, at AcademiaEdu or ResearchGate

Siemens, G. (2012). Learning analytics: envisioning a research discipline and a domain of practice. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 4–8). New York, NY, USA: ACM. doi:10.1145/2330601.2330605


ePortfolios: for whom and wherefore 

April 12, 2015

(unformed thoughts...) I’m exploring a student e-portfolio solution. I probably didn’t need to install an instance of Moodle (2.5.9) and one of Mahara (1.10.2) to know it’s a match made in heaven. It’s been fun patching the two systems together (also linked up Moodle to Google Drive to test the outputs there too – plain and simple) but then I began to wonder. ..Who’s it for? Whose version of heaven does it represent?

Mahara Test Environment

Mahara Test Environment

The very notion of an ‘e-portfolio‘ might produce obstacles rather than opportunities. The word ‘portfolio‘ means various things in different context to a variety of audiences. A portfolio might refer to a physical (or virtual) briefcase, it might refer to a financial portfolio as a collection of stocks, shares and asset notifications; or it could also mean an artist’s portfolio as a collection of works in progress or final outputs; or again it can still have other meanings, to a portfolio as a body of responsibilities or projects currently held by someone in the workplace. Why then does the vast majority scholarly literature on e-portfolios in a university context represents a uniform interpretation – an assemblage of reflections and representational artefacts?

I am afraid I’m going to bang an old drum, that of the need for learning to consider the foundational aspects of epistemological beliefs. There is an ongoing debate as to the extent to which the portfolio is owned by the students (the majority view) or the institution. Where a portfolio is independent of any formal assessment processes it is fairly easy to define the portfolio in terms that students take full control of its structure and output (within whatever restrictions the technology imposes). However, if there is any relationship between the representational space and formal assessment processes the ownership (of process if not products) is at east shared. This changes the way we advocate the use of a portfolio. If student and tutor have a shared perception of learning as a personal reflective journey that the tutor can encourage but remove themselves from the process of meta-cognitive growth we hope students might experience.

There is also an important cultural dimension to our expectation that students will want to record and reflect in a portfolio context. I don’t mean ‘cultural’ with respect to international students, as important as that is, but I confess I am ignorant as to the existence of a diurnal recording tradition outside the occidental world. If there is no historical context for writing ones daily occurrences, experiences and reflections, it is a ‘big ask’ to expect students to engage in such a process from ‘scratch’. There is a global tradition of thoughts and observations and of travel writings and so perhaps a more suitable metaphor for the majority of learners to grasp onto would be the learning journey (re: journal).

Surely we want our graduates to have ‘basic’ digital literacy skills but isn’t a template driven portfolio solution really short-changing them? Shouldn’t they leave University with the ability to set-up a digital presence on the web for themselves, to select a service that suits their particular context be it LinkedIn for the ‘career driven’ or Academia.edu for the apprenticed faculty. Would not a WordPress solution suit most for a public facing self-representation. Shouldn’t institutions be getting out of the way of learners and rather than seeking to curate learning outputs,  instead enabling learners to take digital-flight.


Is Higher Education lacking its affective dimension?

March 31, 2015
Affective Domain March 2015

Affective Domain March 2015

Whilst the majority of writings and reflection concerning the use of taxonomies of educational objectives remain focused around the cognitive domain, typified by Bloom (1984), there is a growing attention being paid to the affective domain, particularly in professional education. Bloom’s now famous research project which resulted in Book 2 of the ‘Taxonomy of Educational Objectives: the affective domain’ led by Krathwohl, which has been much neglected, applied scantily (and often erroneously) to practice (Krathwohl, Bloom, & Masia, 1999).

Often described in terms of feelings or emotions I think it more useful to regard the affective as being a question of degree of acceptance or rejection of values, of the evolution of value structures. It is certainly true that having one’s values challenged can result in emotional disturbance (Mezirow, 2000) but I think there is a danger to using language that describes the domain in terms of emotions, not least because it scares off academics!

In my interpretation of the affective domain, illustrated above as a circular representation, I have retained Krathwohl’s five divisions but choose to use active verbs rather than the passive. So receiving becomes to receive, responding becomes to respond, valuing becomes to value, organization becomes to organize, and the final division originally entitled characterization becomes to internalize.

Original Atkinson Descriptor Sample Verbs
Receiving To Receive Ability to learn from others Sense, experience, concentrate, attend, perceive
Responding To Respond Ability to participate responsibly , respectively as appropriate to the context Perform, contribute, satisfy, cite
Valuing To Value Ability to associate personal and collective values with contextual experiences and express value judgements Justify, seek, respect, persuade
Organization To Organize Ability to structure, prioritize and reconcile personal and others’ value systems Clarify, reconcile, integrate
Characterization To Internalize Ability to articulate one’s own values and belief systems and operate consistently within them Conclude, internalize, resolve, embody

The reason I think it helpful to think about values rather than emotions is that clearly most of higher and professional education, is concerned by changing not just how much or what students know, or even how they know and apply that knowledge, but with how they ‘feel’ about knowing. At its simplest we seek to instil a love of learning and a passion for the subject. In professional education we also seek to instil our professional values into the learning process. Whilst it is clearly very difficult to evaluate the emotional impact that learning has on students there are ways of providing formative assessment to support affective developments.

For those in the Academy who are seeking to merely perpetuate their academic DNA in their students, the latters’ changes in values may not mean a great deal to them. For those of us who teach in order to make the next generation better than we are, better able to adapt to the ever-changing world that they face outside of the Academy, then having an interest in our students affective development is extremely important.

I have argued elsewhere that the relative weighting given in learning design to the domains depends largely on the subject and the context of learning. Clearly there will be foundational modules in a degree programme in which knowledge domain learning will be dominant. I would expect much that is done in an undergraduate degree to be concerned with the cognitive domain, clearly an ability to analyse, evaluate, synopsise and synthesise represent the higher-order skills we expect from graduates on graduation. I have also argued elsewhere that psychomotor skills are also worthy of being part of higher education. But it occurs to me that much of the learning opportunity offered in our current universities neglect and equally important set of skills.

Almost all employers agree that they want to attract applicants who share their values. These oft cited idealized values are in fact widely held; the ability to work well with others, to be an effective communicator, to be an effective listener, to work independently, to take the initiative. It seems unrealistic to expect students to necessarily acquire such skills without being guided through the learning process and taught to identify their own development. And it is fair to say that certainly in the United Kingdom sector a huge amount has been added to the curriculum, with employability strategies and planning personal development (PDP) initiatives, that students do not wont for opportunities.

But I maintain that we should ingrain in our students the values we expect them to demonstrate through the disciplines themselves, not bolt them on and relieve the academics from their repsonsability. To my mind it makes sense to write intended learning outcomes to encapsulate a range of affective outcomes and align learning and teaching activities to rehearse those skills with our students.

Why not include alongside an intellectual skills outcome (cognitive) that states “by the end of this module you will be able to critique at least three different perspectives on (whatever the subject is)” another outcome, this time an affective one that says “by the end of this module you will be able to reconcile two contrasting, and contentious, perspectives on (whatever the subject is)”. There is nothing touchy-feely about the second outcome but it focuses on the students value structures, supporting their ability to structure and reconcile personal value systems in contrast to those held by others. Critiquing sounds very higher education, but to be able ‘to reconcile’ is a much needed skill in the workplace.

Bloom, B. S. (1984). Taxonomy of Educational Objectives Book 1: Cognitive Domain (2nd edition). Addison Wesley Publishing Company.
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1999). Taxonomy of Educational Objectives Book 2/Affective Domain (2nd edition). Longman Pub Group.
Mezirow, J. (2000). Learning as transformation : critical perspectives on a theory in progress. San Francisco: Jossey-Bass.

Paper: Rethinking personal tutoring systems, the need to build on a foundation of epistemological beliefs.

January 21, 2015

Newly uploaded, here is the final paper that was previewed in blog postings during December 2014.

Atkinson, S. P. (2014) Rethinking personal tutoring systems: the need to build on a foundation of epistemological beliefs. BPP University Working Papers. London: BPP University.

Image of the cover of Rethinking Personal Tutoring Systems

Rethinking Personal Tutoring Systems

My argument is that in order to tailor effective support for students we must understand better their fundamental beliefs about learning; that to have a conversation about ‘our’ values we need to understand how others experience their own.

This was the purpose of the POISE project, an HEA Change Initiative and this paper is a summary of its conclusions.

There is much work to be done to make these insights more accessible to rank and file tutors in higher education but the POISE website is a start. As always I am delighted to hear about any use made of the work and to enter into a dialogue with anyone working on similar initiatives.


Follow

Get every new post delivered to your Inbox.

Join 273 other followers

%d bloggers like this: