Simplifying Assessment Alignment

November 22, 2016

Some recent work with programme designers in other UK institutions suggests to me that quality assurance and enhancement measures continue to be appended to the policies and practices carried out in UK HEIs rather than seeing a revitalising redesign of the entire design and approval process.

This is a shame because it has produced a great deal of work for faculty in designing and administering programmes and modules, not least when it comes to assessment. Whatever you feel about intended learning outcomes (ILOs) and their constraints or structural purpose, there is nearly universal agreement that the purpose of assessment is not to assess students ‘knowledge of the content’ on a module. Rather the intention of assessment is to demonstrate higher learning skills, most commonly codified in the intended learning outcomes. I have written elsewhere about the paucity of writing effective ILOs and focusing them almost entirely the cognitive domain (intellectual skills), with the omission of other skill domains notably the effective (professional skills) and the psychomotor (transferable skills). Here I want to identify the need for close proximity between ILOs and assessment criteria.

It seems to me that well-designed intended learning outcomes lead to cogent assessment design. They also suggest that the use of a transparent marking rubric, used by both markers and students, creates a simpler process.

To illustrate this I wanted to share two alternative approaches to aligning assessment to the outcomes of a specific module. In order to preserve the confidentiality of the module in question some elements have been omitted but hopefully the point will still be clearly made.

Complex Attempt to Assessment Alignment

Complex Assessment AlignmentI have experienced this process in several Universities.

  1. Intended Learning Outcomes are written (normally at the end of the ‘design’ process)
  2. ILOs are mapped to different categorizations of domains, Knowledge & Understanding, Intellectual Skills, Professional Skills and Attitudes, Transferable Skills.
  3. ILOs are mapped against assessments, sometimes even mapped to subject topics or weeks.
  4. Students get first sight of the assessment.
  5. Assessment Criteria are written for students using different categories of judgement: Organisation, Implementation, Analysis, Application, Structure, Referencing, etc.
  6. Assessment Marking Schemes are then written for assessors. Often with guidance as to what might be expected at specific threshold stages in the marking scheme.
  7. General Grading Criteria are then developed to map the schemes outcomes back to the ILOs.

 

Streamlined version of aligned assessment

streamlined marking rubric

I realise that this proposed structure is not suitable for all contexts, all educational levels and all disciplines. Nonetheless I would advocate that this is the optimal approach.

  1. ILO are written using a clear delineation of domains; Knowledge, Cognitive (Intellectual), Affective (Values) and Psychomotor (Skills). These use appropriate verb structures tied directly to appropriate levels. This process is explained in this earlier post.
  2. A comprehensive marking rubric is then shared with both students and assessors. It identifies all of the ILOs that are being assessed. In principle we should only be assessing the ILOs in UK Higher Education NOT content. The rubric will differentiate the type of responses expected to achieve varies grading level.
    • There is an option to automatically sum grades given against specific outcomes or to take a more holistic view.
    • It is possible to weight specific ILOs as being worth more marks than others.
    • This approach works for portfolio assessment but also for a model of assessment where there are perhaps two or three separate pieces of assessment assuming each piece is linked to two or three ILOs.
    • Feedback is given against each ILO on the same rubric (I use Excel workbooks)

I would suggest that it makes sense to use this streamlined process even if it means rewriting your existing ILOs. I’d be happy to engage in debate with anyone about how best to use the streamlined process in their context.


Graduate Competencies, Employability and Educational Taxonomies: Critique of Intended Learning Outcomes

July 18, 2015

We hear much about the changing world of work and how slow higher and professional education is to respond. So in an increasingly competitive global market of Higher Apprenticeships and work-based learning provision I began to take a particular interest in students’ ‘graduateness’. What had begun as an exploratory look for examples of intended learning outcomes (ILO) with ’employability’ in mind ended up as this critical review published in an article entitled ‘Graduate Competencies, Employability and Educational Taxonomies: Critique of Intended Learning Outcomes’ in the journal called  Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education.

I randomly identified 20 UK institutions, 80 undergraduate modules and examined their ILOs. This resulted in 435 individual ILOs being taken by students in current modules (academic year 2014-2015) across different stages of their undergraduate journey (ordinarily in the UK this takes place over three years through Levels 4,5 and 6). This research reveals the lack of specificity of ILOs in terms of skills, literacies and graduates attributes that employers consistently say they want from graduates

The data in the table below from the full paper which describes the post-analysis attribution of ILOs to domains of educational objectives (see paper for methodology) which I found rather surprising. The first surprise was the significant percentage of ILOs which are poorly structured, given the weight of existing practice guidance and encouragement for learning designers and validators (notably from the UK Higher Education Academy and the UK Quality Assurance Agency). Some 94 individual ILOs (21.6%) had no discernible active verbs in their construction.  64 ILOs (14.7%) did not contain any meaningful verbs so could not be mapped to any educational domain. This included the infamous ‘to understand’ and ‘to be aware of’. So as a result only 276 ILOs (64%) were deemed ‘well-structured’ and were then mapped against four domains of educational objectives.

Table 8.          Post-analysis attribution of ILOs to Domains of Educational Objectives

  Level 4 Level 5 Level 6 Total
Knowledge(Subject Knowledge) 14 5 11 30
Cognitive (Intellectual Skills) 46 91 61 198
Affective(Professional Skills) 1 4 1 6
Psychomotor(Practical/Transferable Skills) 12 18 13 43
No Verbs 35 32 27 94
Not classifiable 23 30 11 64
Totals 131 180 125 435

Remember what I had been originally looking for were examples of ILOs that represented skills that the literature on employability and capabilities suggested should be there. These could have been anticipated to be those in the affective or psychomotor domains.

So it was rather surprising to see that of the 64% of the full sample that was codeable,  sizeable percentage were cognitive (45.4%), a relatively small percentage fell into the psychomotor domain (9.8%), even less into the knowledge domain (6.8%) and a remarkably small number could be deemed affective (1.4%).

I say remarkable because the affective domain, sometimes detailed as personal and professional skills, are very much the skills that employers (and most graduates) prize above all else. These refer to the development of values and the perception of values, including professionalism, inter-personal awareness, timeliness, ethics, inter-cultural sensitivity, and diversity and inclusivity issues.

Apparently despite all the sterling work going on in our libraries and career services, employment-ready priorities within programmes and modules in higher education, are not integrated with teaching and learning practices. I suggest that as a consequence, this makes it difficult for students to extract, from their learning experience within modules, the tangible skill development required of them as future employees.

There is an evident reliance by module designers on the cognitive domain most commonly associated at a lower level with ‘knowing and understanding’ and at a higher level as ‘thinking and intellectual skills’. The old favourite ‘to critically evaluate’ and ‘to critically analyse’ are perennial favourites.

There is much more to the picture than this single study attempts to represent but I think it is remarkable not more attention is being paid to the affective and psychomotor domains in module creation.

More analysis, and further data collection will be done, to explore the issue at programme level and stage outcomes (is it plausible that module ILOs are simply not mapped and unrelated and all is well at programme level). I would also be interested to explore the mapping of module and programme ILOs to specified graduate attributes that many institutions make public.

I go on in the full paper about the relative balance of different ILOs in each of the domains depending on the nature of the learning, whether it is a clinical laboratory module or a fieldwork module or a literature-based module.

The reason I think this is important, and I have written here before, that this important (it is about semantics!), is that students are increasingly demanding control over their choices, their options, the shape of their portfolios, their ‘graduateness’, and they need to be able to identify their own strengths and weaknesses and make meaningful modules choices to modify the balance of the skills acquired in a ’practical’ module compared with those in a ‘cerebral’ one. I conclude that the ability to consciously build a ‘skills profile’ is a useful graduate attribute in itself…. which incidentally would be an affective ILO.

You can download the full paper here LINK.

Also available on ResearchGate and Academia.edu

Full citation:
Atkinson, S. 2015 Jul 9. Graduate Competencies, Employability and Educational Taxonomies: Critique of Intended Learning Outcomes. Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education [Online] 10:2. Available: http://community.dur.ac.uk/pestlhe.learning/index.php/pestlhe/article/view/194/281


Is Higher Education lacking its affective dimension?

March 31, 2015
Affective Domain March 2015

Affective Domain March 2015

Whilst the majority of writings and reflection concerning the use of taxonomies of educational objectives remain focused around the cognitive domain, typified by Bloom (1984), there is a growing attention being paid to the affective domain, particularly in professional education. Bloom’s now famous research project which resulted in Book 2 of the ‘Taxonomy of Educational Objectives: the affective domain’ led by Krathwohl, which has been much neglected, applied scantily (and often erroneously) to practice (Krathwohl, Bloom, & Masia, 1999).

Often described in terms of feelings or emotions I think it more useful to regard the affective as being a question of degree of acceptance or rejection of values, of the evolution of value structures. It is certainly true that having one’s values challenged can result in emotional disturbance (Mezirow, 2000) but I think there is a danger to using language that describes the domain in terms of emotions, not least because it scares off academics!

In my interpretation of the affective domain, illustrated above as a circular representation, I have retained Krathwohl’s five divisions but choose to use active verbs rather than the passive. So receiving becomes to receive, responding becomes to respond, valuing becomes to value, organization becomes to organize, and the final division originally entitled characterization becomes to internalize.

Original Atkinson Descriptor Sample Verbs
Receiving To Receive Ability to learn from others Sense, experience, concentrate, attend, perceive
Responding To Respond Ability to participate responsibly , respectively as appropriate to the context Perform, contribute, satisfy, cite
Valuing To Value Ability to associate personal and collective values with contextual experiences and express value judgements Justify, seek, respect, persuade
Organization To Organize Ability to structure, prioritize and reconcile personal and others’ value systems Clarify, reconcile, integrate
Characterization To Internalize Ability to articulate one’s own values and belief systems and operate consistently within them Conclude, internalize, resolve, embody

The reason I think it helpful to think about values rather than emotions is that clearly most of higher and professional education, is concerned by changing not just how much or what students know, or even how they know and apply that knowledge, but with how they ‘feel’ about knowing. At its simplest we seek to instil a love of learning and a passion for the subject. In professional education we also seek to instil our professional values into the learning process. Whilst it is clearly very difficult to evaluate the emotional impact that learning has on students there are ways of providing formative assessment to support affective developments.

For those in the Academy who are seeking to merely perpetuate their academic DNA in their students, the latters’ changes in values may not mean a great deal to them. For those of us who teach in order to make the next generation better than we are, better able to adapt to the ever-changing world that they face outside of the Academy, then having an interest in our students affective development is extremely important.

I have argued elsewhere that the relative weighting given in learning design to the domains depends largely on the subject and the context of learning. Clearly there will be foundational modules in a degree programme in which knowledge domain learning will be dominant. I would expect much that is done in an undergraduate degree to be concerned with the cognitive domain, clearly an ability to analyse, evaluate, synopsise and synthesise represent the higher-order skills we expect from graduates on graduation. I have also argued elsewhere that psychomotor skills are also worthy of being part of higher education. But it occurs to me that much of the learning opportunity offered in our current universities neglect and equally important set of skills.

Almost all employers agree that they want to attract applicants who share their values. These oft cited idealized values are in fact widely held; the ability to work well with others, to be an effective communicator, to be an effective listener, to work independently, to take the initiative. It seems unrealistic to expect students to necessarily acquire such skills without being guided through the learning process and taught to identify their own development. And it is fair to say that certainly in the United Kingdom sector a huge amount has been added to the curriculum, with employability strategies and planning personal development (PDP) initiatives, that students do not wont for opportunities.

But I maintain that we should ingrain in our students the values we expect them to demonstrate through the disciplines themselves, not bolt them on and relieve the academics from their repsonsability. To my mind it makes sense to write intended learning outcomes to encapsulate a range of affective outcomes and align learning and teaching activities to rehearse those skills with our students.

Why not include alongside an intellectual skills outcome (cognitive) that states “by the end of this module you will be able to critique at least three different perspectives on (whatever the subject is)” another outcome, this time an affective one that says “by the end of this module you will be able to reconcile two contrasting, and contentious, perspectives on (whatever the subject is)”. There is nothing touchy-feely about the second outcome but it focuses on the students value structures, supporting their ability to structure and reconcile personal value systems in contrast to those held by others. Critiquing sounds very higher education, but to be able ‘to reconcile’ is a much needed skill in the workplace.

Bloom, B. S. (1984). Taxonomy of Educational Objectives Book 1: Cognitive Domain (2nd edition). Addison Wesley Publishing Company.
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1999). Taxonomy of Educational Objectives Book 2/Affective Domain (2nd edition). Longman Pub Group.
Mezirow, J. (2000). Learning as transformation : critical perspectives on a theory in progress. San Francisco: Jossey-Bass.

Adaptation of Dave’s Psychomotor Domain

December 7, 2014

I have received some interesting feedback and critique of my circular representation of Ravindra H. Dave’s psychomotor domain of educational objectives. I have been asked why I have chosen to use the circular design, to alternative verbs and to expand the definition of psychomotor activity.

Firstly the representation of the domain as a circle, which I have done across four domains elsewhere, I believe serves to make the subcategories more fluid. It contains the proto-verbs at the centre, next circle contains active verbs which also represent teaching and learning activity and the outer circle contains the nature of evidence (or assessment forms) that might demonstrate the active verbs. Using the circle one also has an inherently clock-face like visual which makes the dialling-up from the basic to more sophisticated concepts as you travel around clockwise. Maybe its most powerful function is to encourage lateral thinking on the part of learning designers, encouraging them to explore learning and teaching activities as assessment or evidence examples at the same time.

Psychomotor Domain - Taxonomy Circle - after Dave (1969/71)

Psychomotor Domain – Taxonomy Circle – after Dave (1969/71)

Secondly, I have chosen to use active verbs to describe the subcategories of the domain and so there is a clear change from:

Dave’s Original  Atkinson’s Adaptation  Descriptor
Imitation Imitate ability to copy, replicate the actions of others following observations.
 Manipulation  Manipulate ability to repeat or reproduce actions to prescribed standard from memory or instructions.
Precision  Perfect ability to perform actions with expertise and without interventions and the ability to demonstrate and explain actions to others.
Articulation  Articulate ability to adapt existing psychomotor skills in a non-standard way, in different contexts, using alternative tools and instruments to satisfy need.
 Naturalisation  Embody ability to perform actions in an automatic, intuitive or unconscious way appropriate to the context.

This is to articulate more clearly the need to describe learning outcomes as things that the students will actually ‘do’ in line with the principles of constructively aligned learning and teaching design.

The third, more less obvious change, is that I have chosen to expand the definition of psychomotor activity to incorporate a wider range of physical activities that perhaps Dave had not envisaged, particularly those involved with the manipulation of computer software, laboratory and fieldwork equipment and a range of technical equipment. I felt this was necessary because I have seen so many University courses make light of the skills developed in acquiring such expertise, as though such skills are incidental, when clearly it is the primary outcome valued by most students and employers.

For example, the specifics of the volume of water flowing through the Mississippi delta in November (Knowledge) will prove less useful than the ability to master the GIS and computational software used to document those specifics (Psychomotor).

I believe that the majority of what in the UK further and higher context is described as ‘transferable skills’ fall into the psychomotor domain and are worthy of careful attention.


An Opinion on a new approach on Personal Tutorial Systems

November 5, 2014

(Extracted for a draft Working paper)

In the previous four posts I have outlined the changing nature of student support demands, the existing basis for current support, the need to embrace epistemological orientation as a necessary function of higher education institutions and suggested one project that attempts to resources such an effort. I should also state (for the record as it were) that I have had the utmost respect for a great many colleagues who are fantastic academic and personal tutors. It was also my privilege to oversee the work of the Study Support Services when I was Head of the Centre for Learning Development at the University of Hull and have been an (occasional) study advisor (at the LSE) and to postgraduate students at BPP University. However I don’t believe in Sacred Cows (unless they are there to be at least considered worthy of butchering).

Given the increasingly diverse nature of the student population, ever distant from the selected and ‘ever ready ‘ school graduates, it is unsurprising that institutions now provide a myriad of support services. In some institutions there is clarity between study support and student welfare services and in others these can be found bundled together in ‘student services’. In some institutions these services are aligned alongside library provision and in others provided wholly or jointly with student union organisations. There is a need for institutions to recognise the evolutionary nature of this provision and to question whether there is not in need to wipe the slate clean. There is an obligation on institutions to provide students what they need but also what they want and the onus is then on us to ask students new questions.

Students are clearly a diverse group of individuals and we must provide individually tailored solutions if we wish to maintain the diversity within our learning communities. Clearly it is impractical, not least from a financial perspective, to design solutions targeted with each individual but it is surely possible to provide mechanisms that enable students to pull down services as and when they require them in a more meaningful way. Whilst there are certainly faculty members who are heroic in their endeavours to service both the academic needs of their students and provide welfare and personal guidance this is an unrealistic expectation. While in some disciplines, notably in health, such personal tutorial support focused on the affective development of students, clearly is a requirement imposed on faculty, this is not universally true. We should re-evaluate the need for personal tutorial support and where appropriate specify academic guidance and mentoring from welfare duties. I believe with correct epistemological orientation and the embedding of skills within courses the academic role should be largely limited to academic ‘mentoring’ in reviewing choices, strengths and weaknesses, and academic progression.

The degree to which support for the affective development of students is provided will then depend largely on the nature of the learning community itself. The ability of faculty and students to create a supportive community able to furnish appropriate aid on demand will depend on the extent to which students operate within a closed system. In a non-residential commuter institution, and in distance provision, the enormity of diverse needs simply cannot be sustained within our existing tutorial systems. Instead we should provide students with the maturity to understand their own needs and facilitate their access to appropriate support most probably sourced from outside the institution. Partnerships, both formal and informal, with welfare services (counselling services, financial advisers, spiritual services, housing support and other) can be paid for by the institution itself based on use or delegated to the individual student. A more formal arrangement of the division of academic mentoring and effective support is more likely to ensure quality advice and guidance is more universally accessible. Institutions must recognise however that instituting such a division of labour may appear to disenfranchise those tutors whose strengths include the personal, human touch, support which they’ve become accustomed to providing their students.

I believe this begins with an orientation to individual epistemological beliefs, a conversation that can begin before students even begin their formal programme of study, to ensure not that we are all on the same page but that we understand the page where on. POISE is one attempt at this meaningful orientation that can be integrated into online materials or initiated in face-to-face individual or group interactions. I believe we should also re-evaluate the our course designs to ensure that a full range of skills are built into intended learning outcomes to which constructively aligned teaching activity is targeted. Ensuring that all the domains of educational objectives are embedded within course designs, with supplemental teaching integrated into course delivery if necessary, will enable students to best help themselves on their study journey.

Better prepared students will lighten the burden on academic guidance services and ‘reduce’ the role of the traditional personal tutor. Where students identify needs they must be able to find clearly signposted support services, internal to the institution or increasingly externally, available on demand. This is not an abdication of responsibility on the behalf of institutions, quite the opposite. It is a recognition that increasingly higher education approximates more closely to the world of work than it does to the closeted environment of school. In reality most of our modern universities never had cloisters and students living around the quad, it is time for our support services to students to reflect that truth.

 


POISE Project: an institutional response to epistemological beliefs

November 4, 2014

(Extracted from a Draft Working Paper)

In the belief that student success in learning requires an awareness of one’s own epistemological belief structures one recent project, the POISE project, sought to acknowledge and reinforce the diverse cultural contexts in which learning occurs. It aimed to provide a toolkit to enable a consistent, supportive and transformative orientation to study, as a core provision for ALL students across ALL programmes of study. POISE was an institutional-wide change initiative, in partnership with the Higher Education Academy Change Initiative that reflects the global nature of the professional education BPP University offers to its undergraduate and postgraduate students in Business, Law and Health.

The original aim of POISE was to facilitate engagement with a POISE ‘toolkit’ by each student, and each member of faculty, in order that they “hear their own voice”. In doing so they become aware of their own unique epistemological belief structure and therefore of the uniqueness of others’ equally valid perspectives. This is vital for a higher education institution to actively demonstrate interest in individuals as learners and that such interest is fundamental to facilitating successful educational experiences. Each individual voice is as valid as anybody else’s and when heard students will be able to shape the delivery of teaching and learning activities. This happens because faculty become increasingly aware of diverse perspectives and students ability and willingness for greater engagement with fellow students’ unique frameworks. Given this loftly ambition the project team, a mixture of faculty and students, began identifying the themes in the epistemological literature and linking these to those areas of student ‘need’ of which we were already aware. We felt it was essential to develop a framework for student and faculty engagement based on the literature in order that future materials or issues would be contextualised. We were seeking to avoid the development of diverse and disconnected resources. The aim was to produce a ‘framework’ that would allow opportunities to engage in a ‘dialogue about beliefs’.

The notion of binaries presents an opportunity to engage in a ‘dialogue about beliefs’. We suggest that it is appropriate to establish the beliefs about learning that underpin a student’s (or faculty member’s) approach to learning and teaching, rather than to identify a ‘problem’ and tackle it with an intervention in isolation. For example, if it is believed that a student is not fully aware, or in tune with, the institutions guidance on plagiarism, it would be useful to introduce this dimension of academic practice by first exploring the question of whether knowledge is based on authority or reason. Without a fundamental understanding that the western academic tradition expects students to develop their own reasoning skills and to acknowledge pre-existing authority in a particular way, one cannot effectively explore the detailed nature of academic referencing, citations and intellectual ownership. Based around five dominant themes in the epistemological literature it was decided that we would use POISE (as an aide memoir or pneumonic) and follow a similar pattern, this resulted in the following matrix:

Binary concept Belief statements (after (Schommer, 1990)) Scholarship roots Pneumonic
Quick or not at all Learning is quick or not all (Quick Learning) (Schoenfeld, 1983) Pace
Authority or Reason Knowledge is handed down by authority (Omniscient Authority) (Perry, 1968) Ownership
Innate or Acquired The ability to learn is innate rather than acquired (Innate Ability) (Dweck & Leggett, 1988) Innateness
Simple or Complex Knowledge is simple rather than complex (Simple Knowledge) (Perry, 1968) Simplicity
Certain or Tentative Knowledge is certain rather than tentative (Certain Knowledge) (Perry, 1968) Exactness

After a sequence of internal workshops, complemented by thought-provoking events arranged by the Higher Education Academy as part of the HEA Internationalisation Change Programme, the team began to generate an expansive and highly ambitious ‘cradle to grave’ approach which required production of comprehensive learning support resources. However at an internal design workshop in January 2013 we determined that our enthusiasm to support the student experience in every way possible, although laudable, had created unrealistic expectations for our initiative. We determined that we needed to regain our focus on the original project aims concentrating on creating a solid foundation from which to build effective resources which relate to specific parts of the student experience (i.e. those concerning learning transitions, successful integration into learning communities and improvement of the effectiveness of student learning environments). Through further staff and student focus groups these ideas were developed and refined. It was determined that the original framework grounded in the literature was inaccessible to students, and perhaps the majority of faculty, and so the five dimensions of epistemological belief were re-cast as ‘open questions’

Pneumonic Binary concept POISE Questions Scholarship roots
Pace Quick or not at all Is hard work enough? (Schoenfeld, 1983)
Ownership Authority or Reason Who has the answers? (Perry, 1968)
Innateness Innate or Acquired Who is responsible for my learning? (Dweck & Leggett, 1988)
Simplicity Simple or Complex Is there a simple answer? (Perry, 1968)
Exactness Certain or Tentative Is there always a right answer? (Perry, 1968)

 

The project also benefited from external feedback given by other higher education institutions during the HEA Internationalisation Change Programme and at the HEA sponsored workshop held by Newman University College in February 2013 entitled “Developing Culturally Capable Staff”. This feedback related to the foundation, but also the proposed framework, and delivery of POISE. We concluded that pneumonic designed to recall Pace, Ownership, Innateness, Simplicity and Exactness also served as a project title making POISE about producing effective resources built on the following foundation:

  • PERSONAL – focussed on the individual;
  • ORIENTATION – not ‘cradle to grave’ solutions;
  • To the INTERNATIONAL – we define this as everyone’s context rather than a question of nationality;
  • STUDENT EXPERIENCE – focussed primarily on their learning and awareness of the self as learner.

The original concept had aimed at facilitation of the dialogue between the individual personal student and student. That these questions should frame staff development effort for all tutors in order that they would benefit from a greater personal insight into their own epistemological belief and be better to support the transition from naïve to sophisticated belief systems held by students. The internal School structure of the institution prevented against such an approach and the project resorted to developing a standalone web-based resource. The five themes have emerged as a series of five web pages, each containing a dialogue between two different perspectives, which explore each of the binary opposites outlined above. Each short video (less than three minutes) introduces the broad concept through opposing dialogue.

As part of the project many excellent ‘talking heads’ resources were considered and indeed a search on YouTube reveals dozens of international students talking about their experiences. Our original intention had been to add to this body of shareable testimonies and commentaries with similar live videos of individuals talking. However as the project developed it became clear that the most powerful evidence was not an individuals statements but what emerged in dialogue with others, and so a series of short vignettes of two or more students discussing their learning was deemed more appropriate. Understandably some of the participants in our developmental workshops were concerned that their honest declarations would be judged by others negatively and early attempts to have individuals act out previously heard dialogues were unconvincing. We also identified that all of us, every one of us, will make a ‘judgement’ on seeing and hearing someone speak. We bring all our own personal histories and assumptions to bear. So we wanted to find a way of sharing these valuable insights, short snippets of students’ conversations about the POISE questions, without the person watching ‘jumping to conclusions’. We sought to avoid a tendency to say “ah, yes, Japanese students would say that“, or “thats what British students always say about maths.“ So we decided to use cartoons. The voices are not as natural as one would like, but they are ‘neutral’. It is obvious that they are not ‘real people’, but the dialogue is. The words spoken are students words. In our workshops we have found that students and faculty watching the videos laugh a little at the ‘digital;’ voices on the first clip but soon acclimatize and start to listen to the actual dialogue. The dialogue between students has been only lightly edited and a transcript is therefore available for each video. This also means that as the technology improves we can always redo the cartoons for more and more natural voices.

It was anticipated that this would illuminate some well-documented (but evolving) cultural differences in expectations of study at higher levels and provide the student with a comprehensive personal ‘audit’ which they would use as the basis for discussion with their tutorial support. Faculty, having also engaged with the resource would be enabled with a common frame of reference and be encouraged to explore the similarities and differences in epistemological approaches of their approach with students, highlighting the impact this might have on learning practices. Faculty, the majority of whom are also Personal Tutors, would consequently be exposed to a greater range of supported and documented perspectives, moving beyond the anecdotal ‘challenges posed by International Students’ to a greater, and transparent, acknowledgement of the richness of learning and teaching opportunities contained within these different epistemological perspectives. It was also intended that existing institutional coaching and mentoring skills development for faculty would be used to support staff engagement with the POISE change initiative.

 

Bibliography

Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95(2), 256–273. doi:10.1037/0033-295X.95.2.256

Perry, W. G. (1968). Patterns of Development in Thought and Values of Students in a Liberal Arts College: A Validation of a Scheme. Final Report. Office of Education (DHEW), Washington, DC. Bureau of Research.

Schoenfeld, A. H. (1983). Beyond the Purely Cognitive: Belief Systems, Social Cognitions, and Metacognitions As Driving Forces in Intellectual Performance*. Cognitive Science, 7(4), 329–363. doi:10.1207/s15516709cog0704_3

Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82(3), 498–504. doi:10.1037/0022-0663.82.3.498

 


Starting with Epistemological Foundations

November 3, 2014

(Extracted for a Draft Working paper)

Distributed institutions face an increasing challenge to be able to afford, or deliver efficiently, the central services to meet diverse student needs. Changing contexts, notably the move towards the notion of the lifelong learner, means changing support structures and given the limitations of costs it becomes clear that foundations matter (Field, Gallacher, & Ingram, 2009).

Any review undertaken to inform the student support strategies identified a need for epistemological orientation in learning and teaching activities and assessment practices. The most common language used terms such as ‘orientation’ or ‘induction’ and the focus has tended to be on the ‘first-year experience’ (Nelson, Quinn, Marrington, & Clarke, 2012). Also highlighted were the range of excellent opportunities for greater international and culturally diverse insights to be contributed by staff and students from different backgrounds into the curriculum (Hammer, Bennett, & Wiseman, 2003). This need for enhancement was particularly relevant to the stages of pre-arrival support, pre-sessional support and induction within the Student Life-cycle.

Effective induction, orientation and retention require an appropriate epistemological foundation for all students a number of ways:

  • To enable students to achieve a more effective transition regarding pedagogical methodology from previous educational experiences to their experiences in UK higher education;
  • To facilitate the more effective integration of students into relevant learning communities;
  • To develop faculty awareness, skills and teaching strategies by the more effective application of epistemological orientations (evidenced notably as ‘cultural sensitivity’) into their teaching practice which will thereby develop the effectiveness of student learning environments.

A focus on epistemological orientation in the contemporary context of UK higher education must surely be to regard all students as ‘international students’, in that they are operating within a global context regardless of their discipline, nationality or status, and increasingly diverse nature of students necessarily should also force us to consider all students as ‘transitional students’.

Interest in how students learn is extended in the field of meta-cognitive research into student awareness and motivation for how they learn, the notion of ‘epistemological beliefs’. Research in this area is based in part on the assumption that students have discernible beliefs about the nature of knowledge and that these assumptions, or conceptions, affect their performance in learning activities. Schommer developed a questionnaire used to establish epistemological beliefs amongst college students in the United States and this instrument has been used in other studies to justify a number of cultural differences (Schommer, 1990, 1993). The question as to whether the instrument developed by Schommer is contextually specific, in other words whether it can be used outside of the context for which it was developed, is itself the subject of research (Clarebout, Elen, Luyten, & Bamps, 2001), however a review of epistemological beliefs instruments, drawing on the work of Bennett (M. J. Bennett, 1986), Magolda (Magolda, 1992) Schommer (Schommer, 1990, 1993), Schoenfeld (Schoenfeld, 1983), Dweck (Dweck & Leggett, 1988) and Perry (Perry, 1968) suggested there are fundamental questions students can and should be asked to establish their beliefs about learning. For example, Dweck explores the need for individuals to identify themselves as being and “entity” theorist or and “incremental” theorist, where in the first instance one believes that intelligence is largely fixed and cannot be changed through effort and incremental theorist believes they can increase their intelligence through effort (Dweck & Leggett, 1988) which lends itself to the fairly straightforward question; “Who is responsible for my learning?”.

The empirical date to support correlations between student success and epistemological beliefs is contested. In their review of epistemological beliefs survey instruments and their theoretical underpinnings, Hofer and Pintrich conclude that there is little consensus amongst researchers about the construction of epistemological belief and their relationships with other factors impacting on student attitude and approaches (Hofer & Pintrich, 1997). They express a particular concern with the conflagration of very different concepts in a single survey instrument, particularly the mixing of questions related to intelligence and to learning. In this early evaluation of the field they have already identified the difficulties of ambiguous language and assumptions being made by researchers about the clarity of underlying constructs. Schommer positions her own work as an attempt to diversify the dimensional aspects of earlier work by Perry who she argues produced a unique dimensional and fixed notion of personal epistemological development (Perry, 1968; Schommer, 1990). Schommer attempted to create more complex matrices of interdependent values recognising that beliefs may develop along different dimensions at different rates. She proposes five epistemological dimensions the first three of which relate to knowledge itself, namely structure, certainty, and source and to others which relate to the acquisition of knowledge, namely control and speed. For each of these dimensions Schommer proposes subsets of questions, 63 in all, creating a complex array of questions.

Recent attempts to explore the relationship between epistemological beliefs and meta-cognition, particularly the issue of how and why in epistemological beliefs have an impact on learning have explored the COPES model (Bromme, Pieschl, & Stahl, 2010). The COPES model conceptualise as epistemological beliefs as “internal conditions of learning” suggesting that beliefs are part of an internal self-regulation system (Greene & Azevedo, 2007). This system provides an internal conceptual framework about the nature of what is to be learnt and how such learning occurs. The justification for the study of epistemological beliefs and meta-cognition, and their relationship to self-regulated learning, is important if one accepts the premise that beliefs both scaffold and constrain the learners’ assumptions about both learning content and context. Epistemological beliefs can be seen as a lens through which the learner perceives the substance to be learnt as well as the landscape in which learning is to take place. The system the student puts in place to cope is represented by the notion of ‘self-regulated learning’ (Winne, 2005) .

Within educational psychology framework consisting of four identifiable and interrelated dimensions of belief provides a useful exploratory framework for epistemological beliefs (Hofer & Pintrich, 1997). The first two-dimensions, or binary juxtaposition, explores the nature of knowledge through the concept of certainty and structure. Certainty of knowledge is concerned with perceptions of the stability of knowledge and the strength of supporting evidence and structure, or simplicity, of knowledge seeks to define the complexity or connectedness of knowledge. The second binary juxtaposition is concerned with the nature of knowing, rather than knowledge itself, and contrast the justification of knowledge and the source of knowledge. Justification of knowledge is concerned with the process by which individuals evaluate and validate knowledge claims and the source of knowledge is concerned with notions of place and origin, where knowledge resides. There remains a debate as to the extent to which these dimensions are universal or whether there are domain or subject differences (Buehl & Alexander, 2001).

A core principle emerging from the educational psychology literature is that learners’ epistemological beliefs develop over time as a result of educational processes from the ‘naïve’ towards the ‘sophisticated’ (Bromme et al., 2010). A naïve epistemological framework would be one in which an individual sees knowledge as fixed and canonical, a collection of certain facts whose source is guaranteed by the authority the person declaring the knowledge. A sophisticated perspective might be described as one in which any knowledge claim is regarded as dependent on context and which is continuously challenged and re-evaluated through a process of social interaction. Whilst a naïve view might suggest that knowledge is unchanging whereas a sophisticated view defines knowledge as uncertain and continuously reconstructed. It is worth noting perhaps that the very language itself describing these two different epistemological beliefs could be regarded as pejorative since the term naïve and sophisticated carry significance in the English-language. Convinced, as I am that a profound shift in focus in our institutional support mechanisms for effective learning requires an engagement with the epistemological belief foundations of learners, perhaps a bigger problem is that such educational language, of psychology and meta-cognition, indeed of epistemology, is alien to the majority of teaching faculty and certainly beyond many students. What solutions present themselves?

 

Bibliography

Bennett, M. J. (1986). A developmental approach to training for intercultural sensitivity. International Journal of Intercultural Relations, 10(2), 179–196. doi:10.1016/0147-1767(86)90005-2

Bromme, R., Pieschl, S., & Stahl, E. (2010). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemological beliefs and metacognition. Metacognition and Learning, 5(1), 7–26. doi:10.1007/s11409-009-9053-5

Buehl, M. M., & Alexander, P. A. (2001). Beliefs About Academic Knowledge. Educational Psychology Review, 13(4), 385–418. doi:10.1023/A:1011917914756

Clarebout, G., Elen, J., Luyten, L., & Bamps, H. (2001). Assessing Epistemological Beliefs: Schommer’s Questionnaire Revisited. Educational Research and Evaluation, 7(1), 53–77. doi:10.1076/edre.7.1.53.6927

Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95(2), 256–273. doi:10.1037/0033-295X.95.2.256

Field, J., Gallacher, J., & Ingram, R. (Eds.). (2009). Researching Transitions in Lifelong Learning. London ; New York: Routledge.

Greene, J. A., & Azevedo, R. (2007). A Theoretical Review of Winne and Hadwin’s Model of Self-Regulated Learning: New Perspectives and Directions. Review of Educational Research, 77(3), 334–372.

Hammer, M. R., Bennett, M. J., & Wiseman, R. (2003). Measuring intercultural sensitivity: The intercultural development inventory. International Journal of Intercultural Relations, 27(4), 421–443. doi:10.1016/S0147-1767(03)00032-4

Hofer, B. K., & Pintrich, P. R. (1997). The Development of Epistemological Theories: Beliefs about Knowledge and Knowing and Their Relation to Learning. Review of Educational Research, 67(1), 88–140. doi:10.2307/1170620

Magolda, M. B. B. (1992). Knowing and Reasoning in College: Gender-related Patterns in Student’s Intellectual Development. John Wiley & Sons.

Nelson, K. J., Quinn, C., Marrington, A., & Clarke, J. A. (2012). Good practice for enhancing the engagement and success of commencing students. Higher Education, 63(1), 83–96. doi:10.1007/s10734-011-9426-y

Perry, W. G. (1968). Patterns of Development in Thought and Values of Students in a Liberal Arts College: A Validation of a Scheme. Final Report. Office of Education (DHEW), Washington, DC. Bureau of Research.

Schoenfeld, A. H. (1983). Beyond the Purely Cognitive: Belief Systems, Social Cognitions, and Metacognitions As Driving Forces in Intellectual Performance*. Cognitive Science, 7(4), 329–363. doi:10.1207/s15516709cog0704_3

Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82(3), 498–504. doi:10.1037/0022-0663.82.3.498

Schommer, M. (1993). Epistemological development and academic performance among secondary students. Journal of Educational Psychology, 85(3), 406–411. doi:10.1037/0022-0663.85.3.406

Winne, P. H. (2005). A Perspective on State-of-the-art Research on Self-regulated Learning. Instructional Science, 33(5-6), 559–565. doi:10.1007/s11251-005-1280-9

 


%d bloggers like this: