EDST5126 Week 8: teaching, assessment and feedback in blended and open learning environments 


ONLINE LEARNING……..DON’T THROW THE BABY OUT WITH THE BATHWATER!

I enjoyed tonight’s session…amazing how a little break can envigourate! 

  • Not blended learning, just learning – The format in which education is delivered must be fit for purpose. If education is situated within a constructive alignment paradigm, the teaching and learning activities will be conducted in the most suitable format for the articulated intended learning outcomes and that they are assessed in the most appropriate way…..this may include online at times,  and at other times, face to face……and most likely, a combination of different methodologies…..so not blended learning, just learning , which if constructively aligned will include the most relevant mix of formats. This definition is a more inelegant description of 
  • The perennial Einstein quote about the definition of insanity came to mind: simplyy hoping for the alchemy of  the improved learning in students by maintaining existing face-to-face content but delivering it in an online could be the embodiment of pedagogical insanity.  
  • This led onto an interesting discussion about constructing authentic learning outcomes and that we may need to look at new ways of articulating learning outcomes. I identified communication skills as an example where videos could serve a useful purpose and apparently Churchill has done some work on defining learning outcomes in the digital space, an articulate I will be interested to read and see what paralells can be drawn with other skills based settings like medical education 
  • Further extending the horses for courses concept, we explored other ways of creatively solving educational issues other than online learning, such as increasing hours that educational institutions are open and / or changing the structure of terms such as how UNSW is moving to the trimester. We pondered what unintended consequences may emerge as a result of these changes. There is the possibility of growth and renewal and innovation because of  if the opportunities afforded by this “spring clean” of existing courses, being able to look at current teaching methods and assessments……but there is equally the risk that the existing processes may be degraded if change is not consciously planned and if the opportunities aren’t recognised. I wonder how much strategic planning is occurring to reinvigorate courses and to reap the benefits of this change which is coming. 
  • We spoke about pitfalls in blended and online learning which I have attempted to capture in the mind map below. 


Lucky 13 pieces of food for thought. 

There were a few other serves of food for thought, which arose throughout the session, which I will list below for future reflection: 

  1. The start of innovation is building on what other people have done “the power of the second person”
  2. How can we conceptualise learning outcomes differently so that they are more useful? 
  3. How can we effectively relate learning outcomes to graduate capabilities? Should / can all learning outcomes be explicitly taught? 
  4. Can all learning outcomes be measured or assessed? What is the value of a dichotomous classification of achievement of learning outcomes? When does it all turn into “ticking boxes” and how can we keep them authentic as they were intended? 
  5. Embracing technology is all about teaching the teacher how to teach
  6. Online / blended learning should take no more time for the teacher and student than face to face
  7. How can we move from making higher education institutions from surviving to thriving and what would define that? 
  8. The more complex the tools the more complex the issues and the more expert the educator needs to be
  9. What defines engagement  in education? For teachers? For students? Is it a time investment or is it passion? 
  10. As university funding shifts from government to private, this will shift to balance of power from the public to the individual and may impact how teaching occurs eg demanding more intensive teaching, different ways of teaching. 
  11. What happens when the strategy of an institution is incongruent with the philosophy or practicality of a faculty / school…when one size doesn’t fit all
  12. Simulation is a starting point for medical education not a destination and can have particular use in rare conditions and procedures 
  13. How to we value add to teaching time? How is value defined in education? Is it learning? 

And so onto the task for this session: 


Why is there a rise of  blended learning in traditional on-campus higher education institutions, especially in Australia? 

Experiences of blended learning 

  • To date, my only experience with online learning has been as a consumer: through participation in courses for the FULT and GCULT courses at UNSW, using Moodle to access pre-class material, to view videos online and make notations and to interact on online forums. 
  • The most immersive and authentic experience was with EDST5123 which was conducted whollly online. I personally experienced a degree of “social anxiet” in interacting ***** INCOMPLETE 

EDST5126 Week 3 – Learning organisations and academic leadership 

Well this is another huge topic and I have procrastinated blogging about it because there was a great deal of material to ponder and digest.

This week was quite content and concept dense with several new frameworks, concepts and matrices introduced which are useful tools that can be used to evaluate organisations, change processes and leaders within the context of exploration of issues and means of addressing them, in other words introducing strategies to deal with these issues, including in the higher education context.

In this blog, I will summarise the various frameworks explored, looking at overlaps and inter-relationships between them and apply them to the context of a post graduate speciality training program currently in a phase of flux and renewal and potential change. Having the change to ‘play with’ these models here will be invaluable when it comes to the second assessment, and more importantly as I grapple with and am involved with changes in higher education in the future.

Learning Organisations (LO)

I learning organisation is not, as one might expect, an organisation whose core business is learning, but it is in fact an organisation which learns, and is defined using principles pertinent to learning in the context of education, but applied to organisations.

Three quotes from Senge, Cambron-McCabe, Lucas, Smith, & Dutton (2012) resonated really strongly with me as I was grappling to understand the concept of the LO:

  1. “Learning is at once deeply personal and inherently social” (p. 4) (importance of focusing on the individual, the groups and the organisation, flattening of hierarchies)
  2. “If we are ready for them, living and learning are inseparable” (p. 4) (importance of continual growth, reflection and change and aspirations)
  3. “Schools don’t exist in isolation, instead they have the potential to be fulcrum points for learning in the communities areound them” (p.7) (complexity of systems) 

My understanding was further enhanced through a quote in an OECD guide for educators on LOs, where it is described as a: “multi-level concept involving individual behaviour, team work, and organisation-wide practices and cultures” and as a “place where the beliefs, values and norms of employees are brought to bear in support of sustained learning; where a “learning atmosphere”, “learning culture” or “learning climate” is nurtured; and where “learning to learn” is essential for everyone involved” (OECD, p. 2

Learning organisations


Considering the five disciplines inherent to LOs further:


It seems to me that there are some potential challenges with a blanket acceptance of LO concepts:

  • It reflects a cultural shift and shouldn’t be universally applied or the whole idea will backfire without discussion.
  • Structures of existing organisations may not be conducive to implementing change (ie their structure is too pyramidal)
  • The bigger the organisation the harder it would be to implement – it would be too unwieldy
  • Requires a long term view, which needs to be balanced against the short term needs for productivity and maintaining financial viability etc which naturally must take front seat at times.
  • Cost if implementing them in a structured manner

With reference to Senge’s 5 disciplines – to what extent is your organisation a learning organisation? 

I will briefly consider the faculty which coordinates the post graduate vocational training program with which I am primarily involved from the perspective of Senge’s disciplines.

By means of background when describing the extent to which the faculty is a LO, it is worthwhile to understand recent changes and challenges which have faced the faculty.

The faculty is under the auspices of a larger organisation (‘the college’) and until about 2007, the faculty maintained independence in its policies, procedures, finances and culture. Due to changes in management at the college level, this position was largely reversed and is now subject to the college’s policies, procedures, management structure, has lost independent management of its money and the culture has been eroded and replaced by one of cynicism and frustration. This occurred against a backdrop of an in-depth review of the particular speciality’s training program (Parker, 2015) with view to a potential significant change of the training program to be in alignment with other college training programs in response to recent changes to the practice of the speciality itself. Members of the faculty found it difficult to separate this question from the other issues facing the faculty and it also occurred against other controversial issues facing the management of the college as a whole, including a high profile lawsuit brought about by the college’s fellows (Komesaroff, Kerridge, Isaacs, & Brooks, 2015).

About three years ago, a new breakaway “society” was formed in an effort to claw back the culture and representation of the interests of the speciality, which perceived to be the greatest lost was with the changes that had occurred. Specialist training remains situated under the auspices of the college. This has increased the number of fellows occupying leadership positions (now spread across two organisations).

Senge’s disciplines will be considered from perspective of:

  1. Fellows and trainees of the faculty
  2. Committees of teh faculty
  3. The Faculty / college staff: as part of the bigger college
  4. The “Trainee of the Future workshop”

I feel I can make informed comments, being a past trainee, a current fellow, a member of organisational committees, and as a past employee of the college.

Much of my commentary will related to my observations in relation to the review of our training program that I undertook and follow up activities related to that. In response to my review of training, a workshop was held in September 2016, to which all fellows / trainees of the faculty were invited. It was facilitated by an external facilitator. Of the approaximately 600 fellows and trainees, about 25 were in attendance. I found this workshop a very enlightening and satisfying day, having been deeply involved with the review from the get-go, and felt quite satisfied with how it was run and the outcomes, which felt in alignment with, and true to the original project. Many of my comments will pertain to this workshop as it reflects the underlying philosophy and direction of our current faculty president, whose report from that workshop can be found here.

Change management and how it relates to Senge’s Discplines

Villa and Thousand (1995), as detailed further in Shortland-Jones, Anderson, Baker (2001) describe six ingredients for successful change, the absence of each one of which leads to a predictable issue. Shortland-Jones et al (2001) use this very useful framework to examine changes introduced in their educational academic setting at an instution in WA where a philosophical shift in education was being introduced. This simple framework was a useful approach for evaluating the change which had occurred, which what was observed occurred and a direction forward.

Three quotes from their article particularly resonated with me because they summarise the use of the matrix and the complexity of change and how intimately it is related to subtle shifts in culture:

  • “By listening and responding to the teachers and students as the program is implemented, flexibility is maintained, adaptability encouraged, and ownership achieved.” Pg 1.
  • “This matrix….has been used as a means of analysing, and in some respects evaluating change”  (final page)
  • “Change is as complex as culture itself” (final page)

In the table below, I have looked at the change components on which Shortland’s article is based and have tried to correlate them corresponding disciplines using Senge’s framework.

At times, I feel quite overwhelmed by the multitudes of models and frameworks that exist, each of which has some degree of overlap with others about which I have read. Examining commonalities between different models is one way of  to drawing out the common elements in order to understand the core concepts being identified, using different languages and terminlogies. Having looked at two frameworks this week (Senge and Villa and Thousand), it strikes me that while Senge’s disciplines relate to a learning organisation, they are also directly related to effective change management because learning (and therefore LOs) is virtually synonymous with change – change in perspective, knowledge, practice. To stop learning is to stop changing (is to stop living in many ways!) Below, I compare these two frameworks.

More change models than you can poke a stick at 

Incidentally, in Subject EDST5124, one of the assessments related to examining change in our educational contexts. In that essay, I enjoyed comparing and contrasting changes afoot at both UNSW and the postgraduate college I am involved with (Parker, 2016). In that essay, I briefly summarised four frameworks which I found particularly useful and informative and an image from that essay is reproduced below.

IMG_0613.PNG

Leadership frameworks 

I wonder what teh collective noun for a group of frameworks is…perhaps it is a structure of frameworks. Nevertheless, in exploring Fullan and Ballew’s framework introduced in this week’s lecture, I cam across several others in a blog written by Webster (n.d.) including:

  1. Leadership that gets results (also known as Daniel Goleman’s golf clubs): uses concepts of emotional intelligence and identifies six leadership styles which should be used at the right time and in the right place to achieve good outcomes: affiliation, authoritative, coaching, democratic, pace-setting and coercive
  2. Situational leadership (Hershey and Blanchard): leadership styles are defined as one of four based on high and low scores on directive and supportive behaviours: directing, supporting, supporting and delegating
  3. Framework for leadership (Fullan and Ballew): described three personality characteristics (energy, enthusiasm and hope) along with five core leadership components: moral purpose, understanding change, relationship building, knowledge creation and sharing and coherence making.
  4. Agreement and certainty matrix (Stacey): examines proximity to certainty and agreement and looks at four states of change: simple, complicated, complex and chaos, and requires higher levels of interaction with those bringing about change and those affected by change (See below)
  5. Transformational leadership (Burns, Bass, and Kotter’s 8 step change model)

Agreement and certainty matrix (Stacy) – image from Webster (n.d.):

IMG_0614.PNG

From these five models of change leadership, Webster neatly summarises the four features of successful change leadership as:

  1. Self awareness
  2. Self mabatement
  3. Social awareness
  4. Relationship management

Phew….I am just about structurally frameworked out now. So let’s look at the one which was actually discussed in this week’s lecture, and try to relate it to Chang matrix and Senge’s disciplining and apply it to the faculty conundrum previously discussed.

What can we learn from Fullan  and Ballew’s 5 part Framework for leadership? What are the implications for your workplace? 

IMG_0612.PNG

Using Fallow and Ballew’s change leadership model, I will briefly consider how it relates to the change matrix and Senge’s discplines previously explored and will also examine features of leadership evident in the faculty and the changes it is currently facing as previously described in this blog:

IMG_0615.PNG

Much food for thought here.

There are a lot of frameworks and matrices to evaluate organisations and change and leadership. I wonder how their relative value can be assessed and evaluated and how and if their core commonalities can be drawn out.

References 


EDST 5126 Week 4 – Changing Curriculum and pedagogic design…..some things are timeless. 

I was first exposed to Biggs and Tang’s constructive alignment concept a couple of years ago and have personally found it, and parallel processes quite transformative in my thinking not only about education, but also my practice of medicine and life in general. Central to many aspects of life is having goals (intended learning outcomes). You then need to work out how you will achieve these goals (teaching and learning activities) and finally, you need to know whether or not you met the goals (assessment).

Source

Rehabilitation Medicine practice: I run an day rehab program for people with diabilities at a major teaching hospital in Sydney. I now more explicitly think about the ‘intended learning outcomes’ (patient centred goals which are established between the patient, their doctor and their treating team which are SMART goals (specific, measurable, achievable, realistic and time-contingent). We then design ‘teaching and learning activities’ (therapeutic interventions in which the client participates eg medical treatment of spasticity with injections of botulinum toxin and / or oral medications, plus physiotherapy), and an ‘assessment’ (clinical review) is undertaken at the end of the therapy program to determine whether the goals were met. (Incidentally, something I am very interested in exploring further at some point is how principles of education can be applied to learning new skills in a rehabilitaiton setting…..maybe one for the future!)

Post Graduate Medical Education: in our four year post graduate vocational specialist medical training program, there is an assessment which is currently conducted during the first year of training. Historically, this exam was an “entry exam” into the training program to ensure that trainees had the requisite foundational knowledge and skills for the program: it was not part of the teaching and learning program itself, but an entry barrier. As it was not part of the training program, it did not have associated intended learning outcomes and associated teaching and learning activities.  Due to an evolution of the training program, this exam morphed from being a barrier entry exam to being situated within and as an element of the training program. The problem was that as this exam became an embedded assessment, no intended learning outcomes (curriculum) and teaching and learning activities were developed in parallel. I explored some of these issues further in a review of our training program I was employed to undertake (Parker, 2015).

This has caused angst for:

  1. Trainees who, rightly so, have voiced concerns such as: “how am I meant to prepare for an exam that is a component of a training program when I don’t know what the intended learning outcomes are (there is no curriculum) and there are no associated teaching and learning activities.
  2. Examiners: how can an assessment be set with confidence and authenticity when there is no curriculum against which it can be blueprinted? Although I personally am not responsible for this exam, it comes under the auspices of the Faculty Assessment Committee of which I am a member and I have an ethical issue with this assessment how it is currently situated and have advocated strongly for change.
  3. The accreditation body: the Australian Medical Council (AMC) is responsible for accrediting specialit training programs in Australia. While the misalignment, or more accurately the non-alignment of this examination with intended learning outcomes and teaching and learning activities has not been identified by the accreditation body (yet), this is a potential stumbling block for future accreditation of the training program and needs to be addressed.

Where to from here? The faculty and College are currently undertaking a review of entry to training standards. The fortunate timing of this process offers a natural opportunity to revise how this ‘entry exam’ is defined within the program, so that can be not only seen to be serving its proported purpose, but actually doing what it needs to be!

So, to sum up, constructive alignment is central to many aspects of life from my perspective….and while the contexts may change, the underlying principles of: what am I trying to learn? How am I going to learn it? And how will I know that I have learnt it? Apply. Such fundamental principles are almost universal and ubiquitous. 
References

Biggs, J. B. (2011). Teaching for quality learning at university: What the student does. McGraw-Hill Education (UK).

Parker, S. (2015). The Rehabilitation Medicine Trainee of the Future. Retrieved from https://www.racp.edu.au/docs/default-source/default-document-library/the-rehabilitation-medicine-trainee-of-the-future-report801e52afbbb261c2b08bff00001c3177.pdf?sfvrsn=0

EDST5126 – mid course review: Peer and self review – working together

While the prospect of reviewing one’s work with a peer is an inherently anxiety-provoking activity for me, like with many situations where we face our fears. reality was much better than anticipated, and in fact turned out to be an extremely valuable activity. 

There-s-no-I-in-TEAM-but-there-is-a-ME

Why was peer review effective?

  1. ATTITUDE: the peer with whom I was working adopted a positive, non judgemental attitude of openness with the intention of improvement not judgement
  2. STRUCTURED: both what we were reviewing and the framework being used for evaluation (the SOLO framework (Biggs & Collis, 2014)
  3. POSITIVE PEER PRESSURE: peer review in a non-threatening open-minded environment at this half way point in this subject was an effective motivational aid for me to maintain my progress towards the course requirements. While adhering to structured deadlines can be frustrating, it ensures that the work is done which is particularly important for students who face competing personal, professional and academic commitments.
  4. BI-DIRECTIONAL learning: I learnt as much by giving feedback as I did from receiving it.

What I learnt:

Discussion forums 

  1. Our group (me included) requested the discussion forums to be included in this course. With the benefit of hindsight, given the other media being used to capture work in this course (ePortfolia), maintaining the discussion forums too has added an extra layer of work in an already busy subject with required weekly tasks
  2. I have been inconsistent with posting on the discussion forums and need to schedule timely review of the discussions in addition to writing the weekly blogs…..if I comment in bulk, the relevance and inertia of the discussion thread is somehow lost and has petered out which means that the comments I make do not contribute to the ongoing discussion as the participants have already moved on.
  3. The discussion topics often go off in interesting directions (chasing rabbits down bunny holes)….while this is academically and cognitively stimulating I need to try to bring some of these discussions back to the question of the week.
  4. I should try to reference the academic literature in posts more

ePortfolio

  1. While this has been strongly encouraged in other subjects in which I have enrolled, other than in the Foundations of University Learning and Teaching (FULT) course, I have not engaged with this format but have really enjoyed and can see the value in maintaining this to both fine tune my thinking but also as a means of capturing the artefacts of my work. It is also helping me to become familiar and comfortable with technology increasingly utilised by my students
  2. I really need to achieve a better balance between perfectionism and productivity when it comes to my posts, focusing on pithy, punchy posts
  3. I tend to use the blogs to capture new learning I have acquired in the week’s class (which is of value in and of itself for my future learning and reinforcing new concepts) as well as my reflections……in my final work submission, dissecting out the latter and increasing referencing to the literature in that will be important.
  4. Using the SOLO taxonomy, I am doing, while I do multiple unrelated points well and there are some unanticipated extensions, overall my posts lack the intermediate step of being ‘logically related’. This was also noted in my contributions to the discussion forums.
  5. Using Brookfield’s lenses (Brookfield, 2002) to structure my posts could be a means of focusing my work.
  6. I also need to incorporate comments and / or observations about how what I have learnt will change my thinking and practice.
  7. Despite these comments, it is good to observe that my work is evolving positively throughout the course.

Overall

I am really enjoying the class and the lively input from my peers and especially the organic discussions we have in class. The workload of polished weekly posts is proving to be quite challenging for me personally – keeping on topic, on task and on word limit.  While class discussions can feel somewhat free-range I have noticed that similar themes are brought up from week to week, for whatever the topic of the week is and thus, the key issues facing higher education are becoming increasingly clear to me as recurrent themes are seen to be spanning divergent topics.

References

  • Biggs, J. B., & Collis, K. F. (2014). Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed Learning Outcome). Academic Press.
  • Brookfield, S. D. (2002). Using the lenses of critically reflective teaching in the community college classroom. New Directions for Community Colleges,2002(118), 31-38.

EDST5126 Week 6: Changing learning environments

The main learning institution I am involved with (the College of Physicians) has its main physical location in Sydney…..but unlike most HE institutions, there is very minimal teaching which actually takes place here – it is HQ for administrative. The college’s teaching is disseminated and occurs in hospital and other health settings. In my blog post this week, I explored other elements of the teaching and learning environment and took significant liberty with the term “environment”. I concluded that, despite the lack of physical learning environments, there are a lot of aspects of the other aspects of the “learning environment” that could be enhanced.

IMG_0580

Environment Etymology 

My trusty online etymology dictionary tells me that environment is “a state of being environed” which is “the state of of conditions in which a person or thing lives” (from online etymology dictionary).

In subject EDST5122, one assessment task was to explore our own educational philosoph. My conceptualisation was of an educational tree which includes the elements and processes that influence an individual’s learning, metaphorically related to the environmental and other influences on a tree’s growth.

  1. Parker educational tree

These include:

  • Learning theories and frameworks
  • Global trends
  • Peers – the social learnign environment
  • Teachers – style, approach
  • Physical environment
  • Virtual environment
  • Political environment
  • Financial environment
  • Cultural environment
  • Educational resources such as equipment, technology, expertise

 

Weekly task – case study in educational environment

Identify strengths and weaknesses in your chose institution’s physical learning and teaching spaces. What possible changes to physical and/ or virtual spaces within the institutional resource constraints could improve student learning experiences? What planning should be considered for longer term improvements in your institution? 

Description

  • Australasian Faculty of Rehabilitation Medicine – specialist medical training faculty under the auspices of the Royal Australasian College of Physicians (RACP)
  • 4 year training program consisting of 8 six month terms
  • Knowledge / skills are a combination of codified and tacit, including professional competencies
  • Educational model: Vocational training which incorporates elements of cognitive and traditional apprenticeships. Very limited whole group teaching –  only a monthly synchronous online two hour teaching session.
  • Approximately 250 trainees spread across metropolitan, regional and rural setting
Issue Observation Advantages Disadvantages Comment / suggestion
Geography Great diversity of clinical settings occupy the vast majority of “teaching and learning spaces”. ·       Lessons learnt from being exposed to a wide variety of settings could advantage all trainees ·       Other than ensuring minimum standards for site accreditation, difficult to standardise teaching environments 1.     Review of accreditation standards for sites in the light of rise of technology in medical education

2.     Capitalise on diversity of clinical experiences through informal sharing of experiences / insights by trainees

Teachers Teaching conducted on a volunteer basis by clinical supervisors. Faculty employs no educators ·       Clinicians have real world authentic experience, skills and knowledge ·       Educators have no formal training in education, potentially disadvantaging trainees.

·       The bulk of teaching is conducted by an enthusiastic minority. Risk of burnout

3.     Educational up skilling of educators

4.     Develop educational special interest group of supervisors with an interest in teaching

Technology education gap between supervisors and trainees, resulting in mismatch of knowledge / experience / comfort with using technology to support teaching. ·       Teachers can be taught by students ·       If new technology is rolled out, supervisors may be unable to support their trainees in optimally using this technology 5.     Develop pathways to support supervisors as new technology is introduced.
Current learning environments and activities Trainees have informal social media groups not under the auspices of the institution Greater freedom in self-expression away from the watchful eye of “big brother” No governance or moderation of behaviour / content 6.     With input from trainees, develop a suitably moderated and functional means of trainee interaction
Monthly bi-national training program – non compulsory two hour monthly teaching in virtual environment using “Vidyo” platform funning from February to November with a program which runs over three years. Trainees can attend at RACP offices in capital cities or via their own computer or asynchronously via recorded presentations ·       Didactic presentations

·       Frequent technology issues

·       No pre-set reading material

·       No available pre-set learning objectives

·       Limited interaction

Online training portal Some excellent resources available ·       Limited knowledge of these resources 8.     Marketing strategy needed
Online modules developed by the college Some excellent modules available ·       There doesn’t appear to be a structured plan with the modules that are appearing.

·       Limited knowledge of online learning opportunities amongst trainees and fellows. No Rehabilitaiton medicine specific modules. Unkow

9.     Evaluate effectiveness

10.  Develop further modules, having surveyed members regarding need

The way forward: 

Given the geographical distribution of training, there needs to be s strong focus on

  1. Cataloguing current online learning environments and resources
  2. Establishing where there are defects between current teaching and future needs
  3. Evaluating research, trends and directions in post-graduate medical education internationally to come up with evidence based recommendations
  4. Developing a priority list of actions and a plan for implementation

Learning environments in medical education – what does the literature say?

A brief scan of the literature identified the increasing role of the evolving eLearning environment as a key issue in meducation education with questions such as: how can materials be standardised? What about issues of confidentiality? Can “hidden tacit facets of physician education” (Hafler, et al, 2011, p. 440) be satisfactorily taught via eLearning? (Ruiz, Minzer, & Leipzig, 2006).

I found this article by Gordon et al (2000) very relevant to my particular learning setting. They performed a SWOT analysis of clinical teaching environments and came up with several suggestions for improvement and for focusing future work, and I was pleased to see that these were consistent with the suggestions I had come up with to address identified issues.

References 

  • Gordon, J., Hazlett, C., Ten Cate, O., Mann, K., Kilminster, S., Prince, K., … & Newble, D. (2000). Strategic planning in medical education: enhancing the learning environment for students in clinical settings. Medical Education, 34(10), 841-850.
  • Hafler, J. P., Ownby, A. R., Thompson, B. M., Fasser, C. E., Grigsby, K., Haidet, P., … & Hafferty, F. W. (2011). Decoding the learning environment of medical education: a hidden curriculum perspective for faculty development. Academic Medicine, 86(4), 440-444.
  • Ruiz, J. G., Mintzer, M. J., & Leipzig, R. M. (2006). The impact of e-learning in medical education. Academic medicine, 81(3), 207-212.

 

 

 

EDST5126  Week 5 – Educational Evaluation

 

When you've finished changing, your're finished Franklin

Evaluation – what is it?

The etymology of Evaluation is from the French evaluer – in essence to extract the value out of something

Fox (2017) summarises three key elements of evaluation as:

  1. Assessing the merit / worth
  2. In a systematic manner
  3. With the purpose of making a judgement about the the worth of something, in other words its significance.

I find Burton’s simple three step process of evaluation (1970) very useful in exploring and conceptualising new concepts and new information.What so what now what

Prof Fox’s three point summary covers the first two elements (“What?” and “So what?”) well but seems to not capture so well the final step of “Now what? For me, evaluation must include that final phase of further action which may include change, or perhaps reinforcing the current state. Evaluation without action is interesting and perhaps academically rewarding but somewhat purposeless.

I have put the phases and elements of evaluation outlined in this week’s presentation / readings within Burton’s model

Evaluation what so what now what

What can be evaluated in HE?

Every aspect of higher education – people, processes and everything in between can be evaluated, which I have tried to capture in my image below.

what can be evaluated

Evaluation of impact of change is very important too and can be measured using Tilly Hinton’s IMPEL model, which we came across last year in EDST5124

IMPEL

What is in a tool?

A point which was highlighted in this evening’s class discussion is about the importance of evaluating the evaluation tool. This discussion echoed considerations of  evaluating assessment tools in education which I focused on in my assessments in EDST5122 last year. I have a particular interest in this area of formal assessment in education, being responsible for a high stakes examination for the medical speciality I am involved with.

The key elements of an assessment tool are summarised below. These are equally applicable when considering tools for evaluation in general.

  1. Reliable
  2. Valid
  3. Fair
  4. Feasible
  5. Cost effective
  6. Defensible
  7. Well-perceived
  8. Consistency
  9. Levelness

(Crosby, 2002 in Feather and Fry 2009 and QAA 2012)

These domains were expanded on through discussions this evening where the importance of considering these other areas was highlighted.

  1. Who designed the tool
  2. Who selected the tool
  3. The granularity of the tool
  4. The lens through which the results are evaluated: is it past focused or future focused?

Many of these aspects are highly relevant to the discussion underway about the tool being used to evaluate courses in HE ie moving from CATEI to My Experience. These issues were further highlighted by Uther (2017) in the class discussion forum where medicine courses across Australia were evaluated using various metrics from QILT from the perspective of both undergraduates and postgraduate students. These numbers could be used to support many arguments (“there are lies, damned lies and statistics”) and confounding factors (such as reputation of the university, numbers, culture, learning methods, teaching approaches and real world experiences.

Lie dam lies and statistics mark twain

Relationship between evaluation and research

Features          Research Evaluation
Purpose Produce generalisable new  knowledge Judges merit or worth of something specific
Why Conclusion oriented

= to improve

Decision oriented = to prove
How – methods Research tools Evaluation tools
Setting Often prescribed / controlled “Real world” environment
Directed by Researcher Party with a vested interest / stakeholder
Conclusions Make new research recommendations Recommendations are based on the question that was asked
Outputs Publish results / article Report for stakeholders
Evaluation of outputs Peer review Not usually peer reviewed

 

Features Journalist Academic
Depth More superficial Deeper
Independence from material Independent Vested interest
Relationship to truth Communicate the truth Discover the truth
Active More active More inactive
Product Writing Knowledge
Payment Usually independent of output Output determines future money (new grants)
Effect / impact  / next step Public opinion and funding Further research

May be reported by journalist

The discussions of comparing and contrasting research and evaluation and the approaches of journalists versus academics was interesting (and I have a distinct tendency to like putting ideas and concepts in boxes….it satisfies my mind’s need for order and control!). I struggled at first to understand why this exercise was undertaken……I felt that I had missed some nuanced point (perhaps parallels between these two comparisons with evaluation being more akin to journalists and research being more aligned with academics…..but this didn’t match wholly). I concluded that that the point of the exercise was by dissecting and breaking down the elements  to more deeply understand and appreciate and connect with the concept of evaluation, as it does have the tendency to have a rather nebulous ill-defined feel to it (much like the concept of governance!)

Why Evaluate?

Why evaluate slide.pngIf evaluation results in change and improvement, it most certainly adds value to an organisation……Evaluation must be planned for and systematic and occur at every level, from the big picture, right down to self evaluation, using a range of tools, as relevant to the things being evaluated. An organisation that does not evaluate will potentially either remain unchanged or change in ways which are not beneficial.

Trainee term evaluation in a medical speciality: a  case study

As an educational leader, if a reporter interviewed you about why your evaluation is necessary, how would you respond?

If the same reporter interviewed you about why evaluation can’t be done, how would you respond?

In responding to these questions, I have chosen to focus on the trainee term evaluation which is undertaken by vocational trainees in Rehabilitation Medicine of their six month terms. For the purposes of this scenario, as a leader within the training program, I am being interviewed by a trainee who is writing an article for the quarterly newsletter which is for fellows (graduates of the specialist medical program) and trainees.

TTEf

 TRAINEE: Why do trainees have to complete the term evaluation every six months? It seems pointless because nothing ever changes as a result of it being done. It is a waste of time and effort.  

 SHARI PARKER (SP): The training program is evaluated in a number of different ways and from a number of different perspectives. These processes are multi-directional and have both formal and informal elements.

 At the individual level, trainees have continuous formative assessment by their supervisors throughout the term. Summatively, there are the various training elements you are very familiar with like exams, external training module essays and formal long cases, as well as your six monthly term assessment. Overall, these are used to determine your progress through the training program.

 Trainees get to evaluate the term placement, their work conditions, teaching, facilities and so forth using the trainee term evaluation which is what you are asking me about. I will talk a bit more about that soon.

 Supervisors undergo evaluation regularly too when they seek re-accreditation as a supervisor. This requires a combination of having attended mandatory training, as well as a process of self-evaluation.

 The training sites and terms are evaluated using a structured process on a three yearly basis by the accreditation committee.

From an organisational perspective, the assessment processes and training elements are evaluated: both the processes and the results which reflect on the training program as a whole. If the pass rates are low, this means that there may be an issue with the training program as a whole.

A couple of years ago, I was actually employed by the college to review and evaluate our training program. As a result of this process, a number of recommendations were made, many of which are currently being actioned. You can see my report here .   This was the first time such a review was undertaken and it took into account perspectives of trainees, fellows and reviewed objective data and incorporated local and international trends and perspectives. While this was a very important process, in my opinion, there must be a plan in place to undertake such an activity perhaps every five years, rather than such a project being initiated by one particularly insightful and visionary president. This holistic evaluation process must be independent of the incumbent leadership for there to be true growth and development.

From an even wider lens, the training program is evaluated regularly by the Australian Medical Council which is responsible for accrediting specialist medical training programs in Australia. They look at a range of elements and key indicators in the training program and make recommendations. You can see a bit more about their processes here

 The trainee term evaluation is a really important cog within this overall evaluation process, each element of which is vital.

Just so you can be familiar with what actually happens after you complete the form, the data is entered into a spread sheet by college staff.  The “new fellow” representative on the trainee committee is the person who is responsible for reviewing this data and addressing any significant “red flag” issues such as inappropriate behaviour or bullying.  In such instances, the new fellow directly contacts the trainee concerned and together they come up with a plan moving forward.

The new fellow provides a report to the Education committee annually and feedback is provided to individual training sites as part of their three yearly re-accreditation site visit so that any systemic issues or patterns that are evident can be addressed.

All feedback is de-identified which his really important because there are few trainees, and particularly in small areas, trainees could be identified. The aim is to both act on the information that you provide but also to protect your confidence.

So, to summarise, the term evaluation is a vital part of the process of evaluation for the rehab medicine training program as a whole and is also important for the individual sites. Armed with this information, it would be great if the return rate for these evaluations could increase so that meaningful information can be provided and acted on. I know that there is a degree of cynicism amongst trainees (and I have been in that boat too!) that the term evaluation is just a “ticking boxes” activity and is shoved in a dusty drawer, never seeing the light of day, but I can personally attest to the importance that this information plays in both improving the trainees’ experience and ultimately the quality of graduating specialists.

If you, or any other trainees have any specific questions about the process or any other aspects of evaluation within the faculty, I would be very happy to address them. Equally, if trainees have any suggestions about how we can improve the process of evaluation, we would warmly welcome them! This feedback is a key part of evaluating the evaluations…..a kind of meta-evaluation!

Decisions decisions decisions…..what to write about for the second assessment!

Evaluation – peer observation of teaching (POT)

A topic which grabbed my attention tonight is one which I may take further in the second assessment task for this course: that of peer observation of teaching in clinical medicine. I first encountered this idea in subject EDST5122 where we were required to evaluate some examples of teaching. This was the first time I became aware of the process of POT and i recognised that, while this is an evolving area of HE which has the potential to improve teaching and learning, that it is largely absent from clinical teaching. That being said, POT is not without its risks, including potentially retrenchment or denial or promotion, changing the culture of an organisation and fostering cynicism of staff towards leadership and also whether this will actually achieve what it is set out to.

One potential outcome I superficially explored in my essay was how POT could be introduced in my own clinical context, potentially developing a template for it to be rolled out in my department, the clinical school at which I teach and possibly further afield. This evening’s tutorial has cemented this interest and it is an area I may delve further into in this subject in the second assessment.

Reflection……..a self evaluation!

What: In writing blog posts for this course,  I tend to try to do too much, resulting in long meandering posts which far exceed the word limit which take up a lot of my time. I try to cover too many areas in too much detail. I lose the woods for the trees. As a result, I am getting behind with the class work for this course.

So what?  The impact can be seen on a number of different levels:

  • Academic: there are rules (albeit flexible) with regards to word limits. If I don’t edit my work substantially, it will be outside the
  • A short punchy focused blog post will have far more impact: firstly because it is more likely to be read and secondly because any valuable ideas will be more clear and not lost within the noise of my logorrhoea.
  • Personally, spending too much time on posts is contributing to a sense of increasing overwhelm and stress. If I don’t get on top of this, there are potentially negative implications for me personally and academically.

Now What?

  • As I catch up with the blog posts I am behind with, I need to learn to drill down and distill the key elements of an issue, to be able to describe them succinctly and communicate them efficiently.
  • I will, with discipline, set a time limit for writing my blogs and stick to it, no matter what!

(note to self…..this reflection is 239 words and not too far off how long I should aim to make my blog posts!) 

rolling-eyes-loud

References

Borton, T. (1970). Reach, teach and touch. London: McGraw Hill. Carper, B (1978). Fundamental patterns of knowing in nursing. Advances in Nursing Practice1, 13-23.

Fox, B. (2017). Session 5: Evaluation (PowerPoint presentation). Retrieved 2017_03_28 from https://moodle.telt.unsw.edu.au/pluginfile.php/2506495/mod_resource/content/1/EDST5126_S5_2017.pptx

Feather, A., & Fry, H. (2009). Key aspects of teaching and learning in medicine and dentistry. A handbook for Teaching and Learning in Higher Education, 424.

The Quality Assurance Agency for Higher Education (QAA). (2012). Understanding assessment: its role in safeguarding academic standards and quality in higher education. Retrieved 2017_03_28 from http://www.qaa.ac.uk/en/Publications/Documents/understanding-assessment.pdf

Uther, P. (2017). EDST5126 Week 5 class discussion forum. Retrieved 2017_04_04 from https://moodle.telt.unsw.edu.au/mod/forum/discuss.php?d=514124

 

 

EDST5126 – Week 2 – Governance in Higher Education

Education more than governance 

dilbert governance

From http://dilbert.com/search_results?terms=governance&

 

“Education” comes from Latin “educates”, with meanings related to bringing up, rearing, bringing out or leading forth (Doublas Harper, 2017).  

 Despite the core purpose of education offering a directionality of knowledge acquisition and development, how this is achieved in a contemporary and rapidly evolving social and technological landscape is less well defined. Several inter-related and generic factors are driving change in higher education, including technology, monetary, market forces, social and political change, and international events. As a result, changes have been observed in the HE sector in relation to funding, competition, workload, priorities, and accountability. These forces and the resulting trends are explored by Marshall, Henry and Ramburth (2013), Commonwealth of Australia (2009), and Gallagher (2010).

I have chosen to reflect predominantly on accountability issues in relation to specialist medical education.

Accountability

While undergraduate medical training is, like other university courses, accountable to the public purse and through frameworks such as TEQSA and higher education standards framework, medical specialist training and colleges have different accountabilities and do not fit naturally within defined HE frameworks.

Specialist medical education occurs after completion of the primary medical degree and is delivered by “Colleges”, resulting in ‘fellowship’ of a particular training scheme, which entitles the graduate to practice as a specialist in a particular field.

Despite this training occurs over six plus years, and incorporates rigorous examinations, assessments and research requirements, the ‘fellowship’ qualification does not fit within the TEQSA framework and the resulting qualifications are not recognised along this continuum. Instead, they are regulated by the Australian Medical Council. They are further held accountable indirectly to the public and the medical fraternity, based on the quality of medical care delivered.

Some colleges are incorporating qualifications from the AQF framework as a component of their specialist training, which further muddies the waters of their classification. For example, HETI (Health Education Training institute) delivers the Master of Psychiatric Medicine which is one requirement for completion of fellowship (specialisation) in psychiatry. There could be scope for development of qualifications by HETI, perhaps in conjunction with the colleges, to deliver recognisable education programs as part of medical speciality training.

The question remains: should specialist medical training conform to other HE frameworks of accountability?

The argument against change (and managerialism)

Komsaroff, Kerridge, Issacs and Brooks’ 2015 article in the Medical Journal of Australia explored the concept of managerialism which they argue has taken over the health and education sectors and specifically the Royal Australasian College of Physicians (RACP).

MJA article RACP managerialism

They describe managerialism as a ‘scourge’, with “destructive” consequences. They describe it as a process where the focus on an organisation moves to the operation of an enterprise with “generalisation of the standards and practices of management” and a shift to privatisation and cost cutting, with standardised organisational structures, with the main regulatory principle being the market.

They indicate that through this process, “other criteria such as loyalty trust, care and a commitment to critical reflection have become displaced and devalued” and that these things “are viewed as quaint anachronisms with less importance and meaning than formal procedures or standards that can be readily linked to key performance indicators, budget end points, efficiency markers and externally imposed targets”.

They identify that this shift to managerialism has been seen in:

  1. Health care with “a shift in power from clinicians to managers” with priorities changing from “from a commitment to patient care to a primary concern with budgetary efficiency”
  2. University sector – where activities are “assessed in relation to the prosperity of the institution as a business enterprise rather than as a social one”, where education “is seen as a commodity like any other, with priority given to vocational skills rather than intellectual values”
  3. The RACP with a shift since approximately 2010 from physician member led activities and direction to ‘corporatisation”, with moves for the council to be replaced by a board, centralisation of activities, and reduction of democratic member input.

Aligning the accountability framework for specialist medical education with that of HE could be seen as an example of managerialism, fitting the institution into a ‘one size fits all” model being driven by market forces.

Where to from here? More questions than answers……

While I agree that it is important to not lose sight of the underlying philosophical purpose of higher education in general and specialist medical education in particular, the need for efficiency and accountability remains and also the rate of change on many fronts. The key question is how can a balance be struck between remaining true to ideologies, yet not allowing this ideological dedication paralyse evolution and progress within the context of international trends.

In addition to finding greater clarity with regards to accountability, other pressing challenges facing medical education include:

  1. How can graduate capabilities be meaningfully ensured in specialist medical training? This includes a shift towards Competency Based Medical Education (CBME) in both post graduate and undergraduate work
  2. Technology – how is it best for technology and simulation be incorporated into medical education? Is this shifting focus as relevant to medical education as it is to other fields? What features of medical education make

More questions than answers for now, but through the journey of this course, I hope to be able to answer them or at least make some inroads into finding greater clarity.

Reflection 

Maheno Shipwreck - fraser Island - QLD T (PB5D 00 51A1273)

I found this first topic really difficult being such a “big picture” one…….and have written about half a dozen starts to it, and procrastinated greatly on completing it! I am more a trees than a forest kind of gal! Many subjects I have enrolled in (quite reasonably) kick off with a “big picture” topic like this one did, and I typically find it hard to reflect on at the start, being almost overwhelmed with the many areas introduced at a surface level, without yet having the knowledge about where and how these themes will all come together, which is revealed throughout the course…..and yet it is important to have a big picture “fly over” at the start.

References

Commonwealth of Australia (2009) Transforming Australia’s Higher Education System, Commonwealth Government, Canberra.

Gallagher, M. (2010) Drivers of Policy Change: The accountability for quality agenda in higher education, The Group of Eight, Canberra

Douglas Harper (2017). Educate (v). Online Etymology Dictionary. Retrieved from: http://www.etymonline.com/index.php?term=educate.

Komesaroff, P. A., Kerridge, I. H., Isaacs, D., & Brooks, P. M. (2015). The scourge of managerialism and the Royal Australasian College of Physicians. Med J Aust202(10), 519-521.

Marshall, S., Henry, R., & Ramburuth, P. (Eds.). (2013). Improving assessment in higher education: A whole of institution approach. UNSW Press.