GCTLT: Assessing and Evaluating: What is your experience of moderation in your teaching context? – Discussion 3 for Ass 3

GCTLT: Assessing and Evaluating: What is your experience of moderation in your teaching context? – Discussion 3 for Ass 3

Before this year my experience has mainly been with post- moderation. Firstly it has been an ‘internal’ process where my colleagues and I will moderate each other’s work to look for consistency between our two sites (Hamilton and Dunedin) and also to check our feedback comments in terms of quality of statements. After this is completed the assessment committee also provides feedback and then marks are released to students.

The other moderation I have had experience in is via an external moderator who assesses the course based on set criteria involving, assessments, student feedback, and a report that I write as course coordinator. We have just debated this process as some staff felt it isn’t effective as the majority of time the moderator ‘concurs’ with the course coordinator’s reports. However, I have found the experience valuable in receiving confirmation for issues that I’ve raised, (as it has meant that there was ‘backup’ to address these in the following year), and also to sustain and grow areas that are working well.

This year I developed a Rubric marking template which meant I had to complete a pre-moderation process. This involved using previous year’s scripts to test the new Rubric. A colleague and I blind marked 10% of old scripts to test for consistency, and to check we complied with being fair and transparent.  The Rubric also went to the assessment committee for approval which generated some general comments including; clarifying some terms used in the template, and noting that if a student doesn’t quite ‘fit’ in one box- you can ‘highlight’ comments across a number of boxes to give more specific feedback (this was helpful advice when it came to actual marking in Semester 2). IT was only after this had been ‘tested’ that it was released to students.

To support this we also developed a teaching strategy that had a number of components to support this assignment. These included a face to face workshop about the assessment and Rubric marking sheet.  Students also had access to online material they could refer to in their own time which included web-links, a word document that outlined general comments from me in regard to the assignment, and an open Moodle forum where they could post any questions or comments. Both myself and my colleague from Hamilton posted comments on this forum. We also posted reflective questions students could explore themselves in their own online forums (such as via the students’ facebook page).

• Why is moderation important?

Internal moderation is important for consistency within a course (especially when you have multiple markers), to ensure consistency between courses and to give assurance to students that their mark is ‘reliable’ as per OP’s assessment policy (2009). ACU National (2008) also identified these aims in the moderation process: comparability of assessment; quality of assessment; and  adherence to academic standards

Internal moderation can also help trouble shoot any issues (ie if one marker is unsure then they can utilise other staff to give an opinion and/or the assessment committee).

External moderation is also important to give consistency across the programme. I have found this helpful for checking if assessments are at the correct year level across the programme as a whole (ie we are not asking first years to complete an assessment task that should be at 3rd year level and vice versa).

Pre and post moderation are important processes to "verify that assessment design and assessor judgements are valid, fair, consistent and reliable; meet approved learning outcomes; and are appropriate to the learning environment". (Otago Polytechnic Academic Policy: Moderation of Assessment, AP908.00, 2007, p 1.) .

• What are the differences between pre and post-assessment moderation?

Pre assessment is a process we engage in where we check information/processes in regard to new assessment strategies before they are given to students. This ensures students only receive assessment strategies which have been ‘trialled and tested’.

Post assessment happens during and after an assessment occurs which I engage in regularly with my colleagues as detailed above.

I thought the comment in ACU National about the ‘spirit’ of moderation was interesting…

·         “Moderation of assessment by self-review, School processes and, where applicable, internal moderators are integral to quality assessment practices each time a unit is offered.

·         Moderation of assessment by an independent moderator/s at regular intervals provides opportunities for independent feedback.

·         Moderation will be most effective when conducted in a spirit of professional learning and quality improvement.” (ACU National, 2008; 433)

References:

ACU National (2008). Principals for Moderation for Assessments. Retrieved from: http://www.acu.edu.au/__data/assets/pdf_file/0004/150934/Moderation_principles-final.pdf

Otago Polytechnic (2009) Otago Polytechnic Academic Policy: Assessment. Retrieved from: http://moodle.op.ac.nz/file.php/62/Policies/AP0900.04_Assessment.pdf

GCTLT- Assessing and Evaluating:- What feedback models or processes do you prefer? Discussion 2 for Ass 3.

GCTLT- Assessing and Evaluating:- What feedback models or processes do you prefer? Discussion 2 for Ass 3.

Personally, I don’t have a preference regarding feedback – I have had benefit from both receiving verbal and written feedback and also delivering these both to students.

I do however have a preference for timing of the feedback- if I’m working on a large project, I like on-going feedback that is structured- i.e. less of the ‘you’re doing a great job’ to being more specific as to what’s working well, what needs more attention and where to next. I also enjoy the opportunity to have multiple attempts before ‘submitting’ or the ability to have another go. I have experienced this through my learning te reo Maori through Te Wananga o Aotearoa and I offer this option to my students when they have a go at their mihi at noho marae.

• Why is feedback important?

Feedback is important to identify areas of learning that students need to improve on or to enhance areas they may be quite strong in. I have gained a lot from reading Race, Brown and Smith (2005), Fostaty Young (2005) and Spiller (2009). In particular their ideas that the assessment process should contain no surprises, that  feedback should contain statements of evidence that are not confined to ‘what’s missing’ and to think about offering Feed –forward comments so students can  clearly know what is expected of them in their next assessments have been useful to my own area of teaching.

I also think encouraging feedback from students is also important- and from other colleagues.

• What are the challenges or issues with assessment feedback?

Time is always a factor. The 1 hour per student marking time is a policy that seems more to do with budget constraints rather than evidence based practice concerning delivering quality feedback.

I also think it’s a challenge to make sure students have the necessary tools to complete the assessments- but that we don’t scaffold too much that they don’t end up learning to climb for themselves. The course I teach in is a bachelor’s course, so this expectation of competency would be adjusted say if I was working with foundation learners.

I have also put more effort in helping students understand expectations prior to the assessment but also in using the feedback given. After they receive feedback from one assessment- we use this information to plan for their second assessment.

• How do you balance how much feedback to give?

I’ve developed Rubric templates for their written assessments. The first page is the “Feedback” grades and comments, and we go through this extensively in class before students submit their work. I then have another box where I write a short paragraph with ‘feed-forward’ comments- such as what they should focus on in their next assignment. I also use this feed-forward box when I mark their next assignment so I can assess if they have addressed any concerns raised (e.g. such as referencing), or followed through with extending their learning. Students are also invited for 1:1 feedback that is via face to face or email. At noho marae- students receive verbal feedback and there are a number of staff that help assess this in a collaborative way.

References

Brown, S., Race, P., & Smith, B.(2005). 500 tips on assessment. London: Routledge Falmer.  

Fostaty Young, S. (2005). Teaching, learning, and assessment in higher education: Using ICE to improve student learning. In the proceedings of the Improving Student Learning Symposium. London, UK, Imperial College. (Volume 13, 105-115.) 

Spiller, D. (2009) Assessment: Feedback to promote student learning. Teaching Development. Wāhanga Whakapakari Ako. The University of Waikato. Retrieved from: http://www.waikato.ac.nz/tdu/pdf/booklets/6_AssessmentFeedback.pdf

GLTLT- Assessing and Evaluation: What does cultural diversity in relation to assessments mean to you?

10 August 2012 - What does cultural diversity in relation to assessments mean to you? Discussion 1 for Ass 3.   
 

Kia ora koutou

As most of the class has already mentioned, cultural diversity is a complex yet rich term that as facilitators we can use to our better our professional practice. I like Monaghan,J. & Just, P.(2000) description of cultural anthropology whereby they explain that the discipline studies ‘social relations’ and ‘cultural logic’ of people. I think these phrases can also be applied to tertiary facilitators/teachers where we are not only working out the relationships between groups of students/colleagues/wider community etc, but also the cultural logic that drive the individuals within these groups. When I think of the cohort of students I teach- it’s working out how I can encourage the best learning without falling into the trap of trying to be everything to everyone which means I have to think of ways of getting to know their learner needs and aspirations .

I also agree with many of the postings that culture can encompass ethnicity but I also understand that the two terms are significantly different (as are ethnicity and nationality). Some students may be quite comfortable identifying with their ethnicity/ies- whereas others are more comfortable identifying through their occupations/ nationality etc.

In relation to cultural diversity and assessments- this means I accept diversity- I don’t ignore or generalise diversity. What has helped me in meeting this challenge is being flexible in offering multiple pathways for how students can access information and/or support to engage with the assessments early on in the paper I teach and having clear expectations and explaining/demonstrating  these expectations to students in a variety of ways.

• What factors need to be considered?

“Cultures make sense of the world through values, norms and behaviour they acquire and assessments need to resonate with their beliefs. By not recognising factors of diversity in the assessment we as educators run the risk of failing to meet educational needs and therefore increase social disadvantages in communities” (Rameka,2007).

I was interested in Rameka’s (2007) comment above as on one hand I fully understand what Rameka is saying as you only have to look at the latest Maori post-secondary qualifications to recognise the systemic failings in our educational system. I think to meet this challenge however, is that tertiary facilitators need to practice and gain feedback about ’ how’ this happens because without this I find that teachers either continue to do what they do (which is ignore diversity and do nothing) as we don’t want to make a ‘mistake’. Liz Ditzel’s posting (24/0/12) is great in giving practical tips.

I challenge my students when they ‘see’ me about what assumptions are made about how I might teach? Because I’m Maori, Tongan and Pakeha- am I going to communicate a certain way, am I going to deliver course material and assess in a different way say compared with my Pakeha colleagues? Do I want to teach in groups or at an individual level, do they think I prefer face to face or is online going to be OK? Do they have higher or lower expectations from me?

 They all tell me that’s rubbish and they will ‘judge’ my skills and abilities as they get to know me. But we all make judgements about each other – that part is normal- it’s how these effect our professional practice that we all need to work out.  There is now extensive research that shows how teachers’ beliefs and values do impact on students’ learning (both positively and negatively) – (see Russell Bishop’s work in this area in the compulsory sector: http://www.waikato.ac.nz/php/research.php?mode=show&author=rbishop).  So for me a factor I have to consider then; is to what extent do my judgements impact on my teaching?

When I look at this statement in OP’s assessment policy: “Assessments improve student learning. In the design of the assessment the values need to be acceptable including fairness, relevance and respect to all the students”, then I can’t help but challenge how the terms fair, relevant and respect are defined? And how these may differ between me/students/ other colleagues?

 I can explain why to my students we don’t eat in class, sit on tables, and give koha when we have guest speakers in, but that’s not only because of my own ethnicity, but because I understand (and tell students) the institution I work for has a MOU with our local runaka and in this agreement it states that as a whole institution we will abide by our runaka’s tikaka (cultural) practices as per the Treaty of Waitangi. I find once students have context and logic presented to them for cultural practices- then the communication (i.e. social relations) flows a lot better as they can then participate in a meaningful way (sharing in cultural logic).

• How can consideration of diversity benefit the learner?

It actually benefits us all by teaching patience and tolerance- (but only through maintaining high standards). By acknowledging diversity amongst learners means I don’t expect my students to be the ‘same’ so I have to have different ways of communicating and presenting information so they can access learning.

It also allows the culture of learning (ako) and facilitation to be challenged in different ways. As part of an in-class activity this year  I submitted a short piece of work to students that I wrote and I then asked them to award a grade with a short rationale as to why. Apart from the ‘joke’ that they were now assessing me, I encouraged them to use my marking criteria as a close guide and to help them come to terms with ‘assessment speak’. They then had to present back as to what mark they would give me and why. I then marked my own work in front of the class, and explained why. I was at least a grade or two below the students’ grades and although they said I should be nicer to myself- it was a good activity to clearly show where the expectation was. We then worked as groups to change, edit and develop the work I had done to gain a higher quality of work. This exercise took about half an hour but I think it was valuable as it offered an alternative method to explaining about assessment, by instead demonstrating marking in real time to students and through the discussion we had.

• What are the challenges?

One of the big challenges is breaking down the myth that Culture diversity is about being ‘different’ from the ‘norm’. I find that once students can articulate clearly about their own cultural context (which we do in one of their assessments) this helps ‘tease’ the term culture out. Culture is a concept that; applies to groups, is learned, includes beliefs, values, ideas , material artifacts and behaviours and changes over times (although it ‘roots’ are in the past), (Peter Morris, 2000).

If we accept Morris’ ideas about culture, then we are all cultural begins! The other challenge of course is trying to be continually ‘savvy’ as to how you use your time with students- to get the most ‘benefit’ out of the learning context….

Ka mihi Gina

References:

Monaghan,J. & Just, P.(2000). Social and cultural anthropology: A very short introduction. Oxford: Oxford University Press.

Morris, P. (2000). Culture a brief look at it’s definition. Retrieved from: http://homepage.smc.edu/morris_pete/resources/Papers-and-Presentations/culture.pdf

Otago Polytechnic (2009) Otago Polytechnic Academic Policy: Assessment. Retrieved from: http://moodle.op.ac.nz/file.php/62/Policies/AP0900.04_Assessment.pdf

Rameka, L. (2007). Mäori Approaches to Assessment.Canadian Journal of Native Education. 30 (1), p. 126 - 191.

Draft Presentation of my FL plan

Kia ora koutou

Congratulations on nearly finishing!

I've separated my draft presentation of my FL plan into 3 parts.

You can choose which you want to watch and as the links connect to swf files you can skip through any material. The files do contain sound but they are fine just to read through as well so if you don't like the beats - you can mute it. I do let you know when there is actual audio clips (there are 3 interviews in total with other learners). Also they take a couple minutes to load...

 Here's a summary...

1. Introduction- looks at my understanding of FL (3 minutes). see LINK

2. Part 2 discusses how key learning theories, concepts (such as Cultural Sensitivity) and organisational policies/strategies continue to shape FL in my course (4 minutes). see LINK

3. I look at one learning approach and discuss how access, equity, diversity, inclusivity, sustainability & open education factors can shape flexible learning specific to this context. (NOTE: the presentation is 5 minutes- but there are 2 interviews worth listening to for about 5 minutes also -your choice!). see LINK

I am presenting next Monday and any feedback is welcome.
In my presentation I intend to pick a few key slides from the above presentations so I can keep to time...
Thanks Gina

PS here are the url details to cut and paste into your browser if the links above are not working....


1. INTRO:                               http://swfcabin.com/open/1340121532
2. KEY CONCEPTS:             http://www.swfcabin.com/open/1340122766
3. EXAMPLE:                        http://www.swfcabin.com/open/1340123101

Activity 12: Tertiary Education Strategy

How do your ideas for flexible learning, and those of your organisation or consultancy fit (or not) with the TES for NZ? 

To tell you the truth I preferred the other activity for this part- but maybe that’s because I survived 10 years working for the Ministry…  The TES is a broad overview with the aim of causing change at a systems level which ‘trickles’ down to communities, groups, individuals. My focus in teaching is a ground up approach that focusses on an individual’s learning, and then goes out to groups, communities, systems. Although they are part of the same education picture- this difference in approach isn’t always easy to translate amongst the two. The other area that is hard to tackle is that the TES obviously is politically motivated to encourage societal change (for the best?), whereas an organisation’s strategy needs to be robust enough to endure political intent and maintain future focus, as obviously a government’s focus changes over time (or more to the point government and Ministers change over time!). So that is the other reason that I am more motivated to look at OP’s strategy... and see how it’s placed.

In broad terms, yes my education philosophy supports the expectations outlined in the TES for the tertiary sector . In terms of how FL can be used to support these ideas, there has been some amazing innovative practices going on across the education sector to draw from -from Homework centres, to Computers in Home projects, and more recently with innovative practices in Christchurch – the disappointing part in this however is that sometimes the funding has been locked within ‘trials’ and it seems to be increasingly difficult to get innovative approaches shifted into operational funding. 

So I would say that ‘yes’ my ideas for FL could fit with the TES’s broad statements, but the practical reality of how these ideas are implemented  takes a bit more effort to align with the TES. For example, there is a major focus on the retention of Maori and Pasifika students in tertiary education in the TES. I’m really lucky that in my course there is key learning in both of these 'focus' areas- but I have little influence how OP or OT attracts and supports Maori and/or Pasifika students into my course. Therefore in meeting TES goal of retention and course completion among Maori and Pasifika my course on one hand would look strong (last year this was 100%), but actual student numbers are very small- less than 10% in my class identified as Maori and I had no Pasifika students in 2011. Is it fair my course seems   potentially more ‘viable’ in regard to meeting one of the major TES goals when in reality the numbers of students entering the course is still low compared with population rates?  

I was also interested in your comment Bronwyn “The tertiary education sector is highly competitive, and now that funding is shrinking even further organisations are scrabbling for a slice of the pie. Is Flexible Learning the answer?” 

I too have heard FL being promoted as a $$$ saving option –as learners are given a wide range of choices to access knowledge and skills, which means that you can mitigate some operational costs such as lecture theatres etc.  But it is very difficult to define true cost-saving given that there are a lot of hidden dollars in the tertiary sector picture. (For example with staff marking time, costs to individual learners for own IT equipment, internet usage etc). I remember at OP staff inductions with HR it was relayed to new staff that having a blended approach has helped saved the institution money- which I believe is true- but I also know staff who have left because of burn-out issues (not just at OP but across the sector as a whole), and I wonder how ‘actuals’ are calculated into this mix of determining cost? 

What do you need to include in your plan to fit with the TES for NZ?

I think the most important action I can take is to continue to offer quality teaching of which FL is part of this picutre. Obviously this is a core intent of our learners, but If I can maintain and grow my quality teaching practice, then my students aren’t going to drop out (and therefore I can help meet TES’s focus on student retention).