Transcript of keynote address (full version), L&T Forum, May 2012
The following is a full transcript of the Keynote Address, "Assuring Graduate Capabilities: Assessing, measuring and evidencing standards", delivered by Professor Beverley Oliver, Pro Vice-Chancellor (Learning Futures) at Deakin University. Delivered at the Learning and Teaching Forum: Assurance of Learning, on 11 May 2012.
Really I want to talk today about assessment, which I think is the pointy end. If you want to review and renew the curriculum and change the world, start with the assessment—but measuring and evidencing, and I want to talk about those two things because they're not always the same. You can't measure everything that counts.
Let's start by talking about your favourite subject; that would be you. How did you get to the table today?
No, I don't want to really hear about the traffic, thank you. How did you get to where you are? I started by saying you are a graduate of a university. You probably started out—possibly started out—in a completely different discipline. You may have started somewhere else; you may have started in employment; you may have an interesting route to get to where you are. Who's had an interesting route, to get to where they are? I have, yes. I wasn't thinking I'd end up doing this. How did you get here, and how do we make sure our students get to the right place as well?
So let's start with your most recent job interview—oh yes. Mine, in fact, was about 5 months ago. I'm new in my job. And I need to say, by the way, that a lot of the work I'll be talking about at this stage is also about my previous institution, which is Curtin University, so I've got a couple of things to draw on. But your last job interview, that was the rate determining step; whether you were successful or not made a big difference. Was it your finest hour? Well, you must have done okay, otherwise you wouldn't have been successful, presuming you were. What did the people at the table look at? What got you there?
Here we have an academic transcript of sorts. Blah-blah-blah-blah-blah, we've all got one of those. They're full of things that really don't tell us very much. But without it, you really can't do very much, because it's your ticket to ride. So if you want to be a pharmacist, you have to have a pharmacy degree just to get through the door. Although interestingly, if—I'll just digress slightly—if you want to teach in a university, you may or may not need qualifications for that, but we wouldn't let you out to practise in any other profession without a qualification. We digress; that was a word from our sponsors, I think.
Okay. What else have you got? What got you there, or what got the last interviewee to the table when you talked about it? This work of fiction called your resume. Who checks it? Who warrants it? Who says it's true? Especially if you apply for a job somewhere on the other side of the country—oh, maybe I shouldn't say that. Somewhere where you're not particularly well-known. But also, our resumes are full of things, particularly if we worked in universities, like, "I chaired Committee X, I did a fabulous job." Well, everyone may not see it that way. That's a claim, it's an assertion, hard to evidence. Or did you actually call on your 3 best friends as your referees, who of course were going to say "yeah, yeah, yeah; great, great, great." I mean, you do pick referees who you're pretty sure will support you, don't you? Don't you? Yes, of course you do. You've also got a digital footprint, of course; we all have. So people may have checked that as well. They're the things that we find out about each other, before you get to spend an hour across the table with someone, and decide whether you're going to give them the next ongoing position. So you may get stuck with these people; you don't have much time to find out.
My point is this, though: that's a game. We have a game in higher education, and you need to learn to play the game. In fact, we have a game in life, and you also need to learn to play that game. And the game is, of course, that you need to do well in that one hour. How much time, in your undergraduate or postgraduate education, did you actually spend preparing for the one hour? I'd suggest, most of your energy went into what was on the bottom left-hand side of the screen, which is this thing called an academic transcript. At the interview table when your life takes a leap forward and your chances improve and your career moves forward, all of the other stuff actually probably counts for more, and what counts for even more is whether you make eye contact across the table, whether you look as though you can engage, and so on. it's all the human factors.
So, just a thought about the time and energy we put into university education on the bottom left-hand side of the screen. I'm not saying we should spend all our lives getting ready for interviews, not at all. But it's just in our curriculum at the moment, we tend to be very knowledge driven, and we tend to measure, measure, measure. But the latest I heard, nobody got taller because they kept being measured.
We have other games, in universities. What are those games, and how do we play it? Well, let me put it to you, that we have a game, called "University Entry and Exit and Success". Here is the course that you're thinking about doing. How do you get into that course? Well, there's a magic number. Does the ATAR mean something in New South Wales? It's the same? Yep. Right. Well, I'm from Perth and of course we have a different acronym. Apparently our trains run on the same standard gauge railway now, but that's fairly recent. But there's an acronym and a number, and we need that number, because we've got mass education and we can't sit down and interview everybody. I know some people do, in some disciplines. Basically, though, you either have recognition for prior learning, some points or something, or an ATAR, that gets you in the door. When you're in the door, you actually cash in the credits. You gather credits, and you actually invest in this thing. So you get numbers, on a piece of paper, or letter grades. And then when you leave we play the game by saying, well, there's a number for that; there's the Course Experience Questionnaire where number 1, you know, based on, how many responses, no-no-no we don't want to know how many responses... Graduate Destination Survey, how many people 4 months after graduation have a job, are looking for a job, are working—is that what they wanted to do? I don't know. Were they any good at the job? Don't know. We don't know. We don't have any data. Pretty scary, isn't it? That's a pretty dicey game. This is a billion-dollar industry we're talking about here.
So maybe we might need to do something different. If you're the student, the game might look, inside, like this, and this might look familiar to you.
[Slide: The game—100 pieces of assessment. Years 1 to 3, each year consisting of 8 boxes, each containing 4 assessment items]
So here you have a course. Let's just say—now, a course, I'm saying, is a degree, is a program. Are we good with that? Good. Inside the course are things called units. Does that make sense? Good. Let's just say there are 24 sort of bog-standard units in a sort of bog-standard undergraduate degree. We good with that? Good. Well, if you're the unit chair or director or coordinator of one of those units you probably have 4 assessment points, or 3, or 5, or 15 if anyone will let you, but let's just say 4. Okay, great, so... If you sort of scale up from there, there's kind of, wait for it, yep, about a hundred of those things. What are they? There are one hundred assessment events—ish—in a bog-standard undergraduate degree. And of course they're all very carefully designed, by experts, who know exactly what they're doing—oh no, that's right, you don't need a qualification to teach in higher education, I forgot!
Well, basically, our colleagues, and us, do what was done to us, sometimes, or we get an assessment toolkit, not unlike the one that you've got on your fabulous site, which is fantastic—well done. But we sort of cobble it together as best we can. A culture in many universities is that these units are actually little silos, and oftentimes what's in that one doesn't really relate to that one and so on. So there may be a lot of underlap there, there may be a lot of overlap, we don't know. Or it may be all beautifully mapped out and so on. I think we're moving towards that; that's great.
So thinking of those sort of 100 pieces of, let me just call them—use a technical term—"busy work", because some of them are. There'll be some multiple-choice tests there, some exams, some essays, some this, some that, some other things, some clinical and so on. If you were the student, what might it look like for you?
[Slide—degree as a series of barriers, with assessments as obstacles of varying height.]
[laughter] Well, welcome to your degree. As you thud through the semesters ahead, and you go through every unit, you might actually find: there it is. And you graduate—well done! What does Paul Ramsden tall us about teaching and learning in higher education? What game do we learn to play? Well, when I was an undergraduate on the other side of the country, last century, I learnt to get 50 per cent. Because I figured that was my ticket out of there. And if I got fifty-one or -two—what was I thinking? As a 17-year-old, that's where I was at. I suspect we have many students like that, who are still the same. Fortunately I grew up and changed my thinking, but—well, I changed my thinking, anyway. But, it can look like that, and the game is to get through there as fast as possible. Or not. Not every student is like this, but many are quite transactional.
Now, I always find a bit of exaggeration helps you to tell a better story, so do forgive me if I'm exaggerating slightly—although I do see some knowing nods and smiles around the room.
I'm a great fan of the work of Mantz Yorke, scholar from the UK. I particularly like this book and I recommend it highly: Grading Student Achievement in Higher Education: signals and shortcomings [Abingdon: Routledge, 2008], downloadable on your iPad now through Kindle. It's expensive, though; it is an expensive book. I think it's fantastic. He says assessment in higher education is broken and needs fixing. I think he's probably right about that. One of the things he talks about, and I'm going to draw on his work a bit here, is that we keep measuring, measuring, measuring, we keep testing, because that's what we do. And we need to make sure people know things. I landed in Sydney last night, and I'm rather glad the pilot didn't have to whip out a textbook and go, "How do I land this plane on a very short runway?" So we do need to do summative and quantitative assessment, and people do need to remember things. I'm good with that. Facts are okay.
But he says perhaps what we need to do, in the realm of employability particularly, is to make judgements. Formative judgements—using qualitative information sometimes, but making judgements. Because, folks, the table you sat around for your last interview, people were making judgements. You probably know people in this room; you make judgements about them. You're making judgements about me now. You can't see my academic transcript; you hope I've got one somewhere. Well, I'm pretending I have—I have. But most of the time we make judgements based on performance. You know in your school or department, don't ever try and work with that person. Don't sign up to do a group assignment or job with that person, they'll disappear, but they'll be there to collect the trophy. You know these things; that's how the world works, obviously. So he says we need to do more of that.
[Slide—"bottom up". Academic at the top, separated by dotted line from boxes representing students. On left, "marks, grades, credits"; on right, "measurement" 50-per-center students scattered throughout the group.]
He says we need to do a bottom-up approach—his words—and that is this, and I'm paraphrasing his words. We have invented a currency; it's called marks, grades and credits. Our students are clever; they know that's the currency that works. It's the only currency they'll trade in. Which is why they sometimes ask questions like—shall we have some audience participation? What's the question? "Excuse me, is this on the exam? Is this assessable?" "Well, no, but—" "Okay, bye!" Not always, but often.
So how does that work? This is your set of units; there's your 50 per cent. You get enough in the bank and bingo they walk past you at graduation and you go, "How did that happen?" Because quite seriously I've taught people at all of those levels, but say in first year, one of my jobs at a previous institution was to teach communication skills, academic literacy, if you like. I was an old-fashioned kind of person; I wanted them to be able to write a sentence before they got the 50 per cent, and sometimes it took a few goes. And sometimes they were actually doing the unit just before they graduated, and I was really desperately trying to help them do that kind of standard thing that I thought was important—call me old-fashioned, but—it's university. Of course, you probably know the story: the student was struggling, they weren't going to make it, they were on their last unit before graduation, their parents were flying from another country to come to—is this sounding familiar? "Do something, Beverley." "Oh, okay!—No, I won't. I still think they need to be able to do that."
But that's that game. It's what he calls the bottom upper game. But also, we have this 50% thing—I'm just using 50% as code for whatever the pass mark is. There is a pass mark usually, not always. You may have to do something else as well. But what we often don't really decide, or define, is 50% of what? In the unit called Bridgebuilding 100, which 50% did you not get? In an outcomes-focused agenda, which is where most of us are going these days, it's difficult because we still use the term 50% or whatever—we generate a mark, we translate to a grade, and we tend to work in the grade, am I correct in that? Is that how it's happening? We don't often say, "What are the minimum standards you need here to get through this unit?" Now, I just jokingly told you one, which is "I want you to be able to write a sentence. You know, as a High Distinction student, you have to be able to use a semi-colon." That's me. But we actually do have standards.
There's a lot of talk around the sector and has been for the last few years, which is why we've got TEQSA, that we don't have any standards. Yes we do. Every person who teaches in a university has a standard. You do—you've got your private standard. The problem is, we haven't had a public conversation about those standards. And you may well know this story, again, going back to my own experience: Student comes to me, second time in the unit, "Look, I just can't seem to pass this unit. You've marked my essay"—it was last century—"You've marked my essay. Look, I can't—but in Unit X, I'm getting a Distinction." "Oh, that's interesting. Bring me one of your assessments. Okay, so the other person is going, 'Tick-tick-tick-tick-tick-tick, can't write a sentence but I get the idea—that's good enough. That's my standard. I'm marking on content.'" So we had a standard, but we didn't agree on the standard. So you see the problem. And, of course, students will learn to play that game. And sometimes they'll play us off each other.
So, maybe we need to change the game. Or maybe the game has changed anyway. The whole world is changing—and I'm going to do a quick snapshot to various international places—around this stuff, and standards, and where it's come from.
[Slides:
- UK Quality Assurance Agency for Higher Education website: Subject benchmark statements
- Tuning Educational Structures in Europe website
- US Association of American Colleges and Universities web page: Value Rubrics: The Essential Learning Outcomes
- US Qualifications Profile]
We have the rise of the evidence based culture in higher education right around the world. If you go to Australia...
[Slide: Australian Qualifications Framework, Senior Secondary Certificate of Education:
- Certificate I
- Certificate II
- Certificate III
- Certificate IV
- Diploma
- Advanced Diploma/Associate Degree
- Bachelor Degree
- Bachelor Honours Degree/Graduate Certificate/Vocational Graduate Certificate/Graduate Diploma/Vocational Graduate Diploma
- Masters Degree
- Doctoral Degree]
...basically, there's kind of 7 things that most universities say most of the time, that they want to promote. And that's them.
[Slide: Capabilities: communication, thinking, problem solving, information literacy, self-management, teamwork, civic engagement. See ALTC Good Practice Guide: Assuring Graduate Outcomes]
That's my rough mapping of graduate attributes from 38 universities in May last year, and as some of you may have heard me say before you know you're middle aged when you do that on the weekend, and you enjoy it. [laughter] Something to look forward to.
So if you want to know what the graduate attributes are most of the time in most universities, folks, guess what, you could have called them out, and that would have been the interactive part of the session. K to 12, kind of the same; the VET sector, kind of the same, the Australian Business Council, they sort of say the same. You get it—anyone want to argue with the list? Not much. It's said more elegantly elsewhere but that's good enough. If you want to read the whole story, there's a good practice guide, now on the OLT site, that you might like to have a look at. It's actually how I generated that table. And who's doing what, as well as all the projects in that area. In the UK, you may know about subject benchmark statements—don't bother reading the words, it's just a bit off a website, but that will give you the idea. They started that several years ago. In Europe there's the Tuning process, you may be aware of that as well. What I will do is give you the slides with URLs on them, so you can have them and click through them. And in the US—and I want to spend a little bit of time on this—there's a group called the Association of American Colleges and Universities. it's a group that you subscribe to as a university or college. It's quite large, it's based in Washington. They have a project called VALUE: the Validation and Assessment of Learning of Undergraduate Education. And they've come up with rubrics, and it's what I've based my work on, and I know some of you have been doing that as well. They have a group of essential learning outcomes. You see the top of them there; apart from Knowledge they would look like our graduate attributes—in other words, generic skills. We need to make sure people have the generic skills, including the people skills, the eyeball contact across the table, in order to be successful. Just in passing, they also have the degree qualification profile, where they've mapped out what the standard is in a bachelor degree, what the standard is at postgraduate level.
This is the Australian Qualifications Framework; it's not really a chocolate wheel. But that's the bachelor degree level which is level 7. There are 10 levels, and the government, and others, and the sector, have tried to define what it is that you should do, because you know and I know that VET is in this space, higher education is in this space, private providers are in this space. A bachelor's is not necessarily a bachelor's is not necessarily a bachelor's. So they've tried to say "Well, if you're offering a bachelor's, this is kind of it. If you're offering a master's, this is kind of it." And that's very contentious in the international market, as you would know. If you look at the bachelor level, these are the sorts of things that are written down. It's very, very thumbnail.
[highlighted words on slide of AQF level 7 criteria: "analyse and evaluate information"; "analyse, generate and transmit solutions to unpredictable and sometimes complex problems"; "transmit knowledge, skills and ideas"; "self-directed work and learning"]
But you did have to get a lot of academics to agree to this, so I guess it was pretty tough. The underlined bits start to tell us the generic skills, so you can see there's some sort of mirroring with what we've been doing in the graduate attributes space for some time.
You may know also about the Collegiate Learning Assessment. Could you put your hand up if that's familiar to you? Let me just give you a quick indication of what this is, 'cause it's still on the radar. Okay, because we have such trouble measuring graduate attributes—and we've been telling the government this for some time, it's very hard to measure communication skills, teamwork skills, inter-cultural capability, blah-blah—we've been saying "Too hard, too hard, too hard," but they say, "Well, actually, we want to know." Too hard? Well, guess what, we've got an answer for you. The Americans use a test, a standardised test, called the Collegiate Learning Assessment test. We nearly had it instigated in Australia last year; it went off the boil; I think it's still on the back burner. So it may be coming to a place near you. Let me tell you a scary story. This is how it happens in the US. You get a group of first-year students in a course. That would be an unrepresentative sample of...anybody. Give them the test. There is nothing in it for the student; they don't need to try hard. I think they get paid in some universities. They actually have an ASAT score that they can...controllability with. And then you get, at the same time, a different group of people in the last year of your course. That would be a similar, unrepresentative sample of people, that you don't get to pick; they're randomly generated. So you get the average of their score, and the average of their score. Guess what happens next! You take that, from that, and you get this magic thing called "value add". Isn't that good? No, not really. That's how the CLA works. That's what on the political radar in higher education at the moment. Lots of people are very, very concerned about this. Collegiate Learning Assessment—keep an eye out for it. We'll see what happens.
So, let me go back to where I wanted to start—that was all just the prelim. Not really. My contention is, supported by the evidence in the field, that the achievement of the capabilities that count [for early professional success] is a key to graduate employability, if that's something we should be focused on. But it's got to be at the right standard. We see report after report after report from the Business Council of Australia and others, in reports named this: How does this—What do you think is in this report? Lifting the Standard of Teaching in our Universities. I think the title gives us a clue to what's in it. In other words, graduates graduates graduates don't have the standards. They can't communicate; they can't X they can't Y. Yes they can; it's just that your standard is not what we think the standard is and so on. So again, we haven't had the conversation.
Mantz Yorke's definition of graduate employability really works for me. And it's in that book, which I think also could be on your Kindle, if you want it [Learning, Curriculum and Employability in Higher Education, Abingdon, Routledge, 2004].
[Slide: "What is graduate employability? achievement of skills, understandings, personal attributes, capabilities that makes graduates more likely to gain employment and be successful in their chosen occupations to the benefit of themselves, the workforce, the community and economy."]
So, it works for me because it actually encompasses educational and the community aspects of how we try to change the world through university education; I particularly like the red box down the bottom ["to the benefit of themselves, the workforce, the community and economy"], because it's for everybody's benefit, it's to change the world as well. we call those capabilities all sorts of things. I tend to use the word "capabilities" because it's more encompassing, and there's also a good literature around it. Let me try some evidence here. This is some work I did with 4 universities a couple of years ago, and the 4 universities were Curtin, RMIT, VU and USQ, the University of Southern Queensland. What we did was we focused on Accounting at undergraduate level. That's usually the largest course in any university. We did some online surveys for these 3 groups of people: graduates of up to 5 years, employers who tend to employ new graduates from any university, not just those ones, and some teaching staff. We said to them, "Here are 14 generic capabilities. How important are they, and how well do new graduates from any university generally evidence them?" Here is a snapshot of what we got.
So around the circle you can see the 14 usual suspects: Knowledge, Speaking, Thinking, you get the idea. Don't worry about which ones are which. The three lines are the different voices. Regardless of which ones they are, you can see from the pattern that most people said, "This is what's expected," in other words, this is what you need for success in this profession. Interesting, isn't it? Values and ethics was probably a bit higher than I thought Accounting might be, but anyway—that was a joke! Sorry! Any accountants in the room? Good. We'll keep going with that joke. No we won't. So just get that sort of pattern in your mind. You see which is the more important, or which ones they say were very important more often. But if you look at what's evidenced...
Expectation? What's evidenced. So what is kind of the standard out there? The pattern is really quite different. Now, this is not to say that Accounting is terrible. There's a lot more under this; there's a fully refereed article that you can go and read the whole shooting match in, but these are just the headlines. The difference is—ignore the blue line, that's the graduates; we were asking them a slightly different question. We were asking them not how well they evidenced these things but how well did your course prepare you in these things? So it's a different question. But the red and yellow are telling us something from academics and employers—let's be fair, though, that's probably a bunch of middle-aged people like me, who went to university in a different century, and maybe have a standard in their head, that...whatever. But what it's telling us is, the general headline: well, they're really good at ICT. Not much else. Now maybe that's a stereotype—who knows? But it's a piece of evidence. It's interesting. You know, we don't universally collect that, I don't think, systematically or systemically, in many universities. And that's gold. I don't know about you, but the universities I've worked in have an Alumni Office. They have a piece of gold for us. They could tell us, "How was it for you when you actually got out there?" But of course what we keep the alumni for is that when they're rich and famous they might give us a donation. Oh, sorry, that was just the ones I've worked in.
So this is the model that I'm coming up with and it's on a website which I'll give you the URL for at the end.
The model is—and you don't need to look too much at the detail—that this is just a quality assurance cycle, so that we can be more assured that our graduates have the capabilities that count. So it's for curriculum enhancement. So what we should do—and it's not rocket science—is be very clear about what the capabilities are. Well, we're probably pretty clear about those, but what standard should we have? Where are they in the curriculum? Rather than the silos of units and units and units, why don't we get a high-level pie chart, graph, whatever? This is what people are doing these days. You may be doing it here. Then we need some evidence, evaluating the outcomes, and then we plan the enhancements. There are 6 points, really, on the circle. And if you looked at closer words and—don't try and read it now, but it's all in detail at a website, coming to you soon—the one up the top ["Expectations—Determine the capabilities and standards for early professional success"] and the one down here, 4 and 5 ["Evidence—Evaluate achievement of capabilities in portfolio approaches to teacher, self and peer assessment" and "Evaluate outcomes; Analyse indicators of course quality including perceptions of graduate achievement of capabilities"] are the ones that I'd like to spend a little bit more time on.
So basically, it's about being clear about the expectations, and then being clear about the evidence. As you see, I was an English teacher once, so I do like alliteration: evidence, expectations. So let's talk about those for a start. A little word about standards. We're a bit confused, I think?
Sometimes when we say "standards", we mean the must-haves. Must have communication skills, must have teamwork skills. That's kind of where we've got to with graduate attributes in most cases, these days. No one's going to argue that a pharmacist needs communication skills. Very few people, though, have gone to the other side—that is, how well must they communicate? Let's go back to my anecdote, "must be able to write a sentence". That was a how-high [outcome]. So we haven't been very good at the how-highs, and that's where we're going now.
So, here's one way of describing your graduates. And I'm talking about a whole course or program here. Wouldn't it be good if we could say to our students, "When you sign up for this course, you're going to be this kind of graduate"?
[Slide—"Expectations" Three standing figures with hands pointing up ("Rock star"), out to the sides ("Good enough") and down ("Not yet")]
Well, you want the Rock Star?. If you want to be a Rock Star graduate, a Rock Star accountant, a Rock Star pharmacist, you'll need to be this; you'll need to be able to show evidence that you can perform at this level in these capabilities. If you just want to be Good Enough—that would be the 50-per-centers—this would be the minimum standard below which we will not let you out of here. And then, if there's another level of performance, that's called Not Yet. In other words, you're not there yet. Now that causes all sorts of problems if you've got a very large cohort and you need the 50 per cent to divide the passes from the fails. This would take some work, but it would help us get the standard.
Imagine yourself at a dinner party. You're sitting at a table not unlike the one you're at now except that the food is a lot better. Someone says, "Oh, you're the course leader of X at UNSW—we've got one of your graduates. Guess what they can't do?" That's what you would put in the middle column. What is the minimum standard. Guess what they can't do. Guess what they don't know. So, that's kind of one way of doing it.
[Slide—Table of "Capabilities" Column heads: Profession, Course, Minimum achievement. Row heads: Communication, Thinking, Problem Solving, Information Literacy, Self-management, Teamwork, Civic Engagement. All cells ticked.]
So, if we go back to those capabilities, what about at a program level, which is where I tend to work more, we actually said there's the usual suspects, they're the graduate attributes, contextualise those to your professional accreditation competencies or requirements if you have them, derive a set of course learning outcomes, and then actually specify a minimum standard. Not easy to do, but people have done it, and I'm going to show you some examples.
So, as part of the fellowship late last year, I started working with course teams around the country—53 of them, actually. And we started using this rubric.
Now, a rubric is really just a grid. If the word "rubric" annoys you, just think of it as a matrix or a grid. Or a table. Across the top—we drew on the American Value Rubrics, because they've had a go at this, we thought, "Why start again?" So we used that as a template, and all the templates have been Australianised at the website that I keep referring to. This is one particular one on writing standards. Now, the ones produced in Australia didn't really end up looking like this; it doesn't matter. It was, in other words, put a template in people's hands; let them change it to suit their course. The general idea is, you may want to specify 5 levels of capability. Across in the purple there, I've put the Dreyfus & Dreyfus levels of novice-to-expert capability [Expert, Proficient, Competent, Beginner, Novice]—you don't have to use that. But the idea would be, what 's the Good Enough? What would be the Rock Star—or the Super Rock Star? How do you describe, so a student clearly understands, "You're not quite there yet?"
So, in the middle column [Competent], you would actually specify the minimum standard. It's very important to understand this is not a marking rubric. Rubrics can be used to mark students, where you put, "It's an essay: 5 per cent for that, 10 marks for that," whatever it is—this is not that. This is a descriptor. And it's at the whole-degree level, and it gives us all something to pitch to, so if you are the unit chair or coordinator of that second-year unit—and I'm just being generic here—and they're your pieces of assessment, when you go to mark them, you've got something to pitch your levels to. This would be a minimum standard.
So course teams have started doing this around the country.
[Slide: List of templates: Written Communication, Oral communication, Critical Thinking, Inquiry and Analysis, Problem Solving, Creative Thinking, Quantitative Literacy, Information Literacy, Reading, Lifelong Learning, Integrative Learning, Teamwork, Civic Engagement, Ethical Reasoning, Intercultural Knowledge and Competence]
There are 143 of these in four large discipline clusters, available at that website [boliver.ning.com]. So 51 course leaders have worked with their teams. Now, that's where we've got to so far. They've generated them all. They look a little bit different, and they've come up with their own capabilities, but at least they've had the conversation. And I don't think it would take that long to have a conversation about what is the minimum standard. So folks, if your table is the course team for the Bachelor of XYZ, maybe over something really simple like a sandwich, you could actually have the conversation and sketch out—you don't need to make it the Encyclopedia Britannica. But I would guess, for many courses in many disciplines at many universities, this conversation hasn't been had. I know at Curtin I did a lot of work in graduate attributes, We even printed a bookmark, which meant, tick, they were done. In other words, it's advertising. "Come to Curtin; you get these things." "Well, how do you know, Curtin?" "Well..." [crosses fingers] And that's no reflection on Curtin. That's where we all were a few years ago. People have moved on.
So, how about your job of setting those assessment tasks in the unit?
[Slide:
setting those assessment tasks...?
assessment
- an opportunity to demonstrate your standard of performance
- an opportunity to create a professional artefact
feedback
- an opportunity for mentoring
- an opportunity to enhance your readiness for the profession]
How do you go about doing that? Can we have a different conversation about that? Maybe we should change the language? If you are producing or making a contribution in your unit towards the enabling of a graduate to enter the profession, even in an arts course, and I say "even" because I'm a literature graduate, and I have a job, because in arts, and even the less professionally defined courses, the students still want to work and probably in the professions somewhere.
What about if we change the language? Instead of "assessment", we talked about opportunities to demonstrate your performance towards becoming the professional? So instead of a piece of busy work like a multiple-choice test, short-answer test... You know, you did them yourself. What have you done with them since. Well, nothing, because they're in the bin. But I've kept a pile of mine—I don't know why. I don't keep much, but I have got a little box of essays, all about Dante's Divine Comedy and Petrarch's poetry, because Italian literature was my major, and my Honours thesis. They are really not much good to anybody. They don't really showcase the capabilities that count that got me the job I just went for 6 months ago. Sure, it taught me to think—I think essays are fantastic for getting you to develop an extended, evidenced argument in writing, a complex extended argument. Fantastic. Do you need to write a hundred of them? Because in an arts degree, when I was teaching in that field, that's kind of what we did. Now, I would have to think very differently about that now.
Could we talk about feedback differently? Could it be an opportunity to get mentoring and an opportunity to get some feedback on how you can get to the Rock Star status, or the Good Enough status if that's where you want to go? So when we're delivering on expectations, if we were clearer, maybe that would help us.
Could we even strip things out? Could we change that picture? Do we need 4 assessments in every unit? Could we have one? This is a conversation my current university is having. Let me put it to you: if you are the collective of the unit coordinators of, let's say,. first semester, second year, do you need to do the 16 pieces of assessment? Could you just have one? Or could you have seven? Do they all have to be at the end? If it's an exam, could it be in the middle? Could you have a viva? Could you actually have a panel of 5 people, with an employer, with a graduate, get the student to come to you once a year? I'll bet it would be less time. I'll bet it would be more valuable, if you think of your own education. Would that have helped you, or did 17 out of 20 kind of do it for you? Well, it depends on the game, of course.
So, just moving on, finishing up, Mantz says [Mantz Yorke, in Grading Student Achievement in Higher Education] we should ask students to tell us "How have you satisfied [through your work, the aims [and standards] stated for your course]?" So it's not all about us, it's about them. It's their responsibility.
[Slide—"top down". Academic at the top, smiling, separated by dotted line from boxes representing students. On left, "learning outcomes"; on right, "judgement" students marked "e" scattered throughout the group.]
And maybe we could do what he calls the "top down" approach—I don't know why he chose those words, but his point is this: Set the standard, get the students to gather the evidence. That's what the "e" stands for, it's not what you thought it stood for, it's "e" for "evidence". This is a focus on learning outcomes and making judgements. So the student can say...
[Slide—Rock star/Good enough/Not yet students again, "I meet these expectations and here's my evidence"]
..."This is who I am. I met the expectations, and here's my evidence. Here's something I can hand across the table when I'm at an interview and say, 'Actually, yes, I have done this,' Or, 'Well, actually, I worked in a team, we tried this thing...'" And so on.
So, eportfolios—I know some people are talking about those today. This is the one we built at Curtin. We called it the iportfolio, because when you put "i" in front of things, you make lots of money, and you sound really cool. No, actually, it was because it was all about you. So we focused on the graduate attributes here. And we asked the students to rate themselves, in a very, kind of, iTunesU, groovy kind of way, about how good they were at those. That was a great idea, and it actually popped up this little graph for you.
The ratings, the yellow line was the student's self-rating, the grey one was the aggregated rating of all the peers, mentors and teachers who'd given feedback as well. It was a great idea, just didn't work. Why? Because that list of graduate attributes was—I mean, it will work, and it's still a work in progress, but—the graduate attributes written like that? They didn't mean anything to the pharmacists. Or the engineers, or the—they were, like, "Mnyeah, I don't get it." And also, we didn't really specify the standard well enough. So we didn't say—we kind of gave them a thumbnail, what's 3 stars, but we didn't give them enough. We could have married it with this idea and it would have worked better, I think, and maybe that's something for the future.
So, this is what you get to do on your wild weekend of middle-aged fun. You can sit around and assess people's whole eportfolios—not. Don't do that. Don't even ask the student to show you.
I hope you noticed my sartorial splendour this morning. Right? I picked these pieces, put them together to make my presentation literally. That's what you do at the interview table. You actually don't invite the person to your wardrobe. This is my wardrobe—not. But as you see, this is all the evidence that I'm a fashion icon—it's just a picture off the Internet. The analogy is, we should never have to rummage around in a student's eportfolio. That's their bottom drawer. It's up to them to pull it out and give us the headlines and the evidence.
So, employability is about the what's-in-it-for-me factor, I think. For me, if that's what counts to the student, that's the hook, and we might even sneak up on them and educate them as well. I like David Boud's work. If you haven't seen Assessment 2020—but I'm sure you reference it in your toolkit—I think it's really, really good. It's a set of principles and 7 propositions put together by a whole group of people, some of whom are sitting in the room, about where would we want assessment to be. It's my guiding light, certainly in my new role. So I guess what I'm hoping for is...
[Slide—"Shift the needle" Dial with needle moving from red, through yellow, to green. At left (red), "Less on marks/grades/credits". At right (green), "More on graduate learning outcomes". Below, "Reward me for collecting, creating and sharing personal, portable, digital, warranted evidence of capabilities that helps me achieve my life goals"]
We need to actually shift the needle a little, to move a little bit less, focus a little bit less, on those marks, grades and credits—we still have to do them, that's the system. But we should actually maybe just move it a bit, so that this is the motivating factor. If I'm a student, reward me for collecting, creating, I could say curating and sharing personal, portable, digital evidence, because my employability will be linked to some website somewhere, even if it's LinkedIn or, whatever it is.
That's how I think we can assure standards in the capabilities that count.
That's the website I've been threatening you with for some time [boliver.ning.com]. When you go there, it's a Ning site. You have to join, just so I can keep the spammers out. There are, as of today, I believe, just about a thousand academics in there from around the world. It's a community of practice and scholarship, anyone interested in those topics. It's a social network. What I've learnt from doing this site is that we love to lurk. I try to get some activity happening in there, people join up, they have a good look around—I've probably put too much content in there—but no one ever interacts with me or answers me. But you could change all that. If you haven't used Ning before, you can actually get one of these yourself, they're a couple of hundred dollars a year, which is pretty cheap when you get a space that you can use, form groups, you get your own page in there—I know there are lots of sites like this, but I found it a really good tool.