Incorporating GenAI in a Case Study Assessment

My name's Dr Jonathon Mackay. I'm a lecturer within the School of Business.  I teach subjects around management and supply chain management. In particular for this year I've been teaching Management 110/208, which is the mandatory management subject for undergraduate business students.

There was a lot of talk early on before Autumn session began about ways of trying to design assessments in such a way that doesn't treat ChatGPT as a potential source of problems, but rather tries to embrace it and look at how students can gather an appreciation and understanding of how to use this software for them when they leave uni to know how to apply different sorts of generative AI within the workplace. So within a lot of these conversations with colleagues it was about how can we navigate this space to sort of get two birds with one stone.

Design assessments in such a way that we're minimising the risk or almost reducing the incentive to use ChatGPT or other generative AI, whilst also not just treating it as this pariah, but embracing it and looking at how we can actually design assessments in a way that gets students to critically analyse the role of generative AI within workplaces etcetera.

Ultimately, the aim of what we wanted to achieve by using ChatGPT was to get students to improve not only their own competencies of learning new technology, like learning how to use the prompts within ChatGPT, but more importantly being able to analyse the types of responses that are generated and to critically evaluate and think about, well, what are the benefits from at least within our discipline from from a management perspective, what are the benefits of them being able to use ChatGPT versus what are the pitfalls? And where do things such as their own individual experiences, their own individual biases, and those that exist within the generative AI? Where do they fit in to these particular types of issues?

We had a live lecture where I, in front of the entire class and in a pre-recorded video too, demonstrated how to use ChatGPT. So it wasn't something that was treated as an afterthought. We very much embedded it directly from the get go within the subject, within the subject outline saying for this assessment we are using ChatGPT. In terms of disclosure for the students, I think the most important thing is being upfront and honest. I said within the first lecture, these are examples of academic misconduct before we even talked about ChatGPT, I talked about plagiarism, collusion essay, uhm... purchasing essays etcetera, etcetera. Then talked about ChatGPT going OK, you can't use this to write your individual answers etcetera. However, we're going to be using it within the subject in these very particular ways.

So I was very clear in not just the subject outline, but also within the lectures and also the Moodle sites and all of the resources that the students got. That it was very clear about what they can use ChatGPT for within that assessment because it was within the structure they were given.

Now for the next part, use ChatGPT for this. So within that, the students were already under the assumption that OK, we're allowed to use it for this, but they're still subject to the rest of the Academic Misconduct Policy for the other parts of using the generative AI. And also within that subject at the time, there were quizzes. So multiple choice quizzes and a few weeks and I'd alert the students.

I'd say that there's a guest quizmaster this week and it's ChatGPT and I would develop the multiple choice quiz using ChatGPT and
then give it to the students to see how they respond to it. So what I did within that process is I took the transcript of the lecture and put it into ChatGPT and gave it a prompt saying develop 15 multiple choice questions based on this. So it was much easier for students to do because it was just about reciting information rather than necessarily analysing content per se. But I think the students really enjoyed that, and for them as well, and for myself and my learning team, it was demonstrating just another way that we as academics can also use ChatGPT. And that to me kind of made me realise a different angle towards this approach, where it's not just about how we can use it within the assessments, but also how as educators we can use it within the way that we design subjects.

In terms of teaching the students around the use of ChatGPT, we had an approach where I would demonstrate to the students within a live lecture that was recorded, but also then there was dedicated time in their tutorials for them to learn how to use the software. Some of the basics around signing up to have an account for it, but then there was time for them to have a bit of almost free play with it. Where we, the tutors will give them prompts, but the students were encouraged to just familiarise themselves with it, whether it be, you know, getting it to develop, I don't know, the a weekly recipe list or anything that was up to them, but just for the students to get familiar with asking the questions. So when it came to the assessment, they were already well and truly familiar with the software. It's incredibly easy to use. It's just about signing up and then putting the prompts in.

I think for us, looking at how the original ideas behind the project, versus how it actually manifested and unfolded within the students, there were a lot of benefits. The biggest benefit was students were easily able to sit down and go, okay, this is our response. This is the ChatGPT response. These are the similarities, and this is the differences. In terms of the feedback from the students, they enjoyed the experience of learning with it, and not only just in terms of it being part of the assessment, but also helping them understand its benefits for themselves in their academic and future professional careers. Students responded really positively to it.

What I found within a lot of the presentations and assessments done by students was that they realized how much inherent bias we all as individuals carry, and that is not something that needs to be treated as something we need to get rid of, but as part of the innate beauty and complexity of us as humans, that we all are creatures who have degrees of irrationality, we're all creatures who aren't able to process the myriad of information that's around us, and that's what makes us unique.

Whereas what, you know, this process aimed to do was to not demonstrate that, okay, well we don't need something like management because ChatGPT can do it. As the students learnt, the AI is good at producing answers, but they're quite, they lack that personal touch. They are incredibly vague for this particular assessment, at least, they're very vague. So it demonstrates to students that it's not necessarily something that's going to replace, per se, humans in the future, but it offers a different tool. You know, the, I suppose, underpinning philosophy, surrounding my attitude towards all of this is viewing technology as a tool to reveal things rather than viewing as a supplement or a replacement to individuals.

We'd learned from the first session that students are, you know, happy and willing to embrace the AI process and learn how to do comparisons between what it says and what they do. And now we're gonna experiment a little bit more with looking at, well, what happens if we get them to think about the implications more so. So rather than saying, you know, here's question one student, here's question one, AI. It's the students have answered all of these questions, none of which involve the solutions to a particular problem. Then within this presentation, they're using the AI to generate the solution, and they're talking about whether they think it's appropriate or not appropriate to their particular case as well.

The advice that I'll give to any colleagues who are thinking about embracing ChatGPT or any form of generative ai, whether it's in their subjects or assessments, is to feel free to make mistakes. This is a learning process. I can guarantee the way that I implemented this was not going to be perfect, and it never will be perfect.

There's always got to be mistakes, but we need to take that leap. You need to take that leap of having a bit of creative thought to go, okay, well, what possible imagination can we have for using different forms of generative AI within our subject? And you'll learn by doing so. My advice really is to just jump in, see where it works. It won't work for every subject. It won't work for every assessment. There's no doubt about that. But the avenues where it will work can be used not only to teach students about it, and not only to reduce potential misconduct, but also to help you as an academic learn more about how you can embrace this particular type of technology rather than just viewing it as a pariah that we need to avoid.

We all have the opportunity to join the students on this journey of trying to embrace the positives that things around generative AI offer. I think for us as educators, we can't treat it as something that is viewed as only as a negative. I think as much as there is the increased risk, it's more so increased opportunity. Ultimately, I give the benefit of the doubt that our students are going to approach their subjects with a degree of academic curiosity.