Generative artificial intelligence and assessment security
This article provides advice on how to improve the integrity and security of assessment in light of greater access to generative artificial intelligence (genAI) tools. Such tools can quickly and effectively generate responses to be used in assessment tasks. While we recognise that we cannot “design out” cheating within our subjects entirely (Bretag & Harper, 2017), this document offers potential tips and mitigation strategies. The assessment types noted here relate to the categories used in UOW’s COSMOS Course Management System.
Three approaches that subject coordinators might like to try include:
- Leave your assessment tasks as-is and purposefully reflect on any impact;
- Run your assessment tasks through an appropriate genAI tool and discuss the output with your students to demonstrate the issues related to the use of genAI; or
- Embrace genAI, for example, embedding the use of genAI tools (with explicit permission and appropriate acknowledgement) in assessment tasks.
Potential mitigation strategies may highlight where genAI tools have been used relate to the level of known capabilities as at March 2023. The rapid pace of change in this space will mean the impact of some strategies, such as the use of current news topics or recently published articles, will change or disappear in the future as the genAI tools learn, develop, and diversify. Therefore, we are currently in an experimental space until there is clarity about how the tools will be embedded in digital platforms such as search engines (e.g. Google, Bing) and productivity software (such as Office 365, Google apps).
Possible COSMOS Assessment Types:
- Quiz
- Exam
- Assignment
Potential risks to academic integrity and security:
- Unsupervised quizzes.
- Reusing quizzes without modification.
- Questions that exclusively assess the lower levels of learning taxonomies (e.g. Bloom’s ‘knowledge’ and ‘comprehension’ levels).
- Using questions from textbook publisher question banks.
- Reusing previously used quizzes without any modifications.
Potential tips and mitigation strategies
- Supervised in-class quizzes.
- Refresh quiz content at regular intervals.
- Question types that assess critical thinking, evaluative judgement, and higher levels of cognition such as ‘analysis’, ‘synthesis’ and ‘evaluation’.
- Use the textbook publisher questions as an inspiration or starting point and create different questions.
- Place a reasonable time limit on completion of the quiz, and/or time limits on answering each question if an online quiz.
- Randomise quiz question selection from a question bank (online quizzes) or have multiple versions of the quiz distributed randomly throughout the class (face-to-face paper quizzes in a tutorial or workshop).
- Adding to existing question banks to increase the randomisation and variety.
- Only show one quiz question at a time (online quizzes).
- Delay feedback on whether answers are correct/incorrect and release detailed feedback after all students have completed the quiz.
- Questions that refer to more recent topics or events (as of March 2023, ChatGPT is only trained on a data set up to September 2021).
- As of March 2023, genAI is not great with the accuracy of mathematical/calculation questions, particularly the ‘standardised test’ verbose style of maths questions and multistep logical inference (Bordow, 2023).
Possible COSMOS Assessment Types:
- Exam
- Professional Task
Potential risks to academic integrity and security:
- Unsupervised exams (on-campus or online).
- Reusing examination papers without modification.
- Questions that exclusively assess the lower levels of learning taxonomies (e.g. Blooms’ ‘knowledge’ and ‘comprehension’ levels).
- Closed-book online exams.
- Extended time take-home exams.
Potential tips and mitigation strategies
- Invigilated exams (on-campus or online) in line with approved UOW exam format options.
- Rewrite your examination papers at regular intervals.
- Question types that assess critical thinking, evaluative judgement, and higher levels of cognition such as ‘analysis’, ‘synthesis’ and ‘evaluation’.
- Open-book online exams must avoid using questions that elicit fact recall responses.
- Place reasonable time limits on exams.
Possible COSMOS Assessment Types:
- Assignment
- Essay
- Journal/Blog
- Proposal
- Reflection
- Report
- Professional Task
- Artefact/Resource
Potential risks to academic integrity and security:
- Single output-focussed task where students use genAI without permission to produce the written response.
- No plagiarism checking.
- No reference checking.
- Use of paraphrasing tools that limit plagiarism detectability.
- Uploading copyrighted material to a genAI tool.
Potential tips and mitigation strategies
- Tasks that involve students submitting drafts for feedback or show their work process over time, via set ‘check ins’.
- Tasks that involve reflection on student experiences (e.g., reflections on practicums or classes).
- Tasks that build on the students’ work from previous assessment tasks.
- Leverage established resources such as Turnitin to assist in identifying unoriginal assessment content (Rogerson & McCarthy, 2017; Rogerson, 2017).
- Careful checking of accuracy of references, as text-based genAI tools such as ChatGPT may fabricate or provide incorrect ‘peer reviewed’ citations, or scrape and use poor quality or ‘predatory’ journals.
- Writing prompts that refer to more recent topics or events (as of March 2023, ChatGPT is only trained on a data set up to September 2021).
- Writing prompts (i.e., prompt engineering) that require critical thinking and development of logical arguments (this type of writing task could also be changed to a debate style of task). Use of rubrics that allow assessment of the prompts and their sequence, not just the artefact that students produce.
- Replacing or supplementing a writing task with one that requires students to submit (or write about) another form of media, such as: an audio file or podcast; a video recording or presentation; or a drawing, model or diagram.
- If making a purposeful decision to allow genAI use, have students acknowledge, cite, and detail its use in completing the assessment (see UOW Library Referencing Guides).
Possible COSMOS Assessment Types:
- Presentation
- Proposal
- Professional Task
Potential risks to academic integrity and security:
- Oral presentations where students may read from scripts.
- Uploading copyrighted material to a genAI tool.
Potential tips and mitigation strategies
- Live viva voce-style presentation (e.g. question and answer with the audience/educator) tasks that involve questioning and justifications, as this avoids potential for students completing the entire assessment by reading from scripts produced by genAI. Redesign rubrics to increase the weighting awarded to student responses to questioning.
- If making a purposeful decision to allow genAI use (for example in preparation of a script or performance guide), have students acknowledge, cite, and detail its use in completing the assessment (see UOW Library Referencing Guides).
Possible COSMOS Assessment Types:
- Professional Task
- Assignment
Potential risks to academic integrity and security:
- Students use genAI (such as ChatGPT, GitHub Copilot, Codex) to generate, debug, and optimise code without permission.
- Uploading copyrighted material to a genAI tool.
Potential tips and mitigation strategies
- Tasks that involve students in learning the programming language are still important as explanations and code produced by genAI may contain inaccuracies.
- Tasks where errors are intentionally incorporated into genAI-generated code, where students are required to identify and rectify errors without using genAI tools.
- Incorporate live, viva voce-style explanations of coding decisions into assessment tasks.
- If making a purposeful decision to allow genAI use, have students acknowledge, cite, and detail its use in completing the assessment (see UOW Library Referencing Guides).
Possible COSMOS Assessment Types:
- Lab/Prac/Simulation
- Professional Task
Potential risks to academic integrity and security:
- Students use genAI (via digital devices or wearable technology) for unauthorised assistance during practical assessments.
Potential tips and mitigation strategies
- Ensure phones and smart watches are not within reach of students during the assessment.
Possible COSMOS Assessment Types:
- Lab/Prac/Simulation
- Professional Task
Potential risks to academic integrity and security:
- Single output-focussed task where students use genAI (such as DALL-E, Midjourney) without permission to product the artefact.
Potential tips and mitigation strategies
- Tasks where creative elements such as sketches/designs can be compiled into a portfolio to show stages of the journey towards producing a finished piece, with items annotated to provide insight into student thinking.
- If making a purposeful decision to allow genAI use, have students acknowledge, cite, and detail its use in completing the assessment (see UOW Library Referencing Guides).
How can I access support?
For learning and teaching support (for example assessment design and implementation):
Access the L&T Hub self-help resources
via the L&T Hub search
Request a conversation with an L&T specialist
(02) 4221 40048:30 AM to 5:00 PM Monday to Friday (AEDT)
ltc-central@uow.edu.auAdditional support and services can be found on the Support page of the L&T Hub.
Related information
- Moodle Quiz Questions | L&T Hub Article
- The role of quality assessment design in strengthening academic integrity | L&T Hub Article
- Timed online assessment tasks using the Moodle Quiz | L&T Hub Article
- UOW Library Guides: Referencing & Citing | UOW webpage
- Utica University Bloom’s Taxonomy Resource | External resource
References
Bordow, S. (2023, February 21). Do the math: ChatGPT sometimes can't, expert says. Arizona State University News. https://news.asu.edu/20230221-discoveries-do-math-chatgpt-sometimes-cant-expert-says
Bretag, T., & Harper, R. (2017, May 12). Assessment design won’t stop cheating, but our relationships with students might. The Conversation. https://theconversation.com/assessment-design-wont-stop-cheating-but-our-relationships-with-students-might-76394
Rogerson, A.M. (2017). Detecting contract cheating in essay and report submissions: Process, patterns, clues and conversations. International Journal of Educational Integrity. Contract Cheating Special Collection. 13(10), 1-17. https://doi.org/10.1007/s40979-016-0013-y
Rogerson, A.M. & McCarthy G. (2017). Using Internet based paraphrasing tools: Original work, patchwriting or facilitated plagiarism?. International Journal of Educational Integrity. 13(1), 1-15. https://doi.org/10.1007/s40979-017-0021-6
