The wait is over! Turnitin’s answer to AI-generated texts is now enabled, starting April 4, 2023. The tool is called AI Detection Tool.
Beginning 4/4/2023, Feedback Studio Report has been expanded to display the Similarity score as well as the new AI indicator score.
The AI indicator score shows an overall percentage of the document that AI writing tools may have generated.
Click on the AI indicator will bring up the AI report in another Browser tab.
- Turnitin has rolled out its “AI Detection Tool” on 4/4/2023.
- Assignments with Turnitin enabled (either by Framework or External Tool) will be able to see this Tool on 4/4/2023.
- The AI Report will be available to instructors and admins. The AI report is NOT available to students at this time.
- The tool cannot be turned off or customized at this point.
Minimal disruption on faculty workflow is expected.
- No change to how faculty create their Turnitin enabled assignments.
- No change to how students submit their assignments.
- No change to how the Similarity Score is currently being presented in Canvas.
- The only major change is, faculty will have additional information from the AI Detection Tool to help them determine if and to what extent an AI generator was used to complete the assignment submission.
Additional Questions and Answers
Turnitin’s AI writing detection model is trained to detect content from the GPT-3 and GPT-3.5 language models, which includes ChatGPT. It is not trained on GPT-4, Google’s Bard, nor Facebook’s LLaMA.
“The percentage indicates the amount of qualifying text within the submission that Turnitin’s AI writing detection model
determines was generated by AI (with 98% confidence based on data that was carefully collected and verified in a
controlled lab environment).”
“This qualifying text includes only prose sentences, meaning that we only analyze blocks of
text that are written in standard grammatical sentences and do not include other types of writing such as lists, bullet
points, or other non-sentence structures.”
Turnitin only flags something as AI-written when it is 98% sure it is written by AI.
In other words, Turnitin errors on the side of minimizing the likelihood of false positives. By achieving 1% false positive rate (incorrectly identifying fully human-written text as AI-generated) in a lab situation, it results in 15% false negative rate (incorrectly identifying fully AI-generated text as human-written).
“For example, if we identify that 50% of a document is written by AI, we are 98% sure that at least 50% is written by AI
with a less than 1% false positive rate, but it could contain as much as 65% AI writing.”
Let’s watch their explanation on false positives.
Despite the claim, preliminary reports have casted doubt on the accuracy of the tool in real life application, especially when the submitted texts have been modified by humans or paraphrasing bots. (We tested a new ChatGPT-detector for teachers. It flagged an innocent student. Washington Post)
The Similarity score and the AI writing detection percentage are completely independent and do not influence each other.
The Similarity score indicates the percentage of matching-text found in the submitted document when compared to Turnitin’s comprehensive collection of content for similarity checking.
The AI writing detection percentage, on the other hand, shows the overall percentage of text in a submission that Turnitin’s AI writing detection model predicts was generated by AI writing tools.
English. For the first iteration of Turnitin’s AI writing detection capabilities, we are able to detect AI writing for
documents submitted in long-form English only.
Check out the AI Writing FAQs (PDF file). You may download the file or read it at the bottom of this page.
Check out the Turnitin’s AI Writing Detection Features Guide (PDF file).
Visit AI writing & Turnitin.
Some concerns prior to its launch: Can Turnitin Cure Higher Ed’s AI Fever? (April 3, 2023)