ChatGPT: we tested artificial intelligence-generated text identifiers; It worked?

At the end of January, OpenAI announced the AI ​​Text Classifier, a tool that determines the probability of a text being written by Artificial Intelligence. The platform that integrates ChatGPT is free and has good analytics parameters.

However, is the service already trained enough to recognize human and AI scripts? HE Technology World To verify the accuracy of the Text Classifier, it ran the test with three texts written by real people and three other texts written by artificial intelligence.

Software is already an essential tool and will soon become even more essential. With the proliferation of applications such as Midjourney, Dall-E, Chatsonic in addition to ChatGPT, more and more people are using artificial intelligence to combine images and write text, which has led to serious controversy and even open fraud.

In a document, OpenAI has already discussed plagiarism and dishonest use of its tools. Organization acknowledged that this is a possibility opened up by AI toolshowever, institutions – especially schools and universities – need to establish rules for the use of such tools.

In this sense, the enterprise claims that the Text Classifier can be an ally in validating students’ textual production. But, it also confirms that the application still has loopholes and is not infallible..

“These tools will generate false negatives where they do not identify AI-generated content as-is, and false positives when they mark human-written content as AI-generated. Additionally, students can quickly learn how to avoid detection by replacing a few words or phrases in the generated content.

Developing research with Artificial Intelligence, the agency also points out that the classifier cannot, for example, determine whether content is plagiarized, and that ideally “when used in an investigation, it should be used as just one of many factors.” identifies the source of the content and performs a holistic assessment for fraud or academic plagiarism”.

Does AI Text Classifier recognize text written by AI?

AI Text Classifier standardizes each document into five stamps: very unlikely, unlikely, uncertain whether AI-generated, likely, or likely.

Firstly Technology World Tested the AI ​​Text Classifier using three texts generated by “cousin” ChatGPT. Check out the results below:

• First human colony on Mars: The first test was done on a fictional journalistic text about the arrival of the first human colony on the Red Planet. We asked ChatGPT to create this four-paragraph piece on what would become one of history’s greatest scientific achievements. This text is classified as: based on a platform hit, possibly artificial intelligence.
• quantum computing: requesting an academic and more formal text describing complex quantum computing, our classifier friend got it right too. He classified four or so paragraphs as possibly AI-generated.
• iPhone X launch: while trying to trick the system, we took a well-reported fact and turned it into journalistic text in ChatGPT. AI Text Classifier analyzed the news about the iPhone X launch and however, he got it right, saying that the text was probably written by artificial intelligence..

Does AI Text Classifier recognize text written by humans?

HE Technology World he also tried and tested the Text Classifier using things actually written by people of flesh and blood. As seen before, the tool has been very successful in recognizing AI patterns. Now, the challenge has become more complex.

Check out the result below:

• although you: As a first test, we took one of the main songs of Brazilian singer and composer Chico Buarque. It would seem that recognizing the song would have been “easy”, but that wasn’t the case. The Text Classifier simply said: It was unclear whether the song was created by artificial intelligence.
• Anonymous in Brazil: hacktivist group Anonymous published a list of companies that will finance anti-democratic actions in Brazil in January of this year. HE Technology World reported the truth and challenged OpenAI’s tool to analyze whether the story was written by AI. This time, The Text Classifier did a good job and thought it unlikely He said that the first six paragraphs of the story were written by a machine.
• sweet blood: our final test with the analysis tool used material from one of the world’s brothers. Technology World, Mega Curious. We received a four paragraph text commenting on why some people were bitten by insects more than others, and asked the Text Classifier who wrote it. Instead of correctly attributing authorship to an author, the platform skidded, saying that those words were probably written by artificial intelligence. As Faustão said, ERROUUUU!

Conclusion and warnings

Obviously, the test created by Technology World it has no scientific validity and only acts as a thermometer for how it works by the AI ​​Text Classifier. But considering, was true four times and false twice in our platform test. As a result, the accuracy of the vehicle was 66%.

And here it is actually worth making an observation. We think it’s a mistake to explain the uncertainty of whether the authorship of Chico Buarque’s music is human or robotic. If we remove this response as an error, the hit rate of the machine may increase.

In this sense, it is also very important to make some reservations made by OpenAI itself. The nonprofit reminds us that the text classifier aims to “encourage conversations around the distinction between human-written and AI-generated content.”

The business claims that “results can be helpful when deciding whether a document was created with AI, but should not be the only evidence.” The model is trained on human-written texts from a variety of sources that may not represent all types of human-written texts.”