.png)
AI, especially ChatGPT, aids many in tasks from coding to writing emails. This is a problem for tech recruiters who want to assess candidates for certain skills, such as programming or written expression.
Fortunately, there are signs that can help recruiters detect if candidates are using ChatGPT. We must confess, though, that writing this article has proved as hard as predicting tomorrow’s $BTC price. ChatGPT so-called “outsmarters” are constantly showing up — and continually toppled down. In September 2023, even OpenAI themselves admitted that AI writing detectors don’t work.
This article compiles telltale signs of ChatGPT usage and debates its ethical implications in professional settings. These are the best ways to detect ChatGPT text in candidates for developer roles:
It’s official: Tools for detecting AI text don’t work any more (and might never have)
AI detectors don’t work. Those are OpenAI’s words. But this fact wasn’t always as clear as it is now.
As soon as ChatGPT dropped in late 2022, plenty of tools also using ChatGPT sprung up, claiming they could uncover AI-written prose. The truth is, by anecdotal evidence, small-scale studies, and all-out press releases, most, if not all, of these tools don’t work properly.
Especially when dealing with newer versions of ChatGPT, these tools are not reliable or accurate. In fact, even OpenAI admitted that their own AI Text Classifier, which was launched in early 2023, was ineffective; they discontinued it in July 2023. Yes: OpenAI stated that the tool — their own very tool — had been put down because it was inaccurate. That’s why we opted out of featuring tools such as GPTZero, GPT-2 Output Detector and DetectGPT by Stanford University. We’ll be opting for a more organic approach instead.
Seeking patterns in vocabulary, structure, and up-to-date knowledge
Large language models come in various forms, and one of the most easy-to-access is GPT-3.5, which is the technology running behind the free version of ChatGPT. Suppose you ask a GPT-3.5 engine to provide tips and tricks for detecting GPT-written content. In that case, it tells the following:
- Sophisticated vocabulary: Look for applicants using uncommon words in their responses that may be beyond their usual vocabulary. For instance, “I’m greatly proficient in the domain of crystallising intricate predicaments, especially in JavaScript.” This statement is a bit wordy, which can tell an AI wrote it.
- Lack of personalisation: Chatbots like GPT-3.5 might struggle to provide genuinely personalised responses. If an applicant’s answers seem generic and lack examples, it could be a sign of AI involvement.
- Contextual understanding: If an applicant displays an in-depth understanding of a topic across different questions or conversations, it might be worth investigating further. For instance, a candidate maintaining a discussion about complex programming concepts related to infrastructure without backend knowledge could signal AI assistance.
Usual American spelling: Even if the European Union standard is the British spelling, which translates into most Europeans writing their correspondence in British English, ChatGPT might churn out cover letters in American English, with plenty of ‘z’ characters and next-to-none ‘u’ letters. After all, OpenAI is a California-based company.
Example:
However, these tips are not enough to spot GPT-written content, as the engine can also adapt to different styles and contexts; of course, it can write in British English or Oxford spelling if requested to do so. Here are some more patterns that you should look out for, especially if ChatGPT is poorly prompted:
- Paragraph starters: If you notice that the applicant’s paragraphs start with similar words or phrases, such as “However,” “Moreover,” “In addition,” or “Furthermore,” it could be a clue that they are using ChatGPT to generate their text. GPT-3.5 tends to use these words to connect sentences and paragraphs, but human writers usually vary their transitions and introductions.
- Word repetition: If you see that the applicant repeats certain words or synonyms frequently in their text, it could be another indication that they are using ChatGPT to generate their text. AI often uses the same words or synonyms to describe concepts or ideas, but human writers usually avoid repetition and use more diverse vocabulary.
- Buzzwords: If you hear that the applicant uses buzzwords or jargon that are not relevant or appropriate for the topic or situation, it could be a red flag that they are using GPT-3.5 to generate their text. Human writers typically use clear and precise language that matches the audience and purpose. ChatGPT might use words like “cutting-edge” to describe something stone-cold and old as Java.
- Large-language model disclaimers: Social media users from the scientific community have sworn that they’ve been reading academic papers that start with the phrase: “As a large-language model trained by OpenAI, I can’t…”, or even feature the “Regenerate Response” punchline. That’s how pervasive this clue is. You know where it comes from: it’s the disclaimer that OpenAI gives you when you’re asking for something that’s allegedly beyond the solution’s capabilities. You might want to Ctrl+F it on an applicant’s drawn-out cover letter and see where it sends you…
Example:
This letter uses the usual [brackets], talks about an “innovative approach,” and, moreover, doesn’t say anything to the point. It’s vague and general, apt for any cover letter for any job in the world. It mentions an attached resume, but maybe you haven’t asked for it yet. It’s a generic, ChatGPT offspring
Conducting behavioural interviews to compare
In a behavioural interview, the interviewee is asked to demonstrate their knowledge, skills, and abilities through hypothetical situations. Human resources professionals may use behavioural interviewing techniques to determine if the candidate wrote a cover letter or with a tool like ChatGPT. The evident downside is that conducting said interviews is time-consuming since it requires cooperation from said candidates, who should get back to the HR professional.
An HR professional can evaluate a candidate's ability to provide detailed and specific examples by asking questions about the skills and experiences mentioned in a cover letter. In this case, the odds are that the candidate did not thoroughly check their letter, and so it cites a JavaScript project made up by the AI, the HR professional conducting the interview could spot an incorrect use of the tool.
Is it essential to try to spot ChatGPT?
Should HR experts grapple with the downright complex challenge of detecting ChatGPT on a cover letter? Or should they accept it as a new part of the tech landscape and don’t mind it?
The answer is yes: it’s essential to at least spot it — and even if the job accepts it. If the job posting explicitly encourages candidates to use AI tools, then the implicit message is that the candidate should use them correctly. As such, a write-up that frequently uses buzzwords such as “cutting-edge” or “revolutionary” seems like a wrong use of an otherwise very robust tool. And, in the tech realm, HR experts should be mindful of candidates who don’t take the best advantage of the latest tech out there.
In the latest Stack Overflow survey, over 70% of programmers admitted they used AI tools, primarily to write code or run their code through it. Moreover, developers are constantly told off for being lousy communicators. So when a fantastic large language model tool drops to help them prompt their way into succinct, clear, amicable messaging, employers who encourage ChatGPT can expect to see a hike-up in their Slack quality. ChatGPT can help developers elevate their written communication skills and express their ideas more effectively.
Spotting ChatGPT is not part of a blame game or a cat-and-mouse chase. Rather than that, it is part of examining a recruit — verifying if a candidate is correctly leveraging readily available tools.
Partnering up for success in tech recruiting
Ethical or unethical, using ChatGPT to write cover letters is a game-changer for developers on the hunt for a job, as it can save them time and boost the quality of their work. Either way, it's a trend that's here to stay. We reckon its use will go up not in the next few years, but week by week. Therefore, as HR professionals, being aware of the role of ChatGPT in the recruitment scene becomes fundamental. Acquiring the ability to tell apart AI-generated texts from human-written ones can be a useful skill. Or at least, it can be a factor to consider in our evaluation.
If you want to speed up your evaluation process, you can count on WeAreDevelopers’ Managed Recruiting service. This service gives you access to a team of experts who can help you navigate the changing recruitment landscape, including verifying candidates.
It’s about embracing and leveraging AI’s power, rather than being afraid of it or struggling with it. If you’re interested in learning more about how to deal with these new challenges, why not book a call with us?
