We’ve been hearing for months that generative AI like ChatGPT, Bard et al. threaten a huge number of jobs, namely one of the professional groups that will be most affected are programmers. I actually came to read a statement that up to 90% of programmers may lose their jobs in the coming years due to AI capabilities.

Now there is an important question to consider, and that is if generative models are really comparable in capabilities to programmers, because it does little or no good for them to be able to generate code in a few seconds (compared to the minutes or even hours it would take a programmer) if that code later turns out to be incorrect. And the truth is that today it happens much more often than one would expect.

Rapier It has already been empirically verified by quite a few programmers, confirms a study conducted by Purdue University, Indiana. According to the study, reported by The Register, researchers collected 517 questions from the Stack Overflow website, which was created in 2008 and serves as a meeting place for more than 13 million programmers from around the world. Thousands of questions are posted daily on Stack Overflow and answered by the community itself.

So, ChatGPT answered 52% of queries incorrectly. This in itself is hugely problematic, but it’s made even worse by the chatbot’s great ability to elaborate on its responses. In other words, he is so persuasive that, like a good liar (although I obviously do not equate one thing with the other), he can convince his partner of the truth of his answer, even if it is incorrect.

There is something far more boring than programming and that is having to revise code written by others, or so many programmers have told me over the years. So if we find that what ChatGPT has to offer is not reliable, so a complete review is required, much of the supposed utility is lost. But it is true that there are specific tools like GitHub Copilot and that other chatbots like Bard have focused on this point, but even today it is clear that these models do not offer the required level of reliability for this type of task.