Cláudio Lúcio, data specialist and founder of A3Data, on identifying and preventing AI ‘hallucinations’.
Would you trust a machine that, in the blink of an eye, can fabricate a convincing lie? In the current scenario, where Generative Artificial Intelligence (GenAI) and Large-Scale Language Models (LLMs) are on the rise, so-called ‘hallucinations’ – moments when AI generates false or disconnected information – are becoming an urgent problem that results in the question: how far can we trust these creations that shape the digital future?
The relevance of this topic cannot be underestimated, given that the reliability of these technologies is critical for their application in relevant industries such as medicine, finance, and law.
In the healthcare industry, for example, misdiagnoses can result in inappropriate treatments. In the financial sector, wrong forecasts can cause huge losses. And in the legal field, inaccurate information can compromise important court cases. Thus, understanding how to identify and prevent these failures is not just a technical necessity, but a matter of business safety and sustainability.
Recent studies published by Vectara, a startup founded by former Google employees, point out that AI hallucinations occur with a frequency ranging from 3% to 27% in simple tasks, such as summarizing news articles. This means that most of the metrics used to measure the quality of automatically generated writing, such as text summarization and translation, are not adequate to quantify the level of hallucination of that same text.
This is because these metrics (such as ROUGE, BLEU, and METEOR) evaluate textual quality based on similarity to human references, but do not consider the factual accuracy or veracity of the information contained in the text. Therefore, there are active research efforts to define effective metrics to quantify hallucination.
In this sense, there are several ways to minimize and avoid this behavior, including proactively. For this, it is necessary to have knowledge in the business area in question and in the type of task that the model is performing, and thus have governance, creating robust development methods and monitoring in real time.
To illustrate this action in a more simplified way, in the technological area we call ‘guard rail’ (which are those protective railings that are used on roads) the rules and limitations that are determined to prevent LLMs (which, in this case, would be ‘cars’) from ‘leaving the road’ and generating harmful, false responses or that violate some ethical principle.
The bill that is being processed in the Senate brings a very interesting concept, which is the TEIA, an acronym that summarizes four essential concepts for an AI governance model: ‘T’ stands for ‘transparency’, by requiring more clarity in AI decision-making processes; the ‘E’ for ‘explainability’, which allows users to understand the reasons behind the decisions made; the ‘I’ for ‘intelligibility’, which ensures that AI results are understandable to humans; and the ‘A’ for ‘auditability’, which allows you to verify that the systems are working correctly and that the results are being generated in a fair and impartial manner.
In this way, we have more guarantee of reliability and accuracy in decisions. Based on this premise, companies can establish clear policies on the use of AI, defining who can access and manipulate AI data and algorithms, and creating review committees to monitor and evaluate the performance of models, in order to ensure that they follow ethical guidelines.
Therefore, it is essential that these different market sectors invest in data governance and rigorous processes for training people with AI-related skills. In addition, it is essential to also include ‘human-in-the-loop’.
As the meaning implies, this expression is defined by the active interaction of humans and the integration of knowledge and experience at various stages of the life cycle of a system (which is largely automated), in order to improve the accuracy, reliability, and effectiveness of the system. In addition, it is essential to include the active interaction phase of humans, integrating their knowledge and experience into various stages of a system’s lifecycle, largely automated, in order to improve the accuracy, reliability and effectiveness of the system.
In summary, the vulnerability of these areas of activity requires a proactive and expert approach to ensure the safety and effectiveness of AI systems and, above all, the maximum integrity of information for users.
