Gastón Milano, CTO, Globant Enterprise AI, says as AI systems become more conversational and human-like, organisations face growing risks around trust, accountability and decision-making.
Anthropic has announced its new Claude’s constitution, which seems to have reignited an uncomfortable question: should we be humanising AI at all?
The document outlines the AI model’s values, reasoning and behaviours. It acts as a blueprint of why Claude should behave in certain ways, embedding values into the model’s reasoning rather than its constraints, outlining four priorities: safety, ethics, guidance compliance and helpfulness.
TIME has reported it as the first time an AI company has formally acknowledged the possibility of AI consciousness and even moral status.
The idea of an AI model having morals and consciousness seems like science fiction. There are parallels here to the Star Trek episode The Measure of a Man, which put the question bluntly: Does the android Data possess consciousness, identity and the capacity for growth?
The episode didn’t resolve it, and neither have we in the real world, but when an AI company begins formally asking the same questions about its own model, it stops being fiction and starts becoming strategy.
There is a bigger industry moment here where AI models aren’t just built to function, but to feel. There’s a push for more human-like, conversational AI, for models to ‘speak’ with empathy and confidence to respond in ways that feel more familiar for users. That shift has benefits in terms of keeping consumers engaged, but it also carries risks that few organisations are prepared for.
The risk of humanising AI could encourage the emotional attachment to machines and the gradual long-term erosion of critical thinking and accountability.
Psychoanalysing too much trust
Recent industry research shows that 88% of organisations report regular AI use in at least one business function, compared with 78% in 2025. Convenience and usability are the biggest drivers of that adoption. A tool that feels intuitive is easier to scale. Natural language interfaces lower barriers for non-technical teams, and conversational layers make complex capabilities more accessible.
But humanisation introduces a problematic psychological shortcut; people tend to associate fluent language with authority. Interactions with an AI system that communicates smoothly and confidently feels less like querying a probabilistic model and more like receiving advice from a knowledgeable figure. That is where risk quietly enters the system.
0% knowledge, 100% confidence
Generative AI uses data to predict patterns. It does not ‘know’ anything – it predicts what sounds right. Even as models improve, they still retain a margin of error.
This dynamic is most visible in hallucinations: AI-generated responses that sound credible but are incorrect or fabricated.
In 2023, New York attorney Steven Schwartz faced sanctions for using case citations that didn’t exist and were generated by ChatGPT. The citations looked real, the language was professional, the tone was confident but the content was fabricated.
As a consequence for using AI without due diligence, Schwartz was sanctioned US$5,000 and later told the court: “It just never occurred to me that it would be making up cases”.
A 2025 MIT study found that AI models are 34% more likely to use phrases like “definitely” or “certainly” when generating incorrect information than when providing factual responses. In other words, the more wrong the AI is, the more confident it sounds.
Hallucination rates are falling, some benchmarks show improvements from 21% in 2021 to below 1% in 2025 – but for domain-specific tasks, error rates can still reach 10–50%.
When AI gets it wrong, it gets expensive
With consumers, an AI mistake may be inconvenient, but in enterprise and public sector contexts, it can be consequential.
Research estimates global business losses from AI hallucinations reached US$67.4 billion in 2024, with knowledge workers spending an average of 4.3 hours per week verifying AI outputs to prevent errors propagating through workflows.
Not all businesses have the time and resources to spend on verifying AI outputs, which is where the consequences take place and where anthropomorphic design stops being a UX choice and becomes a governance issue.
Building trust into the design
The solution is not to reject conversational AI. It is to ensure usability doesn’t replace transparency. Organisations should embed governance directly into the AI experience to make system boundaries visible. Practical steps include:
- Confidence signalling: to enable users to understand when outputs are uncertain
- Data provenance: recommendations are traceable and verifiable
- Clear usage constraints: systems are not treated as universal authorities
- Auditability: enterprises can evaluate performance and detect errors over time
For technology leaders, the question is whether enterprises will adopt it with the right controls. Three priorities stand out:
- Operationalise human oversight: review checkpoints, escalation protocols and domain expert validation must be embedded in high-stakes workflows
- Demand transparency by default: interfaces should make uncertainty very clear with confidence scoring and citations being baseline expectations
- Build AI literacy across the organisation: all employees, whether at every level or C-Suite, need to understand both capabilities and limitations of AI models
Sustainable AI growth
Ultimately, the question is not whether AI can appear human – it clearly can – but whether we are prepared for the consequences of treating it as such. As debates around AI consciousness and moral status move from science fiction into corporate policy, the real risk lies in confusing resemblance with reality.
Systems like Claude may be designed to reason with values and communicate with empathy, but they remain probabilistic tools, not conscious entities.
For enterprises, the challenge is to embrace the benefits of more intuitive, human-like AI without surrendering critical judgment to it.
The organisations that succeed will be those that refuse to mistake fluency for intelligence, and instead build cultures where transparency, oversight and informed scepticism guide every interaction.
In a world where AI increasingly feels human, staying clear-eyed about what it is – and what it is not – will define responsible and sustainable adoption.
