Why Avoid Generative Legal AI Systems? Hallucination, Overreliance, and their Impact on Explainability

Absztrakt

The central focus of this article is that the use of GenAI systems in the legal profession requires significant restraint due to the critical risks posed by hallucinations and professional overreliance. The main focus of the analysis is the concept of Generative Legal AI systems (GLAI), an umbrella term referring to AI systems specifically adapted to the legal domain, ranging from document drafting to decision-support tools in criminal justice. Unlike traditional AI systems, GLAI models are built on architectures designed for statistical token prediction rather than legal reasoning, frequently resulting in hallucinations in which linguistic fluency is prioritized over factual accuracy. Such hallucinations obscure the underlying reasoning process, while the persuasive and human-like nature of the generated output encourages excessive professional reliance. The article situates these dynamics within the framework of European AI governance, highlighting the interaction between fictitious outputs and automation bias fundamentally undermines the principle of explainability. The conclusion of the discussion refers to, in the absence of effective mechanisms for meaningful human oversight, the routine deployment of GLAI poses substantial challenges to judicial independence and the protection of fundamental rights.