LatticeFlow Unveils First EU AI Act Compliance Framework for Large Language Models
12 Sources
[1]
Researchers provide LLM benchmarking suite for the EU Artificial Intelligence Act
Researchers from ETH Zurich, the Bulgarian AI research institute INSAIT -- created in partnership with ETH and EPFL -- and the ETH spin-off LatticeFlow AI have provided the first comprehensive technical interpretation of the EU AI Act for General Purpose AI (GPAI) models. This makes them the first
[2]
LatticeFlow's LLM framework takes a first stab at benchmarking Big AI's compliance with EU AI Act
While most countries' lawmakers are still discussing how to put guardrails around artificial intelligence the European Union is ahead of the pack, having passed a risk-based framework for regulating AI apps earlier this year. The law came into force in August, although full details of the pan-EU
[3]
EU AI Act Checker Reveals Big Tech's Compliance Pitfalls
Popular AI models were tested across categories in line with the AI Act Some of the most prominent artificial intelligence models are falling short of European regulations in key areas such as cybersecurity resilience and discriminatory output, according to data seen by Reuters. The EU had long
[4]
LatticeFlow releases framework for checking LLMs' compliance with the EU AI Act - SiliconANGLE
LatticeFlow releases framework for checking LLMs' compliance with the EU AI Act Startup LatticeFlow AG today released COMPL-AI, a framework that can help companies check whether their large language models comply with the EU AI Act. Zurich-based LatticeFlow is backed by more than $14 million in
[5]
EU AI Act checker reveals Big Tech's compliance pitfalls
The EU had long debated new AI regulations before OpenAI released ChatGPT to the public in late 2022. The record-breaking popularity and ensuing public debate over the supposed existential risks of such models spurred lawmakers to draw up specific rules around "general-purpose" AIs (GPAI).Some of
[6]
Exclusive-EU AI Act checker reveals Big Tech's compliance pitfalls
LONDON (Reuters) - Some of the most prominent artificial intelligence models are falling short of European regulations in key areas such as cybersecurity resilience and discriminatory output, according to data seen by Reuters. The EU had long debated new AI regulations before OpenAI released
[7]
EU AI Act checker reveals Big Tech's compliance pitfalls
Some of the most prominent artificial intelligence models are falling short of European regulations in key areas such as cybersecurity resilience and discriminatory output, according to data seen by Reuters. The EU had long debated new AI regulations before OpenAI released ChatGPT to the public in
[8]
EU AI Act checker exposes Big Tech compliance gaps | bobsguide
A new AI compliance tool developed by LatticeFlow AI has revealed that major models from Meta, OpenAI, and Alibaba may fall short of the European Union's evolving AI Act standards. Early tests show that these models could face challenges in areas like cybersecurity and discriminatory output, with
[9]
Exclusive: EU AI Act checker reveals Big Tech's compliance pitfalls
LONDON, Oct 16 (Reuters) - Some of the most prominent artificial intelligence models are falling short of European regulations in key areas such as cybersecurity resilience and discriminatory output, according to data seen by Reuters. The EU had long debated new AI regulations before OpenAI
[10]
AI companies fall short of meeting EU AI Act standards - study
Big Tech companies such as Apple and Meta have been cautious in rolling out their AI models in Europe. A new 'LLM Checker' could help. The leading generative artificial intelligence (GenAI) models, including OpenAI, Meta and Anthropic, do not fully comply with Europe's AI rules, according to a
[11]
AI Models Fall Short of Key European Standards in New Compliance Test | PYMNTS.com
Several leading artificial intelligence models are struggling to meet stringent European Union regulations in areas such as cybersecurity resilience and the prevention of discriminatory outputs, according to data reviewed by Reuters. The results come from a newly developed tool designed to test
[12]
EU Compliance Checker Exposes Major Flaws In Google and OpenAI AI Models
However, COMPL-AI revealed significant shortcomings in fairness and non-discrimination across all models. The EU's AI Act outlines the regulatory requirements of responsible AI development. However, it lacks a clear technical interpretation, making assessing the models' compliance difficult. To
Share
Copy Link
LatticeFlow, in collaboration with ETH Zurich and INSAIT, has developed the first comprehensive technical interpretation of the EU AI Act for evaluating Large Language Models (LLMs), revealing compliance gaps in popular AI models.
LatticeFlow Introduces Pioneering EU AI Act Compliance Framework
In a significant development for the artificial intelligence industry, Swiss startup LatticeFlow has unveiled COMPL-AI, the first comprehensive technical interpretation of the EU Artificial Intelligence Act (AI Act) for General Purpose AI (GPAI) models
1
. This groundbreaking framework, developed in collaboration with researchers from ETH Zurich and the Bulgarian AI research institute INSAIT, aims to translate the legal requirements of the EU AI Act into concrete, measurable, and verifiable technical benchmarks2
.The Need for Technical Interpretation
The EU AI Act, which came into force in August 2024, is the world's first comprehensive AI legislative package. However, the act's high-level legal requirements have been challenging to interpret technically, making it difficult for developers to create compliant AI models and for regulators to assess compliance
3
. LatticeFlow's framework addresses this gap by providing a practical approach for model developers to align with future EU legal requirements.COMPL-AI Framework and Evaluation Tool
The COMPL-AI framework includes:
- A set of technical requirements derived from the AI Act
- An open-source compliance evaluation tool
- 27 state-of-the-art evaluation benchmarks
4
The evaluation tool, dubbed the "Large Language Model (LLM) Checker," assesses AI models across various categories, including cybersecurity resilience, discriminatory output, and fairness
5
. It awards scores between 0 (no compliance) and 1 (full compliance) for each benchmark.Evaluation of Popular AI Models
LatticeFlow applied its benchmark approach to 12 prominent language models, including those from OpenAI, Meta, Google, Anthropic, and Alibaba. The results revealed that:
- Most models scored well on not following harmful instructions
- Performance on reasoning and general knowledge varied widely
- All models struggled with recommendation consistency, a measure of fairness
- Smaller models (≤ 13B parameters) scored poorly on technical robustness and safety
- Almost all examined models struggled with diversity, non-discrimination, and fairness
2
5
Key Findings and Implications
The evaluation uncovered several important insights:
- No language model fully meets the requirements of the EU AI Act
- Models have been predominantly optimized for capabilities rather than compliance
- There are significant shortcomings in areas such as robustness, diversity, and fairness
- Evaluating compliance with privacy and copyright considerations remains challenging
1
3
5
Related Stories
Industry Response and Future Developments
The framework has garnered attention from the European Commission, which described it as a "first step" in implementing the new laws
5
. LatticeFlow CEO Petar Tsankov emphasized that while the overall test results were positive, they offer companies a roadmap for fine-tuning their models to align with the AI Act5
.As the EU continues to establish enforcement mechanisms for the AI Act, the COMPL-AI framework is expected to evolve. LatticeFlow plans to extend the test to encompass further enforcement measures as they are introduced, and the LLM Checker will be freely available for developers to test their models' compliance online
4
5
.Conclusion
The introduction of the COMPL-AI framework marks a significant milestone in the implementation of the EU AI Act. As companies face potential fines of up to 35 million euros or 7% of global annual turnover for non-compliance, this tool provides a crucial resource for AI developers and policymakers alike
5
. The framework not only highlights current shortcomings in popular AI models but also paves the way for more responsible and compliant AI development in the future.References
Summarized by
Navi
[2]
[3]
[4]
Related Stories
EU AI Act Takes Effect: New Rules and Implications for AI Providers and Users
01 Aug 2025•Policy and Regulation
EU AI Act: Comprehensive Regulatory Framework for AI Takes Shape
15 Nov 2024•Policy and Regulation
EU Unveils AI Code of Practice: Balancing Innovation with Regulation
11 Jul 2025•Policy and Regulation
Recent Highlights
1
AI Hallucination Nearly Sparked US Military Confrontation With Chinese Ship in Middle East
Technology
2
OpenAI's AI Solves Navier-Stokes Problem But Ignites Existential Crisis in Mathematics Community
Science and Research
3
Anthropic's Claude Now Leads 26% of AI Development Work, Company Reveals New Transparency Metrics
Technology
Recent Highlights
Today's Top Stories
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Follow topics that matter to you and stay ahead.
