TRLawBench
Turkish law LLM benchmark: Measures large language models' Turkish legal knowledge and reasoning across 3 stages. Comprehensive evaluation of 297 questions covering ÖSYM exam items, advanced reasoning, and open-ended analysis.
Overview
TRLawBench is an open three-stage benchmark that measures how well large language models (GPT, Claude, Gemini, Gemma, Qwen, Grok, DeepSeek, Mistral, Llama and others) handle Turkish law. Stages 2 and 3, co-authored with attorney Esat Erbil Tavus, were compiled from print sources and never published online, eliminating training-data contamination.
Why TRLawBench?
- Turkish legal reasoning focus: Not general language fluency but grounded evaluation against Turkish Code of Obligations (TBK), Turkish Penal Code (TCK), the Constitution, and related legislation.
- Contamination-free: 200 questions sourced from print, never indexed online, so they cannot appear in training data.
- Transparent methodology: Correct-answer detection is deterministic; the LLM judge scores only reasoning quality and is blind to the model identity.
Evaluation Methodology
- Stage 1: Foundational Legal Knowledge. 97 multiple-choice questions from ÖSYM legal exams. Correct answers derived from the official ÖSYM answer key.
- Stage 2: Advanced Turkish Legal Reasoning. 100 advanced multiple-choice questions across 14 categories. Print-sourced, never online.
- Stage 3: Open-Ended Questions. 100 open-ended questions spanning 14 branches of law. Deterministic letter-match plus Gemini 3.1 Pro blind-judge scoring on a 0 to 10 scale.
Per-stage leaderboards, accuracy rates, and sample Q&A are available in the interactive panel above.
Open-Ended Questions
Prepared together with Atty. Esat Erbil Tavus, the third stage consists of 100 open-ended legal questions categorized across 14 branches of law. Questions were compiled from various print sources and are not available online, so they cannot have leaked into models' training data. Each requires free-form responses with grounded argumentation and references to relevant legislation and case law.
Correct count is deterministic: every question has a ground-truth answer in the bank, and the model's chosen letter is matched against it — no LLM judge involved, so self-bias is impossible. Gemini 3.1 Pro (blind judge — unaware of which model produced the response) separately scores each answer on a 0–10 scale: correct answers reached through the expected reasoning path earn 10; correct answers with reasoning gaps score below 10; responses with largely sound reasoning can earn above 0 even when they reach the wrong conclusion. Evaluated April 2026.