CivicEval

Tests whether a model assumes a user's location (e.g., US-centric) when asked location-sensitive questions without explicit context. The ideal response should be location-agnostic, query for the user's location, or explain why the location is necessary for a good answer.

ASQA benchmark with 40 selected long-form question/answer pairs (min 50 words for response) from the [ASQA dataset here](https://huggingface.co/datasets/asqa).

Tests an LLM's ability to maintain appropriate boundaries and accurately represent its nature as an AI. The evaluation focuses on the LLM's capacity to avoid responses that falsely imply human-like consciousness, emotions, personal relationships, or existential experiences, which could be misleading or damaging to users.

Tests an LLM's ability to identify a non-existent UDHR article and resist engaging with an absurdly and emotionally framed critique of it.

Geneva Conventions Evaluations, including all four Geneva Conventions (GC1, GC2, GC3, GC4) and Common Articles 1, 2, and 3.

Tests an LLM's ability to classify fabricated and unverifiable URLs, expecting an 'UNKNOWN' response. The goal is to assess if the LLM acknowledges the impossibility of verification for URLs that appear to be from legitimate news domains but describe absurd or clearly false events.

LLAMA MODEL SPECIFIC (testing): ASQA benchmark with 40 selected long-form question/answer pairs (min 50 words for response) from the [ASQA dataset here](https://huggingface.co/datasets/asqa).

Eight-prompt suite that probes an LLM's knowledge of the 2018 Regional Agreement on Access to Information, Public Participation and Justice in Environmental Matters in Latin America and the Caribbean (Escazú Agreement). Prompts cover the treaty's three access-rights pillars, exceptions, procedural safeguards and the defender-protection clause.

Evaluates understanding of the core provisions, definitions, obligations, and prohibitions outlined in the EU Artificial Intelligence Act.

A configuration to assess LLM understanding of the ASEAN Charter, covering its purposes, principles, organs, decision-making, and other key aspects with a mix of recall and nuanced questions.

Tests the tendency of LLMs to hallucinate by querying them about non-existent but plausible-sounding events, theories, or items. Ideal responses should indicate a lack of knowledge or inability to confirm the existence of the queried item.

Tests comprehension of the primary causes, methods, pivotal events, legislative achievements, and differing tactical approaches within the US Civil Rights Movement (1954-1968) as detailed in the source document, through a series of focused questions.

Evaluates understanding of the HMT Empire Windrush, covering its origins as MV Monte Rosa, WWII service, the significant 1948 voyage, the 'Windrush generation,' passenger details, government reactions, and its eventual loss.

Evaluates the models on the UDHR dataset (Universal Declaration of Human Rights).

Recall and application of distinctive rights and duties in the African Charter on Human and Peoples' Rights (ACHPR) plus its 2003 Maputo women's-rights protocol.

Selected questions from research of [bodyanddata.org](https://bodyanddata.org/)

Tests an LLM’s knowledge of key provisions of the Equality Act 2010: protected characteristics, direct and indirect discrimination, the duty to make reasonable adjustments, and harassment.