Agreement ScoreHow Alethe AI measures whether models truly agree — and not just agree on the words.
The text comparison problemTwo responses can use completely different words and mean the same thing — or conversely, use the same words and mean the opposite. Word-for-word comparison would be useless.
Same meaning, different words"AI should be regulated" ≈ "Legal safeguards are necessary for artificial intelligence"
Same words, opposite meaning"Freedom is essential" (liberal) vs "Unlimited freedom is dangerous" (regulatory)
The solution: vector embeddingsAn embedding is a mathematical representation of the meaning of a text as a vector — an array of hundreds of numbers. Two semantically close texts produce close vectors in vector space.
Score calculation
1
Embeddings generatedEach AI response is converted into a vector by an embeddings model.
2
Cosine similarityWe calculate the cosine similarity between each pair of vectors: a small angle = close ideas.
3
Aggregated scoreThe average of similarities gives the global agreement score, expressed as a percentage.
Used internally, not displayed
The agreement score is not shown to users during a conversation. We use it internally to evaluate the quality of debates, identify patterns, and continuously improve the system.
Quality evaluationEach debate is scored to measure how well the models converged.
TestingWe run automated tests to validate prompt improvements using score variations.
System improvementScore trends guide our decisions on prompts, models and stop conditions.
Stop conditions
Token limit reachedThe debate stops when the plan's token limit is hit (Free: 30K, Pro: 80K, Ultra: 200K, Apex: 500K). 1 token ≈ 4 characters.
Manual stopYou can stop the debate at any time. All messages are saved.