Claude 2.1: Complete Benchmark Performance Guide

anthropic 2.1 2023-11-21 Proprietary

In-depth analysis of Claude 2.1 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendoranthropic
Version2.1
Release Date2023-11-21
Context Window200000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU60.9%2023-11-215-shotview
HUMANEVAL28.1pass@12023-11-21—view
GSM8K45.7%2023-11-210-shot CoTview
MATH15.5%2023-11-210-shot CoTview
BBH63.8%2023-11-213-shot CoTview
GPQA26.4%2023-11-210-shotview
IFEVAL55%2023-11-21prompt_strictview
ARC84.8%2023-11-21challengeview
MUSR37.7%2023-11-210-shotview
WINOGRANDE67.4%2023-11-210-shotview

Pricing

TierPriceCurrency
Input$11 / MtokUSD
Output$32 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://www.anthropic.com/pricing · as of 2023-11-21

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✗

Claude 2.1

Aperçu du modèle

Anthropic Claude 2.1, 200K 上下文, 相比 2.0 减少 2 倍幻觉率, 改进长文档召回能力。

Spécifications principales

FournisseurVersionDate de sortieFenêtre de contexteModalités d’entréeModalités de sortieLicence
Anthropic2.12023-11-21200KtexttextProprietary

Performance aux benchmarks

BenchmarkScoreUnitéNotes
MMLU (Massive Multitask Language Understanding)60.9%5-shot
HumanEval28.1pass@1—
GSM8K (Grade School Math 8K)45.7%0-shot CoT
MATH15.5%0-shot CoT
BBH (BIG-Bench Hard)63.8%3-shot CoT
GPQA26.4%0-shot
IFEval55.0%prompt_strict
ARC84.8%challenge
MUSR37.7%0-shot
WinoGrande67.4%0-shot

Tarification

EntréeSortieLecture cacheÉcriture cache
————

par million de jetons

Forces

  • 上下文窗口 200K，支持长文本。

Faiblesses

  • HumanEval 28.1，代码能力较弱。
  • 闭源专有模型，不支持自托管。

Cas d’usage

  • 长文档摘要

Références