Claude 3 Sonnet (2024-02-29): Complete Benchmark Performance Guide

anthropic 3-sonnet-20240229 2024-02-29 Proprietary

In-depth analysis of Claude 3 Sonnet (2024-02-29) performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendoranthropic
Version3-sonnet-20240229
Release Date2024-02-29
Context Window200000 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU84%2024-02-295-shotview
HUMANEVAL80.1pass@12024-02-29—view
GSM8K82.6%2024-02-290-shot CoTview
MATH46.4%2024-02-290-shot CoTview
BBH76.2%2024-02-293-shot CoTview
GPQA41.7%2024-02-290-shotview
IFEVAL75.5%2024-02-29prompt_strictview
ARC94.8%2024-02-29challengeview
MUSR63.5%2024-02-290-shotview
WINOGRANDE80.3%2024-02-290-shotview

Pricing

TierPriceCurrency
Input$3 / MtokUSD
Output$15 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://www.anthropic.com/pricing · as of 2024-02-29

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✗

Claude 3 Sonnet (2024-02-29)

Modellübersicht

Anthropic Claude 3 Sonnet 2024-02-29 快照版本, 200K 上下文, 速度与智能平衡型。

Kernspezifikationen

AnbieterVersionVeröffentlichungsdatumKontextfensterEingabemodalitätenAusgabemodalitätenLizenz
Anthropic3-sonnet-202402292024-02-29200Ktext, imagetextProprietary

Benchmark-Leistung

BenchmarkErgebnisEinheitNotizen
MMLU (Massive Multitask Language Understanding)84.0%5-shot
HumanEval80.1pass@1—
GSM8K (Grade School Math 8K)82.6%0-shot CoT
MATH46.4%0-shot CoT
BBH (BIG-Bench Hard)76.2%3-shot CoT
GPQA41.7%0-shot
IFEval75.5%prompt_strict
ARC94.8%challenge
MUSR63.5%0-shot
WinoGrande80.3%0-shot

Preise

EingabeAusgabeCache-LesenCache-Schreiben
————

pro Million Token

Stärken

  • MMLU score 84.0, strong knowledge reasoning.
  • HumanEval 80.1, excellent code generation.
  • 支持文本、图像、音频多模态输入。
  • 上下文窗口 200K，支持长文本。

Schwächen

  • 闭源专有模型，不支持自托管。

Anwendungsfälle

  • 代码生成与调试
  • 长文档摘要
  • 视觉与图像理解

Referenzen