Claude 3.5 Haiku: Complete Benchmark Performance Guide

anthropic 3.5-haiku 2024-11-04 Proprietary

In-depth analysis of Claude 3.5 Haiku performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendoranthropic
Version3.5-haiku
Release Date2024-11-04
Context Window200000 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU77.6%2024-11-045-shotview
HUMANEVAL73.8pass@12024-11-04—view
GSM8K75.5%2024-11-040-shot CoTview
MATH53%2024-11-040-shot CoTview
BBH70.2%2024-11-043-shot CoTview
GPQA31.1%2024-11-040-shotview
IFEVAL73.6%2024-11-04prompt_strictview
ARC91.8%2024-11-04challengeview
MUSR52.3%2024-11-040-shotview
WINOGRANDE83.8%2024-11-040-shotview

Pricing

TierPriceCurrency
Input$0.8 / MtokUSD
Output$4 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://www.anthropic.com/pricing · as of 2024-11-04

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✗

Claude 3.5 Haiku

Aperçu du modèle

Anthropic Claude 3.5 Haiku 经济型模型, 200K 上下文, 速度比 Sonnet 快 2 倍, 性能超越 Claude 3 Opus。

Spécifications principales

FournisseurVersionDate de sortieFenêtre de contexteModalités d’entréeModalités de sortieLicence
Anthropic3.5-haiku2024-11-04200Ktext, imagetextProprietary

Performance aux benchmarks

BenchmarkScoreUnitéNotes
MMLU (Massive Multitask Language Understanding)77.6%5-shot
HumanEval73.8pass@1—
GSM8K (Grade School Math 8K)75.5%0-shot CoT
MATH53.0%0-shot CoT
BBH (BIG-Bench Hard)70.2%3-shot CoT
GPQA31.1%0-shot
IFEval73.6%prompt_strict
ARC91.8%challenge
MUSR52.3%0-shot
WinoGrande83.8%0-shot

Tarification

EntréeSortieLecture cacheÉcriture cache
————

par million de jetons

Forces

  • 可靠的通用模型。

Faiblesses

  • 闭源专有模型，不支持自托管。

Cas d’usage

  • 代码生成与调试

Références