Grok-2: Complete Benchmark Performance Guide

xai 2 2024-08-13 Proprietary

In-depth analysis of Grok-2 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorxai
Version2
Release Date2024-08-13
Context Window131072 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.x.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU87.5%2024-08-135-shotview
HUMANEVAL88.4pass@12024-08-13—view
GSM8K93.2%2024-08-130-shot CoTview
MATH76.8%2024-08-130-shot CoTview
BBH84%2024-08-133-shot CoTview

Pricing

TierPriceCurrency
Input$2 / MtokUSD
Output$10 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://x.ai/api · as of 2024-08-13

Compliance

  • Data Residency: US
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Grok-2

Modellübersicht

xAI Grok-2 是 xAI 旗舰模型，131K 上下文窗口，支持文本与图像输入，在 MMLU、HumanEval、MATH 等基准上接近头部闭源模型，集成于 X 平台提供实时信息。

Kernspezifikationen

AnbieterVersionVeröffentlichungsdatumKontextfensterEingabemodalitätenAusgabemodalitätenLizenz
Xai22024-08-13131Ktext, imagetextProprietary

Benchmark-Leistung

BenchmarkErgebnisEinheitNotizen
MMLU (Massive Multitask Language Understanding)87.5%5-shot
HumanEval88.4pass@1—
GSM8K (Grade School Math 8K)93.2%0-shot CoT
MATH76.8%0-shot CoT
BBH (BIG-Bench Hard)84.0%3-shot CoT

Preise

EingabeAusgabeCache-LesenCache-Schreiben
————

pro Million Token

Stärken

  • MMLU score 87.5, strong knowledge reasoning.
  • HumanEval 88.4, excellent code generation.
  • GSM8K 93.2, robust math reasoning.
  • 支持文本、图像、音频多模态输入。

Schwächen

  • 闭源专有模型，不支持自托管。

Anwendungsfälle

  • 代码生成与调试
  • 视觉与图像理解
  • Agent 工作流与工具调用

Referenzen