Mixtral 8x22B: Complete Benchmark Performance Guide

mistral 8x22b 2024-04-10 Apache 2.0

In-depth analysis of Mixtral 8x22B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Version8x22b
Release Date2024-04-10
Context Window64000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://docs.mistral.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU77.8%2024-04-105-shotview
HUMANEVAL45.2pass@12024-04-10—view
GSM8K78.6%2024-04-100-shot CoTview
MATH46%2024-04-100-shot CoTview
BBH74.5%2024-04-103-shot CoTview

Pricing

TierPriceCurrency
Input$1.2 / MtokUSD
Output$1.2 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2024-08-01

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Mixtral 8x22B

Modellübersicht

Mistral Mixtral 8x22B 是稀疏混合专家（MoE）开源模型，总参数 141B、活跃参数 39B，64K 上下文窗口，在多语言与代码任务上表现稳健，Apache 2.0 许可证可商用。

Kernspezifikationen

AnbieterVersionVeröffentlichungsdatumKontextfensterEingabemodalitätenAusgabemodalitätenLizenz
Mistral8x22b2024-04-1064KtexttextApache 2.0

Benchmark-Leistung

BenchmarkErgebnisEinheitNotizen
MMLU (Massive Multitask Language Understanding)77.8%5-shot
HumanEval45.2pass@1—
GSM8K (Grade School Math 8K)78.6%0-shot CoT
MATH46.0%0-shot CoT
BBH (BIG-Bench Hard)74.5%3-shot CoT

Preise

EingabeAusgabeCache-LesenCache-Schreiben
————

pro Million Token

Stärken

  • 采用 MoE 混合专家架构。

Schwächen

  • HumanEval 45.2，代码能力较弱。
  • 闭源专有模型，不支持自托管。

Anwendungsfälle

  • 通用对话与问答

Referenzen