Phi-3.5 MoE: Complete Benchmark Performance Guide

other phi-3-5-moe 2024-08-16 MIT

In-depth analysis of Phi-3.5 MoE performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionphi-3-5-moe
Release Date2024-08-16
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseMIT
Documentationhttps://huggingface.co/models

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU75.7%2024-08-165-shotview
HUMANEVAL69.5pass@12024-08-16—view
GSM8K75.5%2024-08-160-shot CoTview
MATH38.2%2024-08-160-shot CoTview
BBH79.4%2024-08-163-shot CoTview
GPQA40.2%2024-08-160-shotview
IFEVAL74.3%2024-08-16prompt_strictview
ARC91.9%2024-08-16challengeview
MUSR50.3%2024-08-160-shotview
WINOGRANDE78.3%2024-08-160-shotview

Pricing

TierPriceCurrency
Input$0.2 / MtokUSD
Output$0.2 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2024-08-16

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Phi-3.5 MoE

Aperçu du modèle

Microsoft Phi-3.5 MoE 42B 混合专家模型, 128K 上下文, 总参 42B/活跃 6.6B, 性能接近 70B 稠密模型。

Spécifications principales

FournisseurVersionDate de sortieFenêtre de contexteModalités d’entréeModalités de sortieLicence
Otherphi-3-5-moe2024-08-16128KtexttextMIT

Performance aux benchmarks

BenchmarkScoreUnitéNotes
MMLU (Massive Multitask Language Understanding)75.7%5-shot
HumanEval69.5pass@1—
GSM8K (Grade School Math 8K)75.5%0-shot CoT
MATH38.2%0-shot CoT
BBH (BIG-Bench Hard)79.4%3-shot CoT
GPQA40.2%0-shot
IFEval74.3%prompt_strict
ARC91.9%challenge
MUSR50.3%0-shot
WinoGrande78.3%0-shot

Tarification

EntréeSortieLecture cacheÉcriture cache
————

par million de jetons

Forces

  • 采用 MoE 混合专家架构。

Faiblesses

  • 闭源专有模型，不支持自托管。

Cas d’usage

  • 通用对话与问答

Références