DeepSeek V4

DeepSeek released two models: DeepC V4 Pro and Flash.

DeepSeek published a 58-page technical report covering everything they did.

DeepSeek V4 is benchmarked as the third best open weights model, behind Kimi K 2.6 and Mimo V 2.5 Pro.

Flash with 13 billion active parameters can reach reasoning performance comparable to GPT-5.2 and Gemini 3.0 Pro when given a larger thinking budget.

Pro Max is claimed as the strongest open model, beating previous open-source models on reasoning, coding, long context, and agentic benchmarks; on Artificial Analysis it wins Terminal Bench Hard and ranks second or third as an open-weights model.

On reasoning, Pro Max falls slightly behind GPT 5.4 and Gemini 3.1 Pro, representing roughly a 3 to 6 month gap behind frontier closed models.

For agent tasks, ProMax is on par with Kimi K2.6 and GLM5.1 but still slightly behind frontier closed models.

ProMax surpasses Gemini 3.1 Pro on academic long context benchmarks.

DeepSeek left some benchmark entries blank when comparing against Kimi K2.6 and GLM 5.1 because their APIs were too busy to return responses, indicating serving capacity issues.

DeepSeek V4 natively supports a 1 million context window with near state-of-the-art retrieval accuracy and costs 10 to 100 times cheaper than competitors.

DeepSeek API pricing: $0.435 per million input tokens, $0.87 per million output tokens, with a cash hit discount bringing it to $0.3625 per million tokens.

GLM 5.1 costs $1.4 input and $4.4 output; Gemini 3.1 Pro costs $2 input and $12 output; Opus 4.6 costs $5 input and $25 output per million tokens.

No other AI labs can match DeepSeek's pricing without operating at a loss.

DeepSeek is likely making a 50 to 70 percent margin on their pricing.

The price discount was made permanent.

DeepSeek introduced two new attention mechanisms: Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA).

The new attention setup makes DeepSeek V4 the best open-source 1 million context window model, rivaling closed-source models like Gemini 3.1 Pro and closing in on Opus 4.6 in retrieval accuracy.

CSA compresses every four tokens into one learned KV entry, not a simple average pooling but a learned compression using contribution weights.

HCA compresses every 128 tokens into a single KV entry, making memory usage 32 times lighter than CSA and enabling cheap dense global attention.

The model interleaves CSA and HCA at a 1:1 ratio, and both branches have an additional sliding window attention of 128 tokens to preserve recent context.

DeepSeek V4 Pro has 1.6 trillion total parameters with 49 billion active; V4 Flash has 284 billion total parameters with 13 billion active; both support 1 million token context windows.

DeepSeek V4 replaces the residual stream with manifold constrained hyperconnections (MHC) to increase representational capacity and help information survive across depth.

The MoE block uses 256 fine-grained router experts, shared experts, router activation changed from sigmoid to square root soft plus, and a sequence-level balance loss to avoid extreme routing imbalance.

Early layers use hash routing based on token ID instead of learned routing, giving each token a stable expert path without spending capacity on routing.

At 1 million tokens, V4 Pro uses only 27 percent of the inference flops and 10 percent of the KV cache compared to V3.2.

Against a GQA baseline (as used in Llama 2 and 3), V4 Pro reduces total KV cache by 34 times and V4 Flash reduces it by 49 times.

DeepSeek V4 uses FP4 quantization-aware training for MoE expert weights, so the model learns to survive extremely low precision during inference.

DeepSeek V4 Flash was pre-trained on 32 trillion tokens and V4 Pro on 33 trillion tokens, doubling previous pre-training runs; the only other known model with a similar amount is Kimi K2.5 at 30 trillion tokens.

The training used the Muon optimizer (first developed by Keller Jordan in December 2024, verified at scale by Kimi in February 2025) for most parameters, keeping AdamW for embeddings, prediction heads, and some MHC parameters.

Muon provides faster convergence and better training stability according to DeepSeek.

Data processing includes DeepCV3-style preparation, 128K tokenizer, token splitting, fill-in-the-middle, a new DSML special token for XML-based tool invocation, and sample-level attention masking for packed documents.

Post-training: instead of direct RL, multiple specialist models for math, coding, agents, and instruction following were trained separately, then distilled into a unified model via on-policy distillation.

DeepSeek V4 has three reasoning modes: non-thinking, thinking high, and think max, where think max uses longer context, weaker length penalties, and different system prompts.

The release provides six main variants: Pro, Pro Base, Pro Max, Flash, Flash Base, Flash Max.

The inference stack is optimized for Huawei chips with day-zero support, compression of attention and KV cache, and pushing the CSA indexer into lower precision.

DeepSeek V4 is a cost-performance release focused on long context efficiency and test-time scaling rather than chasing the top of the benchmark leaderboard.

DeepSeek is trying to solve the underlying serving cost and reliability problem that many powerful models face in the real world.

The demand for AI and LMS is stronger than ever, and new datacenter buildouts are potentially necessary.