DeepSeek V4

The transcript states 'They just dropped a new model, Deepseek V4.'

The author says 'They don't have nearly as much compute' in contrast to top closed labs.

The transcript explicitly states 'they don't even have the top NVIDIA chips.'

The video says 'their team is like 40 times smaller than OpenAI.'

The author states 'they managed to build a model that's on par with the top closed models out there.'

The author shows benchmark charts and states it's 'pretty much on par with some of the top closed models out there, including Opus 4.6 Max and Gemini 3.1 Pro.'

The transcript states 'DeepS version 4 on average has a higher win rate than Opus 4.6.'

The video says 'Deepseek V4 achieved a perfect score, 120 out of 120' on the Putnam 2025.

The transcript claims 'its retrieval accuracy even beats Google's latest Gemini 3.1 Pro' when pushed to the 1M limit.

The video references the independent leaderboard showing DeepSeek V4 Pro as second best open-source model and edging close to top closed models.

The transcript mentions 'they even open sourced this.'

The author states 'they actually open sourced the model... it's out on Hugging Face, you can download this for free.'

The video says 'they even released a paper on how they built it.'

The transcript says they released a paper 'spilling all the info on how it was designed and how they trained it, including this infrastructure stuff, which is like top secret for the closed AI labs.'

The author states 'this latest V4 Pro model is massive. It has 1.6 trillion parameters.'

The transcript says 'this new V4 also has a context length of 1 million tokens.'

The video explains '1 million tokens is roughly 750,000 words.'

The author states 'a 1 million token context window is insanely hard to actually build correctly.'

The transcript explains that at a million tokens 'the number of comparisons becomes astronomical' and 'high-end hardware starts to choke.'

The video states the KV cache becomes 'absurd... gigabytes sitting in expensive GPU memory.'

The author explains CSA takes chunks of tokens and merges them, reducing sequence length by a factor of four.

The transcript describes the Lightning indexer picking out 'only the most useful pieces' and everything else is skipped.

The video says HCA groups '128 tokens or like an entire paragraph' and the sequence becomes short enough to look at everything.

The author introduces sliding window attention that 'continuously tracks the most recent tokens... with full exact fidelity.'

The transcript explains the three parallel views are interleaved 'layer by layer through the neural network' to achieve a 1M context window without devastating compute cost.

The paper benchmarks show V4 Pro requires 3.7 times lower flops and 'runs on roughly 27% of the compute that was required for the previous version.'

The video states the KV cache is 'almost 10 times smaller than the previous DeepSeek version' and 'only requires 10% of the KV cache memory.'

The transcript explains that at trillion-parameter scale 'signals flowing through the network start to amplify like crazy' causing feedback loops that crash training.

The paper introduces MHC which constrains residuals to a 'manifold of doubly stochastic matrices' where every row and column sums to 1, forbidding blowup.

The transcript mentions 'a rapid sequence of row and column normalizations around 20 iterations' to satisfy the constraints.

The video states they shrunk the overhead of the entire process to 'only 6.7% of runtime' using fused kernels and custom GPU tweaks.

The transcript states the system looks at slightly older versions of parameters to lock onto the underlying trend and stabilise when early signs of a loss spike are detected.

The author explains DeepSeek replaced AdamW with Muon, which first makes big rough adjustments and then tiny precise tweaks, like tuning a guitar.

The transcript describes breaking data into smaller sequential waves so that computation on wave N overlaps with transfer of wave N+1, making latency disappear.

The paper mentions using Tilang to develop fused kernels and a Z3 SMT solver to mathematically prove the kernel code was correct.

The transcript explicitly states 'Deepsek v4 probe was trained on 33 trillion tokens.'

The video explains they started with short 4K token sequences, then raised to 16K, 64K, up to 1M tokens.