All right. Hi everyone, my name is Olive. I work at MiniMax on reinforcement learning research. Today, I'm going to talk a little bit about our new model, MiniMax M3, how we trained it natively multimodal from step zero, and how we trained it with MSA, which is MiniMax Sparse Attention.

A little bit of introduction to our team: we are one of the few labs in the world that works across multimodalities, which means that we have published our LLM models before, we work on speech models, we have our own video generation models, and we have our coding agent models.

Lately, we just released our MiniMax M3, which is open-weight with frontier performance. It is actually the first open-weight model to combine the following three frontier capabilities. The first one is what we call SOTA coding and agent capabilities. The model is able to do a lot of the agentic tasks that you imagine: it can do automatic task decomposition, it can work in your coding agents, and it can drop in for 90% of the coding work that you are currently doing.

The second pillar is that the model has a 1 million context window, which is very long and unlocks basically all long user interactions, such as handling full codebases. We trained it with MSA (MiniMax Sparse Attention), so it achieves $9\times$ faster prefill and $15\times$ faster decode versus full attention.

The third one is that the model was trained natively multimodal from step zero. This means that from scratch, we combined visual understanding with language generation. The model understands both capabilities very well from the start so that we can build many more applications on top of that, such as computer-use agents.

First, I'll briefly explain what sparse attention does. Our core design principle is that a simpler architecture can always lead to better scaling properties. This is the principle we have followed throughout the entire design. Our goal is to always pursue a cleaner and simpler architecture.

The sparse attention consists of two branches. The first one is what we call the index branch, which performs a lightweight full attention computation. It uses four query heads and one key head, and it is trained with a KL loss to match the attention scores of the main branch. The index branch is used to identify the tokens with the highest attention scores. These selected tokens are then attended to by the main branch.

In other words, the sparse branch does not attend to all of the tokens, but only to a subset of them. This subset is selected through the lightweight computation of the index branch. Then, the sparse branch uses the blocks selected by the index branch to determine which keys and values are needed for the attention computation.

What I want to emphasize today is that our design is not purely algorithm-driven. Instead, we took an algorithm and infrastructure co-design approach. A good example to start with is DSA. DSA was a very strong design, but it is also highly optimized for DeepSeek's own architecture, especially the combination of MQA and a large head dimension.

However, when we move to a more commonly used GQA architecture, directly applying the same design becomes less straightforward. It introduces several practical issues, especially from the infrastructure and implementation perspective. So this motivates our design choice: instead of simply borrowing an existing sparse attention design, we rethink the architecture under the constraints of GQA and real system efficiency.

So, what are the issues if we directly apply the existing sparse attention architecture? First, from the algorithm perspective, GQA has multiple KV heads, which should ideally provide more diversity across different KV groups. However, if we directly reuse the same token selection strategy, multiple KV heads will end up attending to exactly the same selected tokens. This essentially cancels out the diversity that GQA is supposed to provide.

Second, from the infrastructure perspective, this design is not very friendly to GPU memory access. GPUs are much more efficient when reading contiguous data. In the MQA setting, we can think of the KV layout as one KV head with a large head dimension, for example, $1 \times 512$. This is relatively contiguous and efficient to read.

However, in GQA, the same total dimension may be split across multiple KV heads—for example, $4 \times 128$. These are no longer one single contiguous read. Instead, the GPU needs to perform multiple independent reads across different KV heads. This increases memory access overhead and reduces the compute-to-memory access ratio.

Another issue here is the top-$k$ computation itself. To select the most important tokens, the indexer needs to compute top-$k$ scores over a large number of token-level candidates. However, top-$k$ has inherent data dependencies, so it is difficult to fully parallelize on GPUs. As a result, token-level selection introduces non-trivial computation and merging overhead.

So, how did we address these issues in MSA? From the algorithm side, we remove the aggregation of the index branch outputs. In the original design, the results from all index branch heads are merged into a single set of selected tokens. But as we discussed earlier, this would remove the diversity brought by GQA. Instead, we keep the selection distinct across KV heads. Under our sparse attention design, when using GQA, different KV heads can attend to different sets of tokens. This helps preserve the diversity of GQA rather than forcing all heads to share the same sparse attention.

On the infrastructure side, we move from token-level selection to block-level retrieval. Compared with token-level selection, block-level retrieval significantly reduces the number of candidates and lowers the overhead of top-$k$ computation and result merging. At the same time, block-level retrieval is more hardware-friendly. It improves memory access efficiency and increases the compute-to-memory access ratio. Importantly, we also show that this design does not introduce a significant performance drop.

Finally, we design efficient kernels for both training and inference. I won't go into the kernel details here today, but this is the system-level component that makes the sparse attention design practical and efficient. With this design, we were able to extend the training of our M3 model to 1 million context very efficiently.

As we talked about earlier, the third pillar is native multimodality. Our model, MiniMax M3, was trained from scratch with multimodal data. What does that mean? Our model was trained with both text and vision understanding from step zero, which is a very different approach from current open-weight models.

Our starting point is very clear: at the current stage, a standalone vision-language model has relatively limited application scenarios. But combining it with text understanding during training can be challenging. The key question and challenge was: how can we introduce multimodal capability without hurting the core text performance?

To answer this, we explored several training strategies at scale. The first and most common approach is to add multimodal training after the pre-training decay stage, which is essentially a CPT-style approach. However, at that point, the model has already largely converged. Adding multimodal data at this stage tends to interfere with the learned text capability and causes degradation.

Another approach is to introduce multimodal training before the pre-training decay stage—for example, after the model has already seen a certain amount of text tokens. We also tried this direction, but found that the results were highly sensitive to the data mixture and training hyperparameters, such as learning rate. In one setting, a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to another run. More importantly, during large-scale pre-training, these parameters cannot be arbitrarily changed just for multimodal training.

This led us to our final strategy: native multimodal training from scratch. In this setting, multimodal capability is introduced from the very beginning rather than being added after the text model has already formed its structure. Under the same text token budget, we find that this approach almost does not hurt text capability.

We actually did many fun analyses during these experiments. One of them is this interesting observation from the attention maps. You can see from this picture that for CPT-style multimodal models, the text tokens often pay very little attention to visual tokens. The visual token regions are mostly dark in the attention map, which suggests that the multimodalities are not fully integrated.

In contrast, with native multimodal training, the attention map looks much more integrated. We do not see a clear separation between text tokens and visual tokens here. This suggests that the model learns to fuse visual and textual information more naturally from the beginning. Overall, our training strategy is not just about adding vision capability; it is about integrating multimodality into the model while preserving its strongest text capabilities.

Of course, we did many experiments on multimodal architecture choices. Our overall architecture is relatively standard: the image or video is processed by a ViT, which produces visual tokens. These visual tokens are then inserted into the language model input by replacing the original image placeholders in the text sequence. After that, the visual embeddings and text embeddings are fed together into the language model.

One important finding is that applying 3D attention inside the ViT is very helpful for vision understanding. In a standard ViT setting, attention is applied within each individual image or frame. For video inputs, this means that visual patches from different frames do not directly interact at the ViT stage. But with 3D attention, patches across different frames can attend to each other directly. This gives the model stronger temporal and cross-frame visual modeling capability.

We found that this improves visual understanding, and the benefit also transfers to some image benchmarks, suggesting that it strengthens the overall visual representation rather than only improving video-specific tasks.

One practical detail is that we cannot simply extend 3D attention to an unlimited number of frames, because that would introduce additional system and load-balancing challenges. In our exploration, using four frames gave us the best trade-off between performance and efficiency.

Overall, with our architecture design, with the training strategies that we explored throughout the stages, and with our analysis and ViT explorations, we were able to train the model from scratch with multimodal understanding. That concludes my presentation today. Thank you.