“The amount of compute needed to train Llama 4 will likely be almost 10 times more than what we used to train Llama 3, and future models will continue to grow beyond that.”

In context

“optimistic about this is that we have a long track record of success with open source. We've saved billions of dollars with open compute project by having supply chains standardized on our infrared designs. Open sourcing tools like PyTorch and React has led to real benefits for us from all the industry's contributions. This approach has consistently worked for us and I expect it will work here too. Another major area of focus is figuring out the right level of infra capacity to support training more and more advanced models. Llama 3 is already competitive with the most advanced models, and we're already starting to work on Llama 4, which we're aiming to be the most advanced in the industry next year. We are planning for the compute clusters and data we'll need for the next several years. The amount of compute needed to train Llama 4 will likely be almost 10 times more than what we used to train Llama 3, and future models will continue to grow beyond that. It's hard to predict how this trend -- how this will trend multiple generations out into the future. But at this point, I'd rather risk building capacity before it is needed rather than too late, given the long lead times for spinning up new inference projects. And as we scale these investments, we're of course, going to remain committed to operational efficiency across the company. The last area that I want to discuss is how AI is shaping our metaverse work, which continues to be our other long-term focus. Last quarter, I discussed how advances in AI have pulled in the timelines for some of our products. A few years ago, I would have predicted that holographic AR would be possible before Smart AI, but now it looks like those technologies will actually be ready in the opposite order. We'r”