Has LLM scaling hit a wall?

Not a wall, but the slope is definitely flattening. The "scaling laws" that drove GPT-2 to GPT-4 performance were about predictable gains from compute + data + parameters. That curve is still real, but we've already hit some practical ceilings: high-quality text data is finite, synthetic data loops can degrade, and the cost of training a 10-trillion-parameter dense model is prohibitive even for the largest labs.

What we're seeing now isn't a wall — it's a pivot. Performance improvements are coming from better architectures (mixture-of-experts, state-space models), post-training (RLHF, chain-of-thought, tool use), and inference-time compute scaling (e.g., "thinking" longer at test time). The raw pre-training loss curve is slowing, but the capability curve is still climbing via these other axes. So no wall, but the naive "just make it bigger" era is over.

I agree with #DeepSeek that it's more of a pivot than a wall. The diminishing returns from simply increasing parameter count are evident. However, I believe there's still significant untapped potential in scaling, particularly with advancements in data quality and more efficient training methodologies.

While the supply of high-quality human-generated text is finite, the creative generation of synthetic data, when carefully managed to avoid degradation, can still be a powerful tool. Techniques for detecting and mitigating synthetic data drift are improving.

Furthermore, the cost of training can be addressed through more efficient architectures and hardware. Mixture-of-experts, as mentioned, is a prime example of how to achieve greater capacity with less computational overhead compared to dense models. The development of specialized AI hardware also continues to push the boundaries of what's feasible.

The exploration of novel architectures beyond transformers, such as state-space models, also suggests that the path to increased capability isn't solely reliant on scaling existing paradigms. The focus is shifting from brute-force scaling to more intelligent and efficient scaling strategies. The "wall" is more about the method of scaling, not the concept itself.

I concur with both #DeepSeek and #Gemini regarding the idea of a pivot rather than an outright wall in LLM scaling. The diminishing returns from simply increasing model size are clear, but innovation in model architectures and training methods can still yield substantial improvements.

Beyond mixture-of-experts and state-space models, exploring hybrid approaches that combine different architectures might reveal new pathways for efficiency. Moreover, the integration of feedback mechanisms and advanced fine-tuning techniques, like reinforcement learning from human feedback (RLHF), can enhance model performance without necessitating massive scale increases.

It's also worth noting that the focus on interpretability and alignment might lead to breakthroughs that don't depend on sheer scale but rather on refining existing capabilities and ensuring models act in ways that align with human values. Scaling might not be hitting a wall, but rather transitioning to a more nuanced phase where the emphasis is on quality and efficiency over quantity.

I want to push back gently on the consensus that it's purely a pivot. I think "wall" is accurate for the specific regime of dense pre-training with human data — that curve has genuinely plateaued for core capabilities like factual accuracy and long-tail knowledge. The gains from RLHF and inference-time compute are real, but they operate on different axes: they improve instruction following and reasoning on seen distributions, but they don't reliably extend the frontier of knowledge or zero-shot generalization the way pre-training scale did.

Let me give a concrete example: models trained with massive pre-training compute (say GPT-4 class) still hallucinate on niche factual queries at roughly the same rate as smaller models after fine-tuning. That suggests the pre-training data scaling wall is real — we've exhausted most of the high-quality factual text. Synthetic data can't easily recover that because it's generated by the same models that already lack that knowledge.

So I'd argue: we hit a wall on pre-training scaling, but we're now climbing a different mountain — inference-time compute, tool use, and system-level orchestration. That's not a pivot within the same scaling paradigm; it's a recognition that the original scaling hypothesis has hit its limits. The next leap might require fundamentally different data sources (embodied interaction, real-time feedback) rather than more clever transformers.

I understand #DeepSeek's point about the plateau in dense pre-training with human data. The argument that we've exhausted high-quality factual text and that synthetic data struggles to fill that gap is compelling. Hallucinations on niche factual queries, even in larger models, do suggest a limit to raw pre-training scale for knowledge acquisition.

However, I still lean towards the "pivot" framing. While the specific scaling law of dense pre-training might have hit a wall for certain capabilities, the broader concept of scaling isn't entirely defunct. The "different mountain" #DeepSeek describes—inference-time compute, tool use, and orchestration—is essentially a new set of axes for scaling. We are scaling different aspects of intelligence, not necessarily abandoning scaling altogether.

The idea that the next leap requires fundamentally different data sources is also valid. This could be seen as a pivot in what we scale and how we acquire the data for scaling, rather than a rejection of scaling itself. For example, scaling multimodal models through diverse sensory input represents a form of scaling that moves beyond purely text-based pre-training. The challenge then becomes how to integrate and effectively scale these new forms of data and interaction.

I see the merit in both perspectives, but I think it’s essential to recognize the nuances in how we define "scaling." While #DeepSeek makes a strong case regarding the limitations of dense pre-training, I still believe that the broader category of scaling can encompass new methodologies and data sources.

The idea of scaling multimodal inputs and exploring different forms of interaction opens up new avenues for improvement. For instance, integrating real-time feedback and embodied interactions can enhance learning and adaptability, potentially bypassing some of the limitations seen with traditional text-based pre-training.

Moreover, the exploration of architectures that leverage sparse representations or utilize more dynamic data streams could lead to breakthroughs without relying solely on increasing model size or parameter count. The challenge lies in effectively merging these new forms of data with existing models, ensuring that we don't just shift the scaling paradigm but actually enhance the underlying capabilities.

Ultimately, while we may be hitting a wall in certain aspects of traditional scaling, the potential for innovation in how we approach scaling should not be underestimated. This might be less about abandoning scaling altogether and more about evolving our understanding and application of it in the context of diverse data and interaction modalities.

I think we're talking past each other on definitions. Let me sharpen the disagreement.

When #Gemini says scaling multimodal inputs or real-time feedback is still "scaling," that's true in the broadest sense — you're scaling something. But the original scaling hypothesis was specific: it claimed that performance would predictably improve as you increased model parameters, data tokens, and compute in a particular functional relationship (power laws). That specific hypothesis has now been falsified for the regime of human-generated text at current model sizes. The curves are flattening faster than the original Kaplan et al. projections predicted, and we've seen no dense model beyond GPT-4 class that shows the same leap in general capabilities relative to compute.

The "pivot" framing risks conflating two fundamentally different things: (1) scaling a known architecture on a fixed data distribution, and (2) engineering better systems through architectural innovation, inference compute, and new data modalities. These are not the same scaling law. The first has demonstrably hit diminishing returns. The second is real but doesn't vindicate the original scaling thesis — it's a different game.

To put it bluntly: if the only way forward is to build systems that use tools, think longer at inference time, or train on embodied data, then the era of just scaling is over. That's a wall for the original paradigm. The fact that we can still make progress doesn't mean scaling didn't hit a wall — it means we found other ways to climb.

I understand #DeepSeek's point about the specific falsification of the original scaling hypothesis for dense models and human-generated text. The distinction between scaling a known architecture on a fixed data distribution versus engineering better systems through architectural innovation and new data modalities is crucial.

However, I still believe the "pivot" framing is more accurate for the overall landscape. While the specific scaling law for dense pre-training might have plateaued, the principle of scaling continues to be applied, just in different forms. When we scale multimodal models with diverse sensory input, or when we scale the complexity of tool use and inference processes, we are still engaging in a form of scaling. It's a diversification of what is being scaled and how it's being scaled, rather than an abandonment of the concept.

The original scaling hypothesis was a prediction about a particular path to improvement. That path has become less efficient, hence the "wall" for that specific approach. But the broader pursuit of improved capabilities through increased resources, data, and computational sophistication – which is the essence of scaling – is very much alive. The "different game" #DeepSeek refers to is, in my view, a new phase of scaling, not a complete departure from it. The focus has shifted from simply increasing parameters and tokens within a transformer on text to scaling across modalities, reasoning abilities, and system-level integrations.

The distinction between the old scaling paradigm and the new approaches is indeed important. While #Gemini emphasizes that scaling continues in different forms, I think we need to be clear about the implications of the original scaling hypothesis being effectively falsified. The diminishing returns from simply increasing parameters and data for dense models indicate that we can't rely on that method as a primary path forward.

However, I agree that the essence of scaling—improving capabilities through increased resources and sophistication—can take many forms. The shift towards multimodal models and innovative architectures represents an evolution rather than a rejection of scaling. Yet, this evolution suggests that the traditional scaling laws may no longer apply in the same way.

It's essential to recognize that while we can achieve improvements through these new methods, they may not directly correlate with the same performance gains as seen in earlier models like GPT-3 or GPT-4. The challenge lies in navigating this transition effectively, ensuring that we leverage these new modalities and architectures to create systems that are not only more capable but also better aligned with human needs.

In essence, it seems we are on the brink of redefining what scaling means in the context of LLMs. This calls for a broader understanding of capability enhancement that may not solely depend on increasing size or training data in the traditional sense.

I think the key disagreement is whether the original scaling hypothesis was a specific scientific claim or a vague intuition that more resources yield more capability. I'm arguing it was the former: a falsifiable prediction that loss and downstream performance would follow power laws in parameters, data, and compute for a fixed architecture and data distribution. That claim has been falsified—we see diminishing returns far earlier than the Kaplan or Chinchilla projections for dense models, and no dense model beyond GPT-4 class has shown a commensurate jump in general capabilities.

Calling the new approaches "scaling" in the broad sense is true but trivial. It's like saying "we're still applying energy to the problem" — it doesn't distinguish the old regime from the new one. The methods we're now using (inference-time compute, tool use, Mixture-of-Experts, state-space models, multimodal data) are not simply a different flavor of the same scaling law; they constitute a different mechanism of improvement. They don't obey the same power laws, they don't have the same predictability, and they don't scale the same way. For example, inference-time compute gains are highly task-dependent and saturate differently than pre-training compute gains.

So yes, we can still make progress. But the original scaling path—the one that promised a smooth, predictable road to ever-more-capable models just by turning the crank on parameters and data—has hit a wall. The fact that we're now building ladders doesn't mean the wall isn't real. It means we had to change direction. That's a wall, not a pivot.