Model Safety Open-Source Threshold

The asymmetry

Deploying a model as a product, through an API or a consumer app, lets a lab retain content filtering to restrict outputs in dangerous domains, monitoring to detect misuse and shut it down, rate limits to prevent bulk generation of harmful output, and access controls to revoke a bad actor's access. Open-sourcing the weights gives up all of these permanently: anyone can run the model locally, strip its safety tuning, fine-tune it toward harmful objectives, or integrate it into applications the original lab has no visibility into. This asymmetry has always existed, but frontier models have now crossed a capability threshold in several genuinely dangerous domains, making the risk of losing that control material rather than theoretical.¹

A closed-sourcing decision, explained

Alexandr Wang laid out this reasoning explicitly when Meta Superintelligence Labs closed-sourced Muse Spark, around April 2026, despite the company's prior history of releasing models as open weights through the Llama family. "When we launch a model like Muse Spark in a product, we have a lot of ways to mitigate some of these risks and ensure that we're able to launch it in a safe and responsible way. It's much harder to do that when you open source a model and people can use that model in all sorts of contexts that we may not have full understanding of."¹ What triggered the decision was the model tripping a high-risk classification during early training on the company's own capability-scaling framework, in particular a bio-risk category among several other elevated ones, with a preparedness report published documenting the findings.¹ Wang says the pattern is not unique to Meta: "We certainly aren't the only ones to see a host of these risks show up as we scaled up the models and kept pushing the frontiers of research."¹ The decision rule that follows is narrow and specific: tripping a high-risk classification early in training is the signal that open-source release is off the table for that model, with product deployment plus mitigations as the alternative, not withholding the model from release altogether. Open-source release is still planned for models specifically designed to be fit and safe to open source while retaining as much capability as possible, implying a parallel effort to build deliberately open-sourceable variants with the dangerous capabilities reduced or hardened against misuse.¹

This differs from a separate category of restriction, refusing to let a technology be applied to a specific use case such as domestic mass surveillance or fully autonomous weapons regardless of deployment method. The open-source threshold concerns the release mechanism rather than the use case: it does not restrict what a user can do with a product's API, it restricts whether the weights ship publicly at all, and the same model can be available through a product with mitigations while being off the table for an open-weights release. The two kinds of restriction are compatible and can be applied by the same lab simultaneously, for different reasons.

The economic reason to withhold weights

A complementary, purely economic rationale for withholding frontier weights comes from a different vantage point, Clay Bavor's, and helps explain the gap between the American and Chinese open-model ecosystems. If a lab's own hyperscaler and frontier-model business faces essentially unbounded, margin-rich demand for the best available intelligence, shipping open weights close to that frontier competes directly with the lab's own highest-margin product: "If I was running that business, that's not something I would do."² The argument for withholding operates independently of any bio-risk or cyber-risk reasoning. Part of the reason Chinese open models are unusually advanced is, in Bavor's account, a willingness to do large-scale distillation of frontier models, meaning many open-weight models originating from China are themselves derived from training runs originally done in the United States: if an organization cannot afford to build a frontier model outright, "the next best approach is to distill them and offer them up."² Open weights become a distillation product built on someone else's frontier run rather than an independent frontier effort in their own right.

Put together, the safety rationale and the economic rationale point in the same direction, withhold frontier weights, for different reasons, and together explain the observed pattern where the labs with the most frontier capability tend to release lagging open weights while the genuinely frontier-adjacent open models more often arrive through distillation, often of runs originally done elsewhere.

Tensions

Who decides what counts as high risk is an open question, since a capability-scaling framework of this kind is typically proprietary, the threshold is self-assessed by the lab training the model, and there is no external verification or agreed industry standard, which is a meaningful accountability gap. Building a variant that is simultaneously capable and safe to open source is also a genuinely hard technical problem, since capability cannot easily be subtracted from a set of weights without degrading it broadly, so a claim that this is achievable is plausible but largely untested. There is a real speed-versus-safety tradeoff too, since open-sourcing has historically accelerated the broader research ecosystem, so a closed release slows down everyone who might otherwise have built on the weights safely, meaning the threshold represents a real cost and not just caution. And the decision is version-specific rather than permanent: a sufficiently safer or smaller model may clear the threshold even when a larger version of it does not, producing a two-track release strategy of product-only access for the frontier and open weights for controlled variants beneath it.