AI Model Costs Push Startups Toward Cheaper Open-Weight Models as Harvey's Margins Plunge

Harvey's Dramatic Margin Collapse Exposes AI Model Economics Crisis

Harvey, the legal AI startup valued at $15.6B, saw its gross margin plummet from approximately 50% at the start of this year to minus 50% by June, according to 1

1

. The dramatic collapse came after customer usage of its AI agents spiked following a March update, exposing the unsustainable economics of relying on closed AI models from providers like OpenAI and Anthropic1

1

. The company experienced a twentyfold rise in token usage costs this year, as both major providers shifted from flat subscriptions to token-based billing that charges enterprises for model usage on top of base fees1

1

.

The crisis forced Harvey to fundamentally rethink its approach. In August, the company released its first in-house model, post-trained on Kimi K3 from Chinese lab Moonshot1

1

. Gross margins turned positive again after the launch, combined with other changes to how the startup uses AI1

1

. The turnaround demonstrates how AI model costs have become an existential threat for startups building on proprietary infrastructure, pushing many toward the shift toward self-built models despite significant upfront investments required.

Rising Costs of Proprietary AI Models Trigger Industry-Wide Shift

Harvey isn't alone in confronting these economics. The rising costs of models from large AI labs have been driven by two key factors: the shift from chatbots to agentic AI, which consumes more computing power, and the transition to token-based billing2

2

. This pricing evolution has made vendor lock-in a critical concern for startups that built their businesses on rented AI capabilities.

Abridge is now building a clinical model on Nvidia's open weights, while Decagon routes 80% of customer queries through models of its own1

1

. Ramp, which raised $750M in June, is weighing in-house training for the first time. "It made absolutely no sense a year ago," Ramp co-chief executive Karim Atiyeh said. "It's starting to make a lot more sense now"1

1

. Major investors including Sequoia Capital and General Catalyst are actively funding this transition, recognizing that trade-offs between open and closed AI have shifted dramatically in favor of open-weight models1

1

.

Strategic Autonomy Versus Infrastructure and Talent Requirements

Open-weight models offer compelling advantages beyond cost. They enable firms to create their own models with their own data, cutting the risk of outsourcing critical technology to another company2

2

. This strategic autonomy has become increasingly valuable as providers demonstrate their willingness to terminate relationships. OpenAI moved to end its Cursor contract after SpaceX bought the company, citing past terms of service breaches by Musk businesses1

1

. Mistral's Arthur Mensch spent July arguing that closed models give providers immense leverage, pointing to Anthropic cutting off Windsurf while building Claude Code1

1

.

However, the economics aren't universally favorable. Some companies have found that building on open-weight models doesn't work due to higher upfront costs and the need for specialized talent, computer infrastructure, and sufficient proprietary data2

2

. Self-hosting requires computing capacity, storage, cybersecurity controls, monitoring tools and skilled employees, while proprietary closed platforms typically bundle these responsibilities into their price2

2

.

Hybrid Approaches and Data Privacy Concerns Shape Future Strategy

Most startups adopting cheaper open-source models aren't abandoning proprietary providers entirely. Harvey still relies on Anthropic's Opus for its hardest tasks, as Anthropic noted in a slide shown at an investor forum1

1

. AT&T has pioneered a hybrid approach, routing employee queries to cheaper models when appropriate, cutting costs for coding and advanced AI tasks by as much as 56% while seeing performance quality decline by only 2%2

2

. The company aims to increase the share of queries powered by open-source models from the current 40% to between 60% and 70% in coming years2

2

.

Data privacy concerns complicate the picture, particularly around Chinese open-weight models. While Chinese labs can charge less than U.S. companies due to more efficient models and China's lower energy costs, some companies' clients worry about data privacy when using these alternatives2

2

. Europe's Legora, a Stockholm-based Harvey competitor that reached $100M in revenue inside 18 months serving more than 1,200 firms, has never disclosed whose models it runs on1

1

.

Not everyone believes the shift matters long-term. "Having your own model or not is such the wrong question," said Menlo Ventures partner Matt Kraning. "In most cases, it tends to be a lot of cosplay"1

1

. Yet as token usage continues climbing and providers consolidate power, startups face mounting pressure to control their AI infrastructure. Watch for more companies to follow Harvey's path, balancing the economics of open-weight models against the capabilities only closed AI models currently provide.