We Still Don't Know How People Use AI

We Still Don't Know How People Use AI

Every few months, a headline tells us what people "really" do with ChatGPT or Claude. Almost all of those headlines trace back to the same place: usage reports published by the AI companies themselves. MIT Technology Review's latest piece on a research project called the AI Observatory makes an uncomfortable point — we have almost no independent data about how people actually use these systems, and the little we have suggests the company reports are missing a lot.

The AI Observatory

The AI Observatory was co-led by Anka Reuel, a Computer Science PhD candidate at Stanford's Trustworthy AI Research Lab, and Shayne Longpre, a recent PhD graduate from the MIT Media Lab, with researchers from MIT, Stanford, the Data Provenance Initiative, and other institutions.

The scale is modest by industry standards but unprecedented for independent research:

All conversations were collected with user consent through seven existing research datasets, covering ChatGPT, Gemini, Claude, Grok, and dozens of other models. The goal is simple: give researchers and policymakers a source of usage data that does not come from the marketing department.

For comparison, Anthropic's Economic Index is based on 1 million Claude conversations and OpenAI's usage report analyzed 1.5 million ChatGPT conversations. The observatory's dataset is a drop in that bucket — but it is one of the only buckets researchers are allowed to look into at all.

What the Company Reports Leave Out

The most cited number here comes from the Anthropic Economic Index, which focuses on work- and productivity-related uses of Claude. When the Observatory team applied Anthropic's filtering methodology to their own dataset, they found that 48% of conversations would have been filtered out.

And the filtered half looks very different. Compared to Anthropic's published analysis, the non-work conversations were far more likely to involve:

• Health and relationships: 44.2% vs 31.2%
• Harassment and hate: 27.5% vs 5.66%
• Sexual content: 16.7% vs 2.4%
• Adult or illicit topics: 7.9% vs 2.1%

OpenAI's own 2025 report quietly admitted the same thing from the other direction: only about 30% of consumer ChatGPT use was related to work. So when a policymaker cites "how people use AI" from a company report, they are often citing a deliberately narrow slice of reality.

"No single company report tells the whole story," as Longpre put it.

Usage Changes Over Time

Because the datasets span 2023 to 2025, the Observatory could watch behavior drift. In WildChat, one of the largest datasets included, conversations got longer and more elaborate over time — more prompt tokens, longer responses, more turns per conversation.

Small talk increased significantly, which the researchers read as rising AI companionship. Meanwhile, the rate at which assistants disclosed being chatbots went down. Those two trends together explain a lot about the companion-AI discourse that dominated 2025.

One genuinely encouraging signal: exchanges labeled sensitive — including sexual harassment and hate speech — became less frequent over time. That may indicate platforms deployed more effective safeguards, though the researchers are careful not to overclaim.

Every Model Has Its Own Culture

The most interesting finding for anyone who works with multiple models: usage patterns differ dramatically between them.

People used Grok and Gemini more for information retrieval. Grok was especially popular for news and politics — and also where misinformation tended to concentrate. Coding questions flowed toward Anthropic. Gemini attracted social and roleplay uses. ChatGPT was the homework assistant.

Even different versions of the same model developed different communities. Conversations were shorter on GPT-3.5 and longer, more iterative ones on GPT-4o — which tracks with everything we learned about that version's tendency to create emotional attachment.

If you are building products on top of these APIs, this matters. Your users' expectations, prompt styles, and failure modes will partially depend on which model culture they migrated from.

Why This Matters for Developers

There is a general lesson here about telemetry, and it applies to any product you build: whoever controls the filter controls the story. You can see the effect with a tiny simulation:

import random

random.seed(42)

def sample_conversation():
    kind = random.choices(["work", "personal"], weights=[30, 70])[0]
    sensitive = random.random() < (0.05 if kind == "work" else 0.20)
    return kind, sensitive

pop = [sample_conversation() for _ in range(100_000)]

# What the full population looks like
all_sensitive = sum(s for _, s in pop) / len(pop)

# What a "work-only" report sees
work = [(k, s) for k, s in pop if k == "work"]
work_sensitive = sum(s for _, s in work) / len(work)

print(f"sensitive share, all traffic:    {all_sensitive:.1%}")
print(f"sensitive share, work-only cut:  {work_sensitive:.1%}")

Run it and you get roughly 15.5% versus 5.0%. Same underlying population, opposite conclusions — depending entirely on which slice you publish. That is structurally what happens when a company reports only on the conversations its methodology keeps.

This is also why human review of automated claims stays essential, something we argued in our piece on human review steps in AI pipelines. Aggregated dashboards flatten exactly the variance that matters.

The Data Problem Is Not Going Away

To be fair to the Observatory, its own data has limits. Because participation was voluntary, sensitive uses are probably underrepresented — people are less likely to consent to sharing those conversations. The researchers say so themselves.

But the asymmetry remains stark: the labs hold billions of conversations and publish curated summaries, while independent researchers fight over tens of thousands of consented samples. As Reuel put it, anyone making decisions based on current AI usage data risks "completely operating in the wild."

Until the companies open their data — with real privacy protections — every "here's how people really use AI" headline deserves the same question: measured by whom, and filtered how? If you build agents yourself, the closest you get to ground truth is still your own instrumentation. Tools like Goose and Claude Code generate logs you control; treat them as your private observatory.

Sources

• MIT Technology Review: How people use AI