The last decade of AI progress was built on text and images scraped from the open web. That supply is running out, and it was never the right material for the next frontier. As AI systems move into meetings, classrooms, homes and factories, they must handle what happens between people: who speaks next, when to interrupt, where to look, how to hand over an object, how a group reaches a decision. We call the field that studies and builds these capabilities Social AI. This paper defines Social AI, proposes a four-part taxonomy of human interaction (presentation, conversation, physical and social settings), relates it to the rise of physical AI, and analyzes the economics of the data it depends on. We argue that consented, multi-party, multimodal interaction data is a structurally scarce asset, that consent quality is a primary driver of its long-run value, and that a dedicated data layer is needed to serve academics, research labs and industry alike.
01Introduction
Intelligence, in people, is social before it is anything else. Children learn language by taking turns. Teams solve problems by reading each other. Most of what we call judgment is the ability to act well around other minds.
Computational intelligence has followed a different path. Large models learned from text written by individuals and images captured by individuals, then were deployed into one-on-one exchanges with a single user at a time. That design produced remarkable capabilities, but it leaves a gap exactly where human life is densest: in groups, in shared physical space, in real time.
That gap is now the binding constraint. Voice agents must manage turn-taking in calls with several people. Meeting assistants must understand who agreed to what. Robots entering homes and workplaces must navigate around, and cooperate with, the people already there. Each of these is a social problem first and a perception or generation problem second. Our thesis is simple: the future of Computational Intelligence is Social AI, and its progress will be paced by access to the right data.
02What is Social AI?
We adopt the definition proposed by Mathur, Liang and Morency, who describe Social AI as the multidisciplinary, multimodal effort to build agents that can sense, perceive, reason about, learn from and respond to the affect, behavior and cognition of other agents, whether human or artificial.1 Three properties distinguish it from adjacent fields.
- It is multi-party. Conversational AI typically models a single user and a single assistant. Social AI must model many participants with different goals, roles and perspectives at once.
- It is multimodal and time-locked. Social meaning lives in the alignment of signals: a pause before an answer, a glance toward the next speaker, a nod that says "keep going." Text transcripts discard most of it.
- It is grounded in settings. The same sentence means different things in a negotiation, a classroom and a dinner party. Context is part of the signal.
The same authors identify four core technical challenges for the field: ambiguity in social constructs, nuanced signals, multiple perspectives, and agency and adaptation in social agents.1 A follow-up survey points to multimodal alignment, group dynamics and cultural awareness as open directions.2 Every one of these challenges is, at root, a data problem: models cannot learn distinctions that their training data does not contain.
03A taxonomy of interaction
To make Social AI tractable as a data problem, we organize human interaction along two dimensions: how participation is structured (one addressing many, or many addressing each other) and where it takes place (mediated through screens, or embodied in shared physical space). This yields four interaction types.
| Type | Structure | Typical settings | Signals a model must learn |
|---|---|---|---|
| Presentation | 1 → N | Keynotes, lectures, seminars, webinars | Pacing, emphasis, audience attention and reaction, handling questions |
| Conversation | N ↔ M | Meetings, podcasts, interviews, negotiations, support calls | Turn-taking, interruptions and repair, backchannels, agreement, persuasion |
| Physical AI | Human ↔ human, human ↔ robot | Collaborative tasks, guidance, hand-offs, assistance | Gesture, gaze, proxemics, joint attention, timing of physical actions |
| Social settings | Groups in context | Gatherings, workshops, service encounters | Group dynamics, roles, norms, topic flow, inclusion and exclusion |
04Social AI and physical AI
NVIDIA's glossary describes physical AI as models that understand and interact with the real world using motor skills, and the company has framed it as the step after perception AI and generative AI: systems that perceive, reason, plan and act.3 The commercial expectations are large. Goldman Sachs recently raised its humanoid robot forecast to roughly 6.5 million units by 2035, with a market value of about $138 billion, up from an earlier $38 billion estimate.4 Longer-range estimates for the 2050 robotics ecosystem run into the trillions of dollars, though they vary widely by scope and method.5
Most physical AI work today focuses on the physics: grasping, locomotion, navigation. But the environments where these machines will create the most value, homes, hospitals, stores and offices, are full of people. A robot that can fold laundry but cannot tell when someone wants to pass, or that hands over a tool at the wrong moment, is not deployable. Social competence is the last mile of physical AI.
Physical and social AI therefore share a data requirement that neither text corpora nor robot telemetry satisfies: synchronized recordings of people interacting with each other and with machines, captured from multiple viewpoints, with the timing preserved.
05The data bottleneck
Epoch AI estimates the effective stock of high-quality, human-generated public text at roughly 300 trillion tokens and projects that frontier training will exhaust it sometime between 2026 and 2032, depending on how models are trained.6 Synthetic data and multi-epoch training can stretch the supply, but they cannot create the kind of signal a model never saw: how real groups of people coordinate in real time.
For social interaction, the shortage is far more acute, because the open web never held this data in usable form. Public video is mostly one speaker facing a camera, heavily edited, with no consent for training and no separation between speakers. The research corpora that do capture genuine interaction are small.
These datasets are valuable precisely because they were collected carefully, with consenting participants: CANDOR paid each participant up to $15 for a recorded conversation, and Meta reports that Seamless Interaction participants consented to recording.8,10 But their scale, all of them together under 10,000 hours, is orders of magnitude below what modern models consume, and most carry research-only licenses that exclude commercial use.
06The economics of social data
6.1 Why interaction data is structurally scarce
Text is cheap to produce and already exists in vast quantity. Interaction data is different on every axis that drives cost:
- It requires several people at once. Every recorded hour of a five-person meeting consumes five person-hours, and every one of those people must agree to be recorded and licensed.
- It must be captured, not scraped. The signals that matter (separate tracks per speaker, synchronized video, room context) are lost once a recording is mixed, edited and published.
- It needs expensive annotation. Turn boundaries, overlaps, gaze targets and group roles take skilled human labeling, often from multiple perspectives because participants disagree about what happened.
- Its long tail is the valuable part. Negotiations, cross-cultural calls and human–robot hand-offs are rare in any natural archive, yet they are exactly where models fail.
6.2 What drives the value of an hour
Public market signals give a floor. A survey of nearly 1,000 published dataset prices puts the median for existing video footage around $42 per hour,11 and consumer apps have paid individuals on the order of $0.30 per minute for recorded phone calls.12 Those figures describe largely single-source, lightly structured material. We model the value of an hour of interaction data as a product of multipliers on that base:
| Driver | Lower value | Higher value |
|---|---|---|
| Participants | One speaker | Three or more, each identifiable as a separate stream |
| Modalities | Mixed audio or transcript only | Per-person audio, video and transcript, plus room or egocentric views |
| Synchronization | Approximate timestamps | Frame-accurate alignment across all streams |
| Annotation depth | Speaker labels | Turns, overlaps, gaze, gesture, roles and outcomes, from multiple annotators |
| Rarity | Common settings (casual calls) | Long-tail settings (negotiation, human–robot, cross-cultural) |
| Consent quality | Unclear, one-party or inherited | Documented, all-party, revocable, covering the licensed use |
6.3 Consent is an economic asset
Consent is usually discussed as an ethical requirement. It is also a financial one. Data with weak consent carries legal risk that compounds over the life of every model trained on it. Several AI meeting notetakers now face class actions alleging wiretap and biometric violations for recording people who never agreed, with at least one case proceeding to discovery.13 When companies fail, the U.S. Federal Trade Commission has taken the position that buyers of customer data should honor the seller's original privacy policy.14
Demand for such data is real regardless: wind-down firms have brokered more than 150 sales of data and code from shut-down startups, and Google reportedly bid $10 million for an airline's internal emails and chats in bankruptcy.15 But data acquired without the consent of the people in it is a depreciating asset: it can be challenged, withdrawn or rendered unusable by regulation. Data collected with documented, all-party consent is the opposite: an appreciating asset whose value grows as regulation tightens and unconsented alternatives disappear.
07Market structure
The market for AI training datasets is still small relative to compute, but growing quickly. MarketsandMarkets projects growth from about $2.7 billion in 2024 to about $11.2 billion by 2030,16 and individual licensing agreements, such as Reddit's reported deal with Google worth about $60 million a year, show platforms treating human-generated data as recurring revenue.17 Downstream, the products that need social data are each growing into large markets of their own: AI voice agents are projected to grow from roughly $2–3 billion in 2025 to $27–44 billion in the early-to-mid 2030s,18 alongside the humanoid robot market discussed above.
Demand for Social AI data comes from three distinct segments, each with different needs:
| Segment | What they build | What they need from data |
|---|---|---|
| Academics | Theory and measurement of human interaction, benchmarks | Documented consent for ethics review, rich annotation, reproducible releases, research pricing |
| Research labs | Social, embodied and full-duplex conversational models | Scale, multimodal synchronization, long-tail coverage, evaluation sets held out from training |
| Industry | Voice agents, meeting tools, social robots, avatars | Commercial licenses, provenance for legal review, domain-specific settings, refresh over time |
Supply is fragmented across millions of individuals, teams, creators and institutions who already record interactions but have no way to license them safely. This is the classic setting for an intermediary: a data layer that aggregates supply, standardizes consent and format, and serves all three demand segments from a common, auditable foundation. We expect value in Social AI to accrue disproportionately to whoever controls consented, well-structured interaction data, much as value in the web era accrued to whoever controlled distribution.
08Open research questions
A Social AI data layer is also a research instrument. Several questions remain open, and we invite collaboration on them:
- Measuring social constructs. How should rapport, dominance or agreement be labeled when participants and annotators disagree? Multi-perspective labels, rather than a single ground truth, may be required.1
- Evaluating social competence. What benchmarks measure whether a model takes turns, repairs misunderstandings or cooperates physically as well as people do?
- Cultural coverage. Norms for interruption, eye contact and personal space differ across cultures. How should datasets be balanced, and how do models generalize across norms?2
- Privacy-preserving release. Which redaction and anonymization techniques preserve social signal (timing, prosody, gaze) while protecting identity?
- Pricing and attribution. How should value flow back to the people whose interactions train a model, especially when a single recording contains many contributors?
09Crosstalk's role
Crosstalk is built to be the data provider for Social AI. We source real human interaction across all four types in the taxonomy, from presentations and conversations to physical and social settings, and deliver it as per-participant video, audio and transcripts with consistent annotation. Four principles guide the work:
- All-party consent. Every person in every recording approves their own participation, and can withdraw at any time.
- Full provenance. Every batch ships with a manifest of source, consent and redaction status, ready for legal and ethics review.
- Interaction-first structure. Separate streams per participant, frame-accurate synchronization, and labels designed around turn-taking, attention and group dynamics.
- One foundation for three audiences. Research licenses for academics, scale and long-tail coverage for labs, commercial licenses for industry.
10Conclusion
The next generation of AI will be judged by how well it behaves around people: in a meeting, on a call, across a table, on a factory floor. That ability cannot be learned from text alone, and the open web does not contain the data needed to learn it. Consented, multi-party, multimodal interaction data is therefore the scarce input of the coming decade, and its value will be set as much by the quality of its consent as by its volume. The future of Computational Intelligence is Social AI. Building it starts with the data.
—References
- Mathur, L., Liang, P. P., and Morency, L.-P. "Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions." EMNLP 2024. aclanthology.org/2024.emnlp-main.1143
- "Towards Social AI: A Survey on Understanding Social Interactions." arXiv:2409.15316. arxiv.org/html/2409.15316v2
- "Physical AI." AI Wiki summary of NVIDIA's glossary definition and CES 2025 framing. aiwiki.ai/wiki/physical_ai
- "Goldman Sachs Just Supercharged Its Humanoid Robot Prediction 5X to 6.5 Million by 2035." Yahoo Finance, 2026. finance.yahoo.com; market value per BigGo Finance
- "Humanoid Robot Market Size: $3B Today to $5T by 2050." Robozaps, summarizing Morgan Stanley, Citi and UBS estimates. blog.robozaps.com
- Villalobos, P. et al. "Will we run out of data? Limits of LLM scaling based on human-generated data." Epoch AI, ICML 2024. arxiv.org/abs/2211.04325
- Carletta, J. et al. AMI Meeting Corpus. University of Edinburgh. groups.inf.ed.ac.uk/ami/corpus
- Reece, A. et al. "The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation." Science Advances 9(13), 2023. pmc.ncbi.nlm.nih.gov/articles/PMC10065445
- Grauman, K. et al. "Ego4D: Around the World in 3,000 Hours of Egocentric Video." CVPR 2022. cs.utexas.edu
- Meta FAIR. "Seamless Interaction: Modeling natural conversational dynamics." 2025. ai.meta.com/research/seamless-interaction
- "What AI training data sells for in 2026: medians from 973 published prices." AI Data Broker. dev.to/aidata_broker
- "Neon App pays users to record their phone calls, sells data for AI training." Malwarebytes, September 2025. malwarebytes.com
- "AI Notetaker Lawsuits 2026: Otter, Granola, Fireflies." BuildBetter. blog.buildbetter.ai
- "FTC concerned about privacy protections in 23andMe bankruptcy." Reuters via KSL, March 2025. ksl.com
- "AI labs are buying data from dead startups to train their models." MIT Technology Review, October 2026. technologyreview.com
- MarketsandMarkets AI training dataset forecast, via "The hidden economy behind AI data licensing takes center stage." Kaptur. kaptur.co
- "A $1.5 billion wake-up call for every AI company on Wall Street." MarketBeat. marketbeat.com
- Voice agent market estimates from MarketsandMarkets and Meticulous Research. marketsandmarkets.com; meticulousresearch.com
© 2026 Crosstalk. This is a v1.0 draft for discussion. Market figures come from third-party reports and press coverage that use different scopes and methods, and are cited as reported. Value drivers in Section 6.2 are a conceptual model, not a pricing commitment.