How MATT3R’s Embodied AI Data Engine powers the future of scenario generation and analysis

Published: June 2025 | Author: Hamid Abdollahi, CEO @ MATT3R

Foundation Models (FMs) have redefined what’s possible across language, vision, and reasoning. Now, they’re entering the driver’s seat.

A new survey by researchers at TUM and Stanford (Gao et al., 2025) dives deep into the role of foundation models in scenario generation and analysis for autonomous driving. With over 330 papers reviewed, it’s the most comprehensive study yet on how large language models (LLMs), vision-language models, diffusion models, and world models are transforming autonomous vehicle (AV) validation.

However, while the models are impressive, the paper highlights a significant bottleneck: data.

Scenarios don’t emerge from thin air.

They emerge from embodiment.

To train, prompt, or fine-tune these models, we need more than just text prompts or static datasets; we require richly embodied, temporally grounded, multi-agent, and multimodal interaction data. In other words, we need the right kind of driving data.

That’s where MATT3R comes in.

MATT3R: The Embodied AI Data Engine for FMs

At MATT3R, we’re building a real-world-to-foundation-model bridge. Our edge computing platform ingests real-world telemetry, perception, and behaviour signals directly from fleets in motion, then reconstructs those moments as high-fidelity digital twins.

This isn't just about storing driving data. It's about turning raw sensory inputs into:

• Scenario scripts

• Agent trajectories

• Scene graphs

• SD maps

• Visual reconstructions

• Semantic metadata

• Natural language summaries

All composable, controllable, and queryable; the exact primitives needed to train, prompt, or fine-tune FMs for driving.

What the Paper Reveals and What It Signals

The paper highlights a shift from rule-based and handcrafted scenario testing to model-driven, generative pipelines. It catalogues LLMs generating test cases from crash reports, diffusion models simulating edge cases, and world models enabling predictive policy simulation.

Yet it also points out key limitations:

• Lack of closed-loop, interactive scenario generation

• Poor integration across modalities (e.g., text, image, map, LiDAR)

• Overemphasis on rare corner cases, underemphasis on routine driving

• No standard pipeline for regulatory alignment or data-to-scenario transformation

These are precisely the gaps MATT3R is filling with an embodied data infrastructure that spans from edge collection to scenario orchestration, all within a single engine.

From Logging to Prompting

Here’s the vision:
Instead of just logging what happened, we use that data to prompt what could happen next.

Imagine this:

• Our K3Y device detects a real-world disengagement or intervention.

• That moment is reconstructed and tagged in the software.

• Our Embodied AI Engine converts it into a language-annotated scenario.

• It’s used to fine-tune a world model’s behavioural forecasting.

• Or to test a new LLM’s risk reasoning capabilities.

• Or to generate thousands of variations in simulation via diffusion pipelines.

All with traceability back to the original moment.

Why This Matters

Foundation models aren’t just getting bigger; they’re getting hungrier. And the next generation of driving intelligence won’t come from synthetic data alone. It will come from models grounded in the messiness, complexity, and diversity of real-world driving.

MATT3R is positioned to be the data substrate that feeds these models.

We’re not just collecting logs. We’re engineering intelligence fuel.

Let’s Build the Foundation Together

If you're building scenario generation frameworks, autonomous stacks, or fine-tuning AV foundation models, we’d love to collaborate.

Reach out if you're interested in:

• Plugging your simulator into our generated scenes

• Training multimodal LLMs with grounded trajectory narratives

• Testing your world model on long-tail real-world edge cases

• Converting driving data into scalable training assets

MATT3R is your foundation for foundation models.