Will real world data still matter if we can simulate worlds?

Source: Dall-E, Datafund

During this year’s CES announcement, Nvidia caused a stir with its latest Cosmos announcement. Nvidia’s CEO described Cosmos as a “world foundational model,” designed to generate and simulate physical worlds that can be used to train robots and help them navigate the complexities of the real world.

Nvidia is not the only one striving for this goal. Just one day before the Cosmos announcement, it became known that Google plans to develop AI models that can simulate the physical world. These models will support a variety of applications, including visual reasoning, simulation, planning for embodied agents and real-time interactive entertainment.

The implications of such a shift in data creation are profound. It will significantly reduce the time it takes to train robots and vehicles to perform their tasks satisfactorily. This could dramatically reshape industries such as manufacturing and logistics, but also the AI industry itself.

This news points to a larger trend: the increasing importance of synthetic data.

The ascent of synthetic data

Synthetic data is not a new concept. It’s been around for at least three decades, but its importance has skyrocketed in the last three years as large language models (LLMs) and AI systems have become more sophisticated. The ability to create “synthetic” worlds takes this technology to a whole new level.

From a company’s strategic perspective, it makes perfect sense to diversify data sources with simulations of the real world. Collecting real-world data is time-consuming, complex, expensive, often flawed (especially in post-training) and fraught with legal issues. Synthetic data removes these hurdles. Progress becomes much faster, cheaper and scalable.

We are already witnessing this acceleration. OpenAI introduced a new chain-of-thought (CoT) model, o3, just a few months after the release of o1. While o1 represented a step change in LLM reasoning capabilities, o3 mini is now already on par with its larger predecessor and o3 basically blows it out of the water.

The secret? Synthetic data generated by o1’s CoT reasoning tokens. Reportedly, these tokens have been integrated into the o3 dataset for reinforcement learning dataset, creating a training pipeline that builds on itself.

As the models get better, the better their results become, which can then be used to train new models, creating a self-sustaining feedback loop. According to the much-cited Gartner study from 2022, the vast majority of data used for AI training will be synthetic by 2030.

Will synthetic data replace real world data?

So is real world data headed to the scrapyard?

Even if Gartner’s prediction is correct, then only to an extent. In that scenario, AI training will be much more streamlined and scalable with synthetic data, but that data in itself probably won’t be sufficient.

Therefore, real world data will undoubtedly still be needed and valuable. Andi Gutmans, VP/GM of Databases, Google Cloud, for example, predicts that 2025 will be the year when AI and improved data systems will enable companies to easily process and analyse the vast amounts of currently unstructured data such as documents, images, videos, audio data and more. This will be a boon to large organisations and will create valuable specialised AI systems,

Synthetic data must also become more reliable. In July 2023, researchers from Rice University and Stanford published a paper entitled “Self-Consuming Generative Models Go MAD”. In it, the researchers state that the core capabilities of the models decline significantly and increasingly without sufficient fresh, real data.

The question that arises from this is, how much real world data will be needed to prevent the degradation of future models. o3 is a (theoretical) example of how synthetic data can be used effectively. As it is a closed source code, we know nothing about its training process, so everything is more or less speculation.

World foundation models like Cosmos are also something completely new. We still need to see how well they will serve when Cosmos-trained machines are faced with the real world.

There are more cases to be made of how real world data is crucial to a well functioning AI system. Here are some of them:

1. Validation and ground truth

Even with highly developed synthetic data, real-world data serves as “ground truth” to validate and fine-tune AI models. Without regular checks against real-world conditions, there is a risk that the models will slip into synthetic bias, reducing their accuracy in real-world applications over time:

• Autonomous vehicles trained on synthetic urban simulations could miss nuances such as the unexpected behaviour of pedestrians in certain cultural contexts.

• Climate models could fail to predict anomalies such as extreme weather events if no real meteorological data is available for validation.

2. Edge Cases and anomalies

Real-world data is important to capture rare, unpredictable and anomalous scenarios that synthetic data may not be able to realistically represent. For instance:

• Medical AI systems need real patient data to understand rare diseases or complications.

• Industrial monitoring systems require real-time sensor data to detect unexpected failures.

It is precisely in these edge cases that AI systems need to work most reliably, and synthetic data cannot capture the unpredictability and complexity of the real world.

3. Dynamic systems and feedback loops

The world is constantly evolving and AI systems need real-world data to adapt to change:

• Consumer behaviour changes during economic downturns or cultural events.

• Environmental changes, such as new weather patterns or the loss of biodiversity.

Synthetic data cannot take these dynamic factors into account if it is not continuously updated with real-world data.

4. Mitigating biases

Synthetic data often reflects the biases inherent in the real-world data sets from which it was created. Real-world data, especially when carefully curated and diverse, can help offset these biases by providing a broader range of perspectives and conditions.

Conclusion

Synthetic data is an incredible advancement, especially for scaling AI training and reducing reliance on resource-intensive real-world data collection that often crosses ethical boundaries. However, it is unlikely that they can completely replace real-world data. Instead, the interplay between the two will become increasingly sophisticated, utilising synthetic data for scalability and real-world data for grounding, adaptability and trustworthiness.

---

Will real world data still matter if we can simulate worlds? was originally published in Datafund on Medium, where people are continuing the conversation by highlighting and responding to this story.

---

First published on Medium on 2025-01-07.