Synthetic Data in 2025: Revolutionizing GenAI Model Performance

Synthetic data generation has advanced significantly, with larger language models now being leveraged for knowledge distillation to produce high-quality training examples. It enables the creation of domain-specific datasets that can be used to finetune smaller or specialised models while preserving accuracy, lowering computational costs, and reducing existing data requirements. For instance, this approach enables efficient finetuning of GPT-4o, Llama etc. for specific use cases with just a few synthetic examples, or training specialized ModernBERT models for classification tasks within your agentic workflow - achieving better performance on downstream tasks while requiring only a fraction of the computing resources. This is particularly valuable for high-throughput, latency-sensitive applications like LLM routing or real-time classification tasks, especially in agentic workflows where multiple models are used in sequence - each round of model inference compounds the efficiency benefits and reduces overall latency.

The Evolution of Data Generation Practices

1. Early Days: Naive LLM API Querying

Model Collapse
Generated samples often lacked diversity, with models falling into repetitive patterns even when prompted differently. This similarity between samples limited the training value of the synthetic datasets.

Structured Output Issues
Without proper constraints, output formatting was inconsistent across samples, making dataset integration difficult. What worked in one generation would fail in another, creating a constant need for format verification and correction.

Reliability Problems
Large-scale generation jobs were particularly vulnerable to:

• API timeouts and rate limiting
• Network connection interruptions
• Inconsistent response formats
• Incomplete generations

These issues often meant paying multiple times for the same data as jobs failed mid-way through hundreds of generations.

2. The Path to Maturity: From Quick Fixes to Robust Tools

2.1. Custom Scripts Evolution Teams developed increasingly sophisticated solutions:

• Caching mechanisms to prevent redundant API calls and preserve successful generations
• Smart prompting systems that:
  • Track and avoid duplicate examples
  • Break down complex queries into manageable chunks
  • Enforce parameter ranges for greater control
  • Post-processing pipelines for consistent data cleaning and validation

2.2. Emergence of Specialized Libraries Tools like Instructor (python.useinstructor.com) emerged to provide:

• Structured data generation
• Type validation
• Automated error handling
• Built-in best practices for synthetic data creation

2.3 Modern Tooling Platforms Advanced platforms like Curator by Bespoke Labs & Distilabel by Argilla now offer:

• End-to-end data generation pipelines
• Quality control mechanisms
• Scalable infrastructure
• Integrated monitoring and validation

Building Synthetic Datasets: A Practical Example

We believe that intent recognition is one of the core building blocks of smart AI-enabled user experiences. That’s why we’re using it to demonstrate synthetic data generation. Please find a barebones vanilla example here (Google Colab) of generating synthetic data for intent recognition. Tools and libraries make things easier by obfuscating some parts of the execution, but at the same time, they may make it harder to understand what happens behind the scenes. For that reason, we wanted to create the simplest and cleanest example possible, making it accessible even to those with less experience in the synthetic data space.

In a nutshell, in this code:

1. We install dependencies, instructor for structured generation, openai to be used as a client for generation, and some other helpers
2. Define the intents or labels we want to create queries for
3. Use Pydantic class to define our output structure
4. Create generate_queries function which takes in intent (str) and outputs 20 queries for intent, also define generate_dataset function which takes list of intents and generates queries for each and outputs list of dicts which contains query, text and some metadata
5. We turn our raw generated dataset into Huggingface Dataset and push it to Huggingface

Output dataset can be found here

We challenge you to replicate this using Curator or Distilabel and let us know which one you preferred and why.

Looking Forward: The Promise of Small Language Models

There seems to be a trend emerging: the shift toward smaller, more efficient and specialized models. We believe the future lies in using LLM generated data to train smaller focused models that can be deployed efficiently at scale (or even locally).

In our next article, we'll explore how organizations are successfully replacing large, general-purpose models with smaller, specialized ones trained on synthetic data. We'll examine real-world cases where this approach has led to significant improvements in both performance and operational efficiency.

Have you experimented with synthetic data or small language models in your work? We'd love to hear about your experiences. Share your insights in the comments below or join our community discussion about the future of AI model development.

Appendix: Can I Use Commercial Models for Synthetic Data?

In short, if you’re using a commercial LLM like Claude, be aware that while you generally own the text it generates, the Terms of Service often prohibit using that output to build a competing language model. Please consult your company's legal team to avoid potential pitfalls. It’s also worth considering an open-source LLM with a more permissive license that explicitly allows you to reuse outputs for training. It’s always better to play by the rules than to find yourself on the wrong side of a TOS.

Further Reading

For a deeper dive into these topics, check out these excellent resources:

1. "Fine-tune classifier with ModernBERT in 2025" - Comprehensive guide on utilizing ModernBERT for classification tasks
2. "Finally, a replacement for BERT: Introducing ModernBERT" - Simon Willison's detailed exploration of ModernBERT
3. "Fine-tune ModernBERT for RAG with Synthetic Data" - HuggingFace guide on combining RAG and synthetic data
4. "Synthetic data & Smol models in 2024" - Comprehensive presentation on the evolution of synthetic data and small models
5. "Synthetic Data Generation with Instructor" - Tutorial on using Instructor library for structured synthetic data generation
6. "Distilabel: Advanced Data Generation Guide" - Detailed guide on scalable data generation with Distilabel