Why We’re Building Our Own Question-Answering Dataset (And Why It Matters)

By [Ksenia Kapoor](/content/author/kkapoor "link-to-author"/index.html)
February 25, 2026
3 min read

Working at the forefront of AI solutions for government applications, we’re constantly pushing the boundaries of what’s possible with language models. But to truly understand how well these models perform, we need reliable ways to evaluate them. This led us to an interesting challenge: creating our own question-answering dataset.

The SQuAD Dilemma

You might wonder why we’re not simply using SQuAD, the popular Stanford Question Answering Dataset. While SQuAD is widely recognized as a benchmark dataset, there’s a catch: most modern language models are trained on it. Using SQuAD to evaluate these models would be like giving students a test they’ve already seen the answers to. We needed something different — something that would truly test these models’ capabilities.

A Hybrid Approach: Combining AI and Human Intelligence

After extensive research, we’ve landed on what we believe is the optimal solution: a hybrid human-in-the-loop approach. Recent research suggests that the quality of machine-generated questions could approach human-level when properly filtered. In practical terms, this means organizations could potentially create high-quality QA datasets with less reliance on expensive and time-consuming human annotation, while still maintaining performance standards comparable to manually created datasets.

Here’s how our approach works:

  1. First, we use language models to generate focused, specific questions based on our internal documents. This leverages AI’s ability to process vast amounts of information and generate diverse questions quickly.
  2. Then, human experts step in to provide and validate the answers.

Why Human Answers Matter

While it might be tempting to let AI handle both questions and answers, having human experts provide the answers helps ensure dataset quality and accuracy. By involving humans in the loop, we significantly reduce hallucination risks while maintaining high answer accuracy. Recent research has shown that even advanced language models can generate plausible but incorrect information, with hallucination rates ranging from 20% to over 90%. Human experts, by contrast, can provide nuanced, factually correct responses while catching subtle inaccuracies that AI might miss.

Yes, involving human experts makes the process more expensive and slower than a fully automated approach. However, we believe this investment is crucial for creating a truly reliable evaluation dataset. We strategically focus human expertise on areas where research shows it’s most crucial — providing accurate answers and validating domain-specific content.

Building for Specialized Domains

Beyond the quality assurance benefits, our approach offers another key advantage: domain specialization. By using our own internal documents and human experts, we’re creating a dataset that reflects specialized knowledge and terminology relevant to our work. This is crucial because general-purpose datasets often fall short when it comes to specialized fields. Domain-specific QA datasets will serve as valuable benchmarks for assessing the capabilities of new and emerging LLM models by revealing how well these LLMs perform in specific contexts.

Looking Forward

This hybrid approach isn’t just about creating a dataset — it’s about building a foundation for better AI evaluation. As language models continue to evolve, having reliable, domain-specific ways to test them becomes increasingly important.

We’re excited about the potential of this approach and its ability to help us better understand and improve AI systems. By combining the efficiency of AI with human expertise, we’re creating something that’s greater than the sum of its parts.