Agencies are increasingly bringing AI into their analytical workflows, drawn by its promise to handle larger datasets, streamline processes, and speed up research. But while the role of analysts is changing, their contribution to meaningful insights is as important as ever.
Capabilities and Constraints in Qualitative Domains
Many federal organizations are using or testing AI tools for various analytical tasks—adopting these technologies more quickly than their state and local counterparts.
Government agencies use qualitative analysis to:
• Run more effective performance audits and evaluations
• Make sense of what happened during disasters and other critical incidents
• Figure out if government services are actually working for people
• Add depth to cost-benefit analyses by capturing impacts that can't be quantified
• Improve data collection by understanding what people need
Agencies are now using AI in several practical ways:
• Using natural language processing (NLP) to sort through comments from surveys and chat transcripts and organize them by topic
• Deploying generative AI to digest long reports and present the key points in easy-to-understand dashboards
• Applying AI to make sense of employee survey responses and spot emerging trends
• Implementing AI chatbots that can generate text, images, and code to streamline communication and documentation
• Leveraging AI-enhanced document review systems to efficiently process mountains of text
• Using specialized qualitative analysis tools that can automatically identify themes, analyze sentiment, and transcribe content
• Automating the transcription of interviews and focus groups
AI Strengths in Qualitative Analysis
When it comes to qualitative analysis, AI brings some impressive capabilities to the table:
• It can work through large datasets in a fraction of the time it would take humans
• It handles the grunt work—transcribing interviews, coding data, and generating summaries—freeing analysts to focus on more challenging tasks
• It can apply the same criteria consistently across all data, potentially reducing certain types of bias
• It can spot patterns in unstructured data that humans might miss
• It can gauge the emotional tone of thousands of comments or responses
AI Limitations in Qualitative Analysis
But let's be realistic—today's AI still has significant blind spots when it comes to qualitative analysis:
• It often lacks real-world context
• It lacks an understanding for use cases it was not trained extensively on
• It can make hidden assumptions that are not visible to the end user
• It can "hallucinate," generating convincing but completely made-up information
• It can miss emotional cues, cultural context, and social signals
• It can propose an answer without enabling the user to easily double-check it
AI tools are valuable, but they're not enough on their own. Human analysts bring a depth of expertise and capabilities that no algorithm can currently match. Here's what makes human involvement essential:
Human analysts don't just identify patterns—they evaluate information deeply and systematically. They can break down complex problems, spot subtle interconnections, and question underlying assumptions that might otherwise go unchallenged.
When intelligence analysts work with information, they blend structured reasoning with intuitive thinking—a combination that AI simply can't replicate yet. Good qualitative analysis requires self-directed, disciplined thinking that can monitor and correct itself as it goes—mental muscles that humans have developed over lifetimes of navigating social complexities.
Humans excel at catching the subtle shades of meaning in qualitative data. We naturally grasp language nuances, interpret tone and intent, and pick up on unspoken cues that often carry crucial meaning.
We can read between the lines of what people say, placing their experiences in context to uncover deeper explanations for why people behave as they do. Humans also naturally follow the flow of stories and understand how events connect and influence each other—creating a rich, holistic understanding that current AI simply can't match.
Humans naturally place information in its broader context. We instinctively consider the social, cultural, historical, and personal factors that shape experiences and behaviors—connections that algorithms typically miss.
Human-driven analysis puts real human experiences at the center, enabling truly meaningful interpretation. Our capacity for empathy lets us genuinely understand and value different perspectives and cultural backgrounds, creating a deeper connection with the data. We can provide what anthropologists call "thick description"—accounts rich with context that make findings meaningful rather than merely factual.
Real-world data is messy. When information is unclear, incomplete, or contradictory—as it often is—humans can draw on experience and judgment to make reasonable decisions amid uncertainty.
Unlike algorithms with fixed patterns, humans can pivot their approach when surprising information emerges. We can recognize the significance of rare events that don't fit the pattern but might have major implications. This flexibility and intuition for handling the unexpected are uniquely human strengths in navigating the inherent messiness of qualitative data.
Leveraging AI and Human Expertise
The future isn't about AI replacing humans—it's about thoughtful collaboration. The real opportunity lies in using AI as a "superpowered assistant" that handles routine tasks while human analysts focus on the work that demands human judgment and creativity.
This "augmented intelligence" approach recognizes that AI works best when it enhances human capabilities rather than trying to replace them. Here's how that partnership is taking shape:
Effective and often complementary models for this "human-in-the-loop" partnership are already emerging:
• Humans take the lead on research questions and study design, while AI handles data collection and initial processing like transcription and first-pass coding.
• AI does the heavy lifting on initial coding and theme identification across large datasets, with humans reviewing and refining these outputs to ensure they make contextual sense.
• Human analysts provide the critical context and subject-matter expertise to guide AI's analysis and keep it aligned with policy objectives.
• AI flags outliers and contrasting viewpoints that might otherwise get lost, which humans can then explore in depth.
This partnership plays to each side's strengths. AI brings speed, scale, and consistency, while humans provide depth, context, and ethical judgment. Together, they produce richer, more useful insights than either could achieve alone.
Challenges and Opportunities
Bringing AI into federal qualitative analysis isn't simple—it comes with both real hurdles and exciting possibilities. Here's what agency leaders need to know:
Several significant hurdles stand in the way of successful AI adoption:
• Talent gaps: Agencies struggle to attract and keep AI professionals, while many existing analysts lack AI literacyDespite the challenges, AI integration offers some compelling benefits:
• Doing more with less: AI can handle routine tasks automatically, freeing up analysts for the complex work that really matters
• Tackling data mountains: AI can process volumes of qualitative data that would overwhelm human-only teams
• Speeding up discovery: AI can rapidly spot patterns and surface insights that might take humans weeks to identify
• Improving procurement: AI can help evaluate proposals more thoroughly and consistently
• Making better decisions: Analysis of comprehensive datasets leads to more informed decisions across government functions
• Serving citizens better: More efficient analysis means agencies can respond more quickly to public needs
• Boosting transparency: AI can provide clearer data trails that show how decisions were reached
• Creating responsive services: Agencies can leverage AI to tailor services that better address what people actually need
Evolving Roles in a Human-AI Ecosystem
The AI landscape is changing rapidly, and these advances will reshape federal qualitative analysis. Here's what's on the horizon:
• AI and machine learning will get much better at understanding nuance in text, catching subtleties that current systems miss
• We'll see more user-friendly AI tools built specifically for qualitative researchers—no computer science degree required
• Generative AI will take on bigger roles in summarizing findings and potentially even suggesting insights that humans might overlook
In response to these advancements, the role of human expertise in federal qualitative analysis will likely evolve:
• Analysts may shift their focus towards higher-level tasks such as formulating complex research questions, designing rigorous study methodologies, interpreting nuanced findings, and ensuring ethical considerations are thoroughly addressed
• Human analysts will need to develop expertise in critically evaluating AI-generated outputs, ensuring their validity, reliability, and alignment with the specific goals of the analysis
Despite these evolving roles and the increasing capabilities of AI, human judgment will remain indispensable in several key areas of federal qualitative analysis:
• Providing nuanced domain expertise that the AI model lacks
• Navigating unforeseen factors and adapting to ambiguous or contradictory information
AEM's AI team stands out for our expertise in realizing the benefits of human-in the-loop approaches in deep learned systems, and we offer capabilities across a range of traditional ML areas. Contact us at ai@aemcorp.com to explore challenges your team is facing.