The Range of Government Qualitative Data: From Public Comments to Internal Reports

By analyzing qualitative information, federal agencies establish stronger connections with constituents and stakeholders, fostering better understanding of their needs and concerns.

In program evaluation, qualitative methods help diagnose factors contributing to successes and shortcomings of initiatives. They can help troubleshoot why certain programs might not achieve intended outcomes, providing detailed understanding of implementation processes and experiences of those involved. This is particularly important for comprehending contextual elements that influence program results, such as geographical characteristics of a region, community attributes, and complex relationships across different government levels.

Understanding the distinction between public and internal qualitative data is fundamental to grasping its full significance within the federal context. Both ends of this spectrum are essential for a well-functioning government, with public feedback shaping government action and internal data optimizing efficiency and effectiveness of existing operations.

Public Comments and Federal Rulemaking

Public participation is a fundamental pillar of the federal regulatory process. Federal law mandates that agencies publish proposed regulatory changes in the Federal Register and provide opportunities for citizens and organizations to offer feedback through public comments.

This process begins when an agency issues a Notice of Proposed Rulemaking (NPRM), a formal announcement informing the public about intended regulatory changes and soliciting input. The NPRM represents a critical juncture in the rulemaking process, providing a platform for stakeholders to voice opinions, share expertise, and raise concerns before a regulation is finalized.

The primary channel for submitting public comments is Regulations.gov, the official federal electronic comment portal. This online platform provides a centralized location where the public can access information about proposed rules and submit feedback electronically. Federal agencies also accept comments through their own websites, traditional mail, and during public meetings or hearings to ensure accessibility and encourage broad participation.

The importance of public comments is underscored by the Administrative Procedure Act (APA) of 1946, which legally requires federal agencies to consider all "relevant matter presented" in submitted public comments and address these concerns in the notice published when the regulatory change is finalized. This legal obligation highlights the significant role that public input plays in shaping federal regulations.

The typical timeframe for public comment periods ranges from 30 to 60 days following NPRM publication, although agencies may extend this period for more complex rulemakings. In addition to NPRMs, agencies may also use Requests for Information (RFIs) at earlier stages of policy development to gather broad information and perspectives before specific rules are proposed. Furthermore, the public can comment on Information Collection Requests (ICRs), which pertain to federal forms, surveys, and other information collection mechanisms.

Federal agencies must meticulously review and consider every comment submitted during the designated comment period. This process involves agency staff reading and analyzing feedback to understand concerns raised, expressions of support, and suggestions for improvement.

Public feedback can significantly impact the final regulatory outcome, as agencies may make substantial changes to proposed rules based on comments received or even withdraw rules entirely if compelling arguments against them are presented. The Administrative Procedure Act further mandates that agencies provide responses to all substantive and relevant comments when publishing the final rule in the Federal Register.

These responses explain how the agency considered various points raised in public comments and whether those comments led to modifications in the final regulatory text. Agencies analyze comments to identify specific concerns, understand perceived benefits, and gather concrete recommendations for improvement.

Comments most effective in influencing outcomes are those providing specific feedback on particular sections of the proposed rule, supporting arguments with evidence such as data or research, and offering clear, actionable suggestions, including specific alternative language when appropriate.

Given that significant rulemakings can generate overwhelming volumes of public comments—sometimes hundreds of thousands or even millions of submissions—agencies may employ specialized comment analysis tools, including Natural Language Processing (NLP) technology, to efficiently review and respond to this feedback.

The Department of Transportation (DOT) recognizes the critical role of public input in its regulatory and policy development processes, utilizing platforms such as the Federal Register and Regulations.gov to solicit feedback.

The DOT periodically reviews existing regulations and proactively seeks public comments on which rules most need review and potential revision, demonstrating commitment to maintaining a relevant and effective regulatory framework. The agency particularly emphasizes seeking public feedback on guidance documents with significant economic impacts, acknowledging the importance of stakeholder perspectives on documents with substantial consequences for industries and the economy.

Qualitative Data in Federal Internal Reports

Within federal agencies, qualitative data plays a pivotal role in internal reports. These documents serve as crucial tools for informing decision-making processes, evaluating effectiveness and impact of ongoing programs, and contributing to formulation and refinement of long-term strategic plans. The qualitative information provides deeper understanding of operational dynamics and stakeholder experiences from an internal perspective.

Internal qualitative feedback offers insights into stakeholder perceptions about the agency, their direct experiences with services provided, and expectations for future interactions. This type of data functions as an early warning system, alerting agencies to potential issues in service delivery and helping focus attention and resources on areas where improvements in communication, staff training, or operational processes might enhance overall quality of products and services.

In specialized sectors, such as financial regulation, the Federal Reserve's qualitative assessment of capital plans exemplifies the use of internal qualitative data. This assessment evaluates detailed analyses and methodologies employed by financial institutions to determine appropriate levels and composition of capital reserves under various economic stress scenarios, leveraging existing supervisory information to ensure institutions maintain robust capital planning processes.

The objective behind collecting and analyzing qualitative data for internal reports is often to achieve continuous improvement in service delivery and enhance program management effectiveness within the agency. Given its internal focus and often sensitive nature, this type of data is typically not intended for public release, underscoring its primary role in facilitating internal reflection, assessment, and enhancement of operations.

The Department of Health and Human Services (HHS) extensively uses qualitative data to understand programs and diverse populations it serves. Administrative data, routinely collected as part of operational processes, often contains valuable qualitative information about individuals and their interactions with services.

In program evaluation, qualitative research provides insights from participants' perspectives regarding implementation, outcomes, and overall impact on their lives. By incorporating unique experiences of individuals served by programs, evaluators ensure findings and recommendations reflect reality and program effects.

Public Comments vs. Internal Reports: Key Differences

Public comments and qualitative data in internal reports represent distinct yet valuable forms of federal qualitative information. These differ significantly in intended audience and primary purpose.

Public comments are primarily generated by external stakeholders—individuals, organizations, and other governmental entities—and directed toward federal agencies to influence development or modification of proposed regulations and policies. Their fundamental purpose is to advocate for specific outcomes, provide feedback on potential impacts of proposed rules, and ensure public concerns and perspectives are considered in the federal rulemaking process.

In contrast, qualitative data in internal reports is typically generated and utilized by individuals and teams within federal agencies. The intended audience is primarily internal staff, agency management, and leadership. The core purpose is to inform and support internal decision-making across various functions, including evaluating effectiveness of agency programs and operations, and contributing to formulation and refinement of the agency's strategic planning initiatives. The insights are geared toward enhancing the agency's internal functioning, improving service delivery, and ensuring strategic goals are effectively pursued.

The level of detail and specificity also distinguishes these two types of qualitative data. Public comments exhibit a wide spectrum of detail, ranging from broad statements expressing general support or opposition to highly detailed analyses of specific regulatory provisions, often referencing relevant legal precedents or scientific evidence. The depth provided often depends on the commenter's expertise, interests, and the aspects of the rule they address.

Qualitative data in internal reports tends to be more focused in scope and requires greater detail directly relevant to the agency's day-to-day operations, specific programs, or interactions with defined stakeholder groups. For example, an internal evaluation report on a customer service program might include verbatim customer interview transcripts exploring specific service interactions or detailed narratives illustrating particular service delivery challenges.

Collection methodologies also differ. Public comments are typically submitted voluntarily by individuals and organizations through publicly accessible channels. While agencies offer guidance on effective commenting, the rigor and methodological approach of individual commenters varies significantly.

Internal qualitative data collection often employs more controlled and systematic methodologies designed and implemented by the agency itself. These methods include structured or semi-structured interviews with specific target groups, focus groups with carefully selected participants, usability testing of agency interfaces, and systematic analysis of existing internal documents. Agencies employ specific qualitative analysis techniques, such as thematic coding, memoing, and comparative analysis, to ensure rigor and consistency in data interpretation.

Finally, the types of insights gained and their application tend to be distinct. Public comments primarily offer insights into public opinions and perspectives, potential regulatory impacts on various sectors and communities, and suggestions for modifications or alternative approaches from outside the agency. These insights primarily inform and potentially shape the final content and direction of federal regulations and policies.

Conversely, qualitative data in internal reports typically yields insights more directly related to the agency's internal workings and immediate stakeholders' experiences, such as service customers or beneficiaries. These insights often highlight operational challenges, effectiveness of current service delivery models, areas for internal process improvement, and perceptions of agency staff and management. The application of these internally generated insights focuses primarily on enhancing the agency's overall efficiency, effectiveness in achieving its mission, and service quality.

The Wider Landscape of Federal Qualitative Data

Federal agencies frequently engage directly with key stakeholders through interviews to gain in-depth understanding of specific issues. These interviews allow for more nuanced exploration of perspectives than written submissions alone. Agencies often convene technical expert panels, bringing together specialists to review policy options, provide feedback, assess potential benefits and drawbacks of different approaches, and consider broader consequences of proposed policies. These expert consultations constitute valuable qualitative data, grounded in research and practical experience.

Even historical records maintained by federal entities, such as the FBI's archives known as "The Vault," serve as rich repositories of qualitative data, containing interview transcripts, investigator observations, and case summaries offering unique glimpses into past events and perspectives.

Focus groups represent another significant federal qualitative data source. These facilitated group discussions generate qualitative information revealing shared experiences, common perceptions, and attitudes toward specific topics. Federal agencies increasingly undertake broader community engagement initiatives to collect qualitative data reflecting diverse community perspectives nationwide. These initiatives ensure voices of various populations, including historically marginalized or underrepresented groups, are considered in federal policy and program development and implementation.

A notable example is the Federal Reserve's Worker Voices Project, which utilized focus groups to understand experiences and perspectives of workers and job seekers navigating the labor market. This project demonstrates how qualitative data can directly inform economic policy decisions through firsthand accounts from individuals affected by labor market conditions.

While surveys often collect quantitative data through closed-ended questions, they can also gather rich qualitative data through open-ended questions. These responses allow participants to articulate thoughts, feelings, and reasoning in their own words, offering insights that numerical responses cannot provide. Many information sources, such as market research surveys and analyses by institutions like the Pew Research Center, incorporate qualitative data from open-ended survey questions to provide context and deeper understanding of quantitative findings, illuminating the "why" behind statistical trends.

Transcripts of government meetings, hearings, and other formal proceedings constitute another essential qualitative data form. These written records capture spoken words, discussions, and arguments during official events, providing detailed accounts of decision-making processes and diverse participant perspectives. Government websites may host audio and video recordings of such meetings, offering even richer qualitative data including non-verbal cues and tonal nuances.

Challenges and Limitations of Federal Qualitative Data

A fundamental challenge in working with qualitative data is the potential for bias and subjectivity to influence both collection and analysis. Individual perspectives, pre-existing assumptions, and personal interpretations can inadvertently shape data gathering, coding, categorization, and ultimate conclusions. The interpretation of narratives, identification of themes, and discernment of patterns within qualitative data are not always clear-cut and can vary depending on the analyst's viewpoint. This inherent subjectivity raises questions regarding overall reliability and validity of findings from qualitative inquiry.

Ensuring objectivity and maintaining rigor throughout qualitative analysis requires meticulous adherence to methodological best practices. This includes developing and consistently applying standardized coding frameworks, strategically triangulating findings across multiple data sources to enhance corroboration, and practicing reflexivity to acknowledge and address potential biases.

Qualitative research often generates substantial volumes of unstructured data, particularly from open-ended survey responses, in-depth interview transcripts, and user-generated social media content. The sheer quantity of information presents significant management and analysis challenges. Federal qualitative data encompasses diverse formats, including textual documents, audio and video recordings, and visual media like photographs and maps. Analyzing these diverse formats requires different analytical skills and specialized tools, adding to the complexity of working with this data type.

Synthesizing insights from multiple qualitative data sources, potentially representing a wide range of perspectives, can be particularly demanding, requiring careful organization, systematic coding, and thoughtful interpretation to identify overarching themes and arrive at meaningful, well-supported conclusions.

When federal agencies collect qualitative data from diverse stakeholders or the general public, they encounter varied and sometimes contradictory viewpoints and experiences. Synthesizing these disparate perspectives into a cohesive understanding of a particular issue or phenomenon can be notably complex. Identifying common threads and recurring patterns across different narratives, while acknowledging and considering divergent viewpoints, demands a careful, nuanced analytical approach.

Researchers must be sensitive to the broader context shaping these perspectives and any underlying power dynamics influencing different opinion expressions. The challenge lies in creating a synthesis that accurately reflects the full spectrum of experiences and opinions without oversimplification or inadvertently marginalizing certain voices or perspectives. This requires thoughtful interpretation and commitment to representing the inherent complexity of collected data.

Best Practices for Government Qualitative Data

To ensure integrity and trustworthiness of federal qualitative data, agencies should establish clearly articulated, specific research questions. Subsequently, they should develop comprehensive coding frameworks directly aligned with these questions, providing a structured analytical approach. Consistency and reliability in data collection are paramount, requiring standardized procedures for all methods, whether through interview protocols, focus group facilitation guides, or observation protocols for field research. Personnel involved in data collection should be adequately trained in these standardized procedures.

To enhance qualitative research findings' credibility, agencies should employ triangulation, corroborating findings through multiple data sources or different collection methods examining the same phenomenon. Another valuable practice is member checking, where participants review researchers' interpretations and provide feedback on accuracy and completeness. Maintaining a detailed, well-documented codebook with clear code definitions and explicit application rules promotes consistency and transparency throughout the coding process. Adherence to ethical data collection practices outlined in relevant guidelines and regulations is crucial, including obtaining informed consent, protecting participant privacy and response confidentiality, and committing to responsible, ethical data use.

Federal agencies should strategically select and apply qualitative data analysis techniques best suited to their specific research questions and unique data characteristics. Thematic analysis involves systematically identifying recurring themes, patterns, and key insights across the dataset. For analyzing large volumes of textual information, content analysis can be valuable, focusing on systematic examination to identify specific keywords, underlying concepts, and recurring language patterns. When data comprises personal stories and narratives, narrative analysis can provide deep insights into individual experiences and perspectives.

To enhance efficiency and rigor in analysis, federal agencies should consider leveraging qualitative data analysis software tools, which offer features to assist with organizing, coding, and analyzing large, complex datasets. Investing in staff training on effective software utilization is crucial to maximizing their potential.

To achieve more comprehensive understanding of issues addressed, federal agencies should actively seek opportunities to integrate qualitative findings with quantitative data when feasible. Qualitative data provides valuable context and helps explain statistical trends and numerical patterns from quantitative analysis. Adopting mixed-methods research approaches, strategically combining qualitative and quantitative data collection and analysis techniques within single studies, leverages inherent strengths of each approach, leading to more holistic understanding of complex phenomena.

The rich insights from qualitative data should be actively incorporated into policy formulation and program design. Deep understanding of stakeholder lived experiences and diverse perspectives can lead to development of more effective, equitable, and impactful interventions. Federal agencies must effectively communicate qualitative findings by sharing them with relevant stakeholders and decision-makers within the agency and, where appropriate, with the broader public.

Clear, compelling, well-supported communication of qualitative insights can play a vital role in informing and influencing both policy and practice at all government levels.

AEM's AI team stands out for our expertise in realizing the benefits of human-in the-loop approaches in deep learned systems, and we offer capabilities across a range of traditional ML areas. Contact us at ai@aemcorp.com to explore challenges your team is facing.