Civic Tech Assessment Resources 4: Qualitative and Quantitative Data

Go to main Civic Tech Assessment Resources page

4. Qualitative and Quantitative Data

Quantitative data are essential for tracking the scope and scale of your platform’s use and outcomes. Qualitative data, based on in-depth interviews, focus groups and/or ethnographic observation, are helpful when you need to go deeper into a question – to understand the “how” and the “why,” as well as “how many.” In many cases it makes sense to combine quantitative and qualitative evidence. Quantitative research will give you the numbers, while qualitative research can surface underlying factors and motivations that can help you understand the drivers behind numerical results. Soliciting stories or testimonials from users about their experiences on your platform can also identify outcomes that you may not have anticipated – both positive and negative. Your performance measures can then be adjusted to monitor for these results. Qualitative and quantitative data may be generated through your platform directly or obtained from other sources. Listed below are some common sources of data used in civic tech assessment.

Data Source Qualitative or Quantitative Definition
Platform data Mostly quantitative, may contain some qualitative All data collected online that describe user interaction with the platform, on every page/space and within every feature. Includes database queries that may be part of a standard monitoring dashboard or that may be undertaken at specific intervals, such as weekly or monthly, to capture user activity. Also includes basic server and website activity monitoring, if applicable, through proprietary or third-party applications, such as Google Analytics, KISS Metrics, etc.
Social media data Mostly quantitative, may contain some qualitative Varies per social media site, (e.g. Facebook, Twitter, Google+, FourSquare) typically delivers available data on users, including who is in their networks, how they are connected to them, and what information they pass on (likes, shares, retweets, keywords). Depending on user settings, can offer in-depth data on age, location, marriage status and level of education. At a minimum, offers basic network connection data. Also, offers raw text data from messages and tweets in some cases (for more, see raw text data entry below). Paired with analytic tools can also be used to track viral issues, influence, brand value and market share.
User surveys Mostly quantitative, may contain some qualitative if open-ended questions are included Typically administered online (through/on the platform or by email). Capable of capturing structured quantitative data as well qualitative data from open-ended questions. Survey data can be linked to user platform data to form a full profile of user activity, knowledge, attitudes and behaviors.
Raw text from messages, tweets, and other posts Qualitative Data from posts, interactions and conversations between users that can be analyzed through keyword algorithms and content analysis to identify issues that are trending, sentiment of discussion, and evidence of offline action. Raw text can be analyzed to track key issues, concepts and correlations between data feeds.
Independent data sets (e.g. Census data, raw data from legislatures, Civic Health Survey data, Social Capital Survey Data) Quantitative Data collected from various government departments, foundations, think tanks or other organizations that report statistics over time for various populations, processes or geographies. Can be used as proxy data to learn more about users or to compare users with the general population. In some cases, can also be used to establish correlations between platform data such as raw text from posts and other raw data, such as text from legislative sessions over the same time period.
Focus groups Qualitative Descriptive data from a facilitated conversation with a relatively small group of people (e.g., 8-12 persons). Sometimes conducted with sub-groups (e.g., youth, low-income parents), focus groups can provide in-depth information about participant experiences, perceptions, insights, attitudes or beliefs.
Interviews Qualitative Descriptive data from a one-on-one conversation that may be structured (following a strict protocol) or unstructured (a more open and evolving conversation) to gain insights into participant experiences, perceptions, insights, attitudes or beliefs.

 

Data Collection and Outcome Analysis

Following is a closer look at one outcome – users engage more in civic life – to illustrate potential approaches to data collection and analysis. Sample survey questions for each of the civic tech objectives reviewed earlier can be found in section 5 of Additional Tools & Resources

Example: Do users engage more in the civic life of their community as a consequence of their participation on the platform?

To create metrics that help to track your progress, you can use survey data alone or in combination with other kinds of data. For example, to track whether users attribute increased levels of civic engagement to the use of your platform, you could ask this question on a user survey:

In the last 12 months, has your level of involvement in any of these activities INCREASED as a result of your use of [platform]? [Scale: No, did not increase at all; Yes, increased a little; Yes, increased a lot]

  • Attended a public meeting in which local issues were discussed
  • Contacted an elected representative about a local issue outside of OG
  • Worked with other residents to make change in the local community
  • Donated money to help a local organization
  • Signed a petition
  • Performed local volunteer work for any organization or group

The results of your survey will tell you the percentage of users that attributes changes in behavior to platform use, offering important testimony to the impact of your platform. You may also be interested in knowing which types of users are experiencing the greatest impact, such as low-income users or local business owners. To build a dashboard of key metrics, incorporate the specific measures you need to track for the target users most critical to your goals. For example:

Objective Key Metric Data Collection Method
Participation in app will increase civic engagement of low-income people % of low-income users who say that participation in the app affected whether they voted in a local election, led a community effort or started a new civic initiative in their community Survey question: Has your participation in [platform/app] affected whether you did any of these things? Voted in a local election; led a community effort; or started a new civic initiative in your community

 

Combine survey data with online data to learn even more…

Once you have offline data for a key metric in hand, you can combine it with online data to learn even more about how your platform affects your users. For example, using the outcome measure above, compare platform data describing low-income user activity to outcomes documented through survey research to see if there are any correlations between online and offline activity. Over time, you may also see patterns in your outcome metrics that will allow you to reduce the amount of data you need to collect to assess ongoing impact. For example, let’s say that in addition to asking users if their involvement in certain civic engagement activities has increased, you also ask them if they feel more confident that they can effect change in their community as a result of using the platform:

How much impact do you think that people like you can have in making [local geography] a better place to live? (Please check one.)

A big impact
A moderate impact
A small impact
No impact at all

As additional analysis, identify what percentage of low-income users report that they are more confident that they can effect change, and then look at survey data describing actions for these same users, and see if higher levels of user confidence correlate with higher levels of user action. If there is a strong correlation you may conclude that user confidence is a good predictor of user civic engagement, and therefore opt to focus on a single key performance indicator related to user confidence going forward.

Network Tool: Network Health Scorecard


Regular check ups can help network builders track progress and decide what the network needs next. The Network Health Scorecard provides a quick series of questions that can yield a useful assessment of the health of your network - diagnosing strengths and areas for growth.

Stay Connected

Sign up to receive periodic updates from us with new tools, resources and articles.
  • This field is for validation purposes and should be left unchanged.