CIVIC TECH ASSESSMENT RESOURCES 4: QUALITATIVE AND QUANTITATIVE DATA

4. Qualitative and Quantitative Data

Quantitative data are essential for tracking the scope and scale of your platform’s use and outcomes. Qualitative data, based on in-depth interviews, focus groups and/or ethnographic observation, are helpful when you need to go deeper into a question – to understand the “how” and the “why,” as well as “how many.” In many cases it makes sense to combine quantitative and qualitative evidence. Quantitative research will give you the numbers, while qualitative research can surface underlying factors and motivations that can help you understand the drivers behind numerical results. Soliciting stories or testimonials from users about their experiences on your platform can also identify outcomes that you may not have anticipated – both positive and negative. Your performance measures can then be adjusted to monitor for these results. Qualitative and quantitative data may be generated through your platform directly or obtained from other sources. Listed below are some common sources of data used in civic tech assessment.

Data Collection and Outcome Analysis

Following is a closer look at one outcome – users engage more in civic life – to illustrate potential approaches to data collection and analysis. Sample survey questions for each of the civic tech objectives reviewed earlier can be found in section 6 of Additional Tools & Resources.

Example: Do users engage more in the civic life of their community as a consequence of their participation on the platform?

To create metrics that help to track your progress, you can use survey data alone or in combination with other kinds of data. For example, to track whether users attribute increased levels of civic engagement to the use of your platform, you could ask this question on a user survey:

In the last 12 months, has your level of involvement in any of these activities INCREASED as a result of your use of [platform]? [Scale: No, did not increase at all; Yes, increased a little; Yes, increased a lot]

  • Attended a public meeting in which local issues were discussed

  • Contacted an elected representative about a local issue outside of OG

  • Worked with other residents to make change in the local community

  • Donated money to help a local organization

  • Signed a petition

  • Performed local volunteer work for any organization or group

The results of your survey will tell you the percentage of users that attributes changes in behavior to platform use, offering important testimony to the impact of your platform. You may also be interested in knowing which types of users are experiencing the greatest impact, such as low-income users or local business owners. To build a dashboard of key metrics, incorporate the specific measures you need to track for the target users most critical to your goals. For example:

Combine survey data with online data to learn even more…

Once you have offline data for a key metric in hand, you can combine it with online data to learn even more about how your platform affects your users. For example, using the outcome measure above, compare platform data describing low-income user activity to outcomes documented through survey research to see if there are any correlations between online and offline activity. Over time, you may also see patterns in your outcome metrics that will allow you to reduce the amount of data you need to collect to assess ongoing impact. For example, let’s say that in addition to asking users if their involvement in certain civic engagement activities has increased, you also ask them if they feel more confident that they can effect change in their community as a result of using the platform:

How much impact do you think that people like you can have in making [local geography] a better place to live? (Please check one.)

A big impact
A moderate impact
A small impact
No impact at all

As additional analysis, identify what percentage of low-income users report that they are more confident that they can effect change, and then look at survey data describing actions for these same users, and see if higher levels of user confidence correlate with higher levels of user action. If there is a strong correlation you may conclude that user confidence is a good predictor of user civic engagement, and therefore opt to focus on a single key performance indicator related to user confidence going forward.