Developed with the Center for Evaluation Innovation this two-part guide to network evaluation includes a brief that outlines the frameworks, approaches and tools to address practical questions about designing and funding network evaluations and aCasebook that provides profiles of nine evaluations.
Download at: www.networkimpact.org/networkevaluation
The civic tech field has expanded so widely in recent years, it’s hard to think of a major city or an area of civic life that these technologies don’t touch. In this dynamic environment, the John S. and James L. Knight Foundation has been a field leader, investing over $25 million since 2010 in projects ranging from neighborhood forums, to civic crowdfunding platforms, to efforts that promote government innovation. For eighteen months, Network Impact worked with Knight Foundation grantees and other civic tech leaders to find out how they measure success, focusing on tools they’re using to track platform performance and assessment challenges they face along the way.
We started by identifying key outcomes related to these common civic tech objectives and gathered case examples of assessments from the field:
- Build place-based social capital
- Increase civic engagement
- Promote deliberative democracy
- Support open governance
- Foster inclusion and diversity
Our work also led us to think about tracking the performance of a platform through its lifecycle – recognizing that assessment priorities vary with stage of development, from early testing of a minimum viable product to later-stage scaling of a tested concept.
The result of this research: two guides to evaluating civic tech that summarize assessment best practices, including leading methodologies and metrics that can help innovators monitor progress towards their goals and evaluate the impact of their efforts. Some of these assessment best practices focus on connections between users, both online and off-line, with an important network dimension.
Assessing Civic Tech: Case Studies and Resources for Tracking Outcomes is a publication of the Knight Foundation with Network Impact that focuses on measuring the impact of civic tech platforms on people, places, and processes.
How To Measure Success: A Practical Guide to Answering Common Civic Tech Assessment Questions is a Network Impact publication that offers examples and advice for monitoring a platform’s ongoing performance using tools and approaches that are effective and practical.
Additionally, the Knight Foundation wrote up their key lessons from investing in civic tech that are also worth a read.
How Code for America is using the Assessing Civic Tech guide
The release of this guide is coming at the right time. Demonstrations of what is possible are up in running in communities of every size across the United States. Now we need to find out not only what works, but what works best over time.
At Code for America, the guide will be particularly helpful for Fellowship teams and volunteer Brigades who are thinking about the questions they need to ask and the changes in attitudes they need to measure to assess progress towards increasing civic engagement and open governance. The process and case studies documented in this guide will be useful for structuring these assessments.
At Code for America, we believe that it is critically important to identify the residents, community groups, or government staff who will be using the particular public service program or benefit, then work with them early in the assessment design process. This guide provides important examples of how to frame an evaluation to include and work with intended beneficiaries. It offers sample questions and resources that will be very helpful to organizations and individuals who are beginning to explore how they can include measures of civic engagement and changing attitudes in their assessment of their efforts.
Connecting to Change the World builds on an earlier resource that Pete and I developed called Net Gains. This latest collaboration with John Cleveland includes examples and lessons that have emerged from our work with social impact networks over the last decade or so. During that time, we’ve been introduced to many new networks and deepened our work with others. As a consequence, we have a better understanding of what makes some networks highly “generative.” By generative, we mean networks with a renewable collaborative capacity to generate numerous activities simultaneously. These are networks that activate members’ connections on an emergent basis as need and opportunities arise.
Examples in the book include RE AMP – more than 165 nonprofit organizations and foundations in eight Midwestern states working together on climate change and energy policies, Reboot- a network of young Jewish American “cultural creatives” who are exploring and redefining Jewish identity and community in the U.S. and the U.K., ten regional networks of state agencies and nonprofit providers that have organized to end homelessness in Massachusetts, and five regional and two national networks of rural-based organizations that are promoting public policies that benefit rural communities in the U.S. In all of these networks, members have been very deliberate about creating, strengthening and maintaining network ties in order to establish a base of connections from which many activities can arise at the same time or over time. This foundation is the starting point for the progression from connecting to aligning to production or joint action that we also discuss in the book.
Net Gains provides practical advice for the growing community of network builders developing networks for social change. The handbook draws from the experiences of network builders, case studies covering a diversity of different networks, and emerging scientific knowledge about “connectivity.” The guide is divided into four parts, each focusing on a specific element of network building and offering strategies for successful development of networks at different stages in their evolution, from the moment of their inception, to the management of their ongoing production.
The handbook can be downloaded here.
Taking your network’s temperature regularly is easy—and helps to inform continuous improvement of the network’s effectiveness.
When the 14 organizations in the Southwest Rural Policy Network met in November 2010 to discuss how well their network was doing, they didn’t just share their latest impressions. They had data stretching back nearly a year and a half. Since June 2009, as part of their formal work plan, they had self-assessed the network five times using a Network Health Scorecard. The assessment covered four essential categories: the network’s purpose, performance, operations, and capacity. The process only takes a few minutes after the network’s quarterly meeting—but reveals a great deal about how network members judge the network.
In November, Joyce Hospodar, the network member who chairs the network’s evaluation committee, summarized the scores—on a scale of 1-5, with 1 being low/5 being high—for the past five quarters: Purpose scores were holding steady. Performance scores peaked the previous spring. Operations and Capacity hovered around 4.0, but dipped recently.
Interestingly, when network members also scored where they thought the network’s health was compared to a year earlier, the ratings were all substantially higher than at the outset.
The scorecard is a tool, one source of evaluative feedback a network can use to gauge how well it’s doing and what sort of improvements might be useful. “Note,” says network coordinator Mikki Anaya, “this assessment only measures one aspect of the SWRPN’s effectiveness—the capacity/organizational efforts of the network.” A different evaluation will look at the network’s policy advocacy activities.
The one we developed has a total of 22 questions divided into the four categories. Several networks have adapted the questions to better reflect the specifics of their network. But in any case, the evaluative process is the same:
- Identify key indicators of the network’s well-being
- Regularly collect data from the members
- Analyze the data and share it with members
- Determine what changes are needed
Kudos to the members of the SW Rural Policy Network for picking up on this tool and incorporating its use into their network practice.
Answering some basic questions about the network can yield a useful diagnosis.
As I’ve designed networks and coached network builders, the question always comes up: How can we know how the network is doing? In some ways, the answer is complicated. There are different types of networks and networks evolve through different stages–factors that should be taken into account. But it’s also true that some things about networks hold for any network at any stage of life.
Working with colleagues at Cause Communications and nuPOLIS, and testing ideas with the marvelous networks of Rural People Rural Policy, an initiative of the W.K. Kellogg Foundation, we’ve developed a handy Network Health Scorecard. With just 22 questions and a 1-5 scoring system, it focuses on key aspects of any network: purpose, performance, operations, and capacity. It’s designed for group use–network members answer each question and then discuss their answers–or on your own.
Try it and post comments to let us know how it worked for you so we can improve this tool for network builders.
Madeleine Taylor is the lead entrepreneur at Network Impact and a principal in Arbor Consulting.