Developed with the Center for Evaluation Innovation this two-part guide to network evaluation includes a brief that outlines the frameworks, approaches and tools to address practical questions about designing and funding network evaluations and aCasebook that provides profiles of nine evaluations.
Download at: www.networkimpact.org/networkevaluation
The civic tech field has expanded so widely in recent years, it’s hard to think of a major city or an area of civic life that these technologies don’t touch. In this dynamic environment, the John S. and James L. Knight Foundation has been a field leader, investing over $25 million since 2010 in projects ranging from neighborhood forums, to civic crowdfunding platforms, to efforts that promote government innovation. For eighteen months, Network Impact worked with Knight Foundation grantees and other civic tech leaders to find out how they measure success, focusing on tools they’re using to track platform performance and assessment challenges they face along the way.
We started by identifying key outcomes related to these common civic tech objectives and gathered case examples of assessments from the field:
- Build place-based social capital
- Increase civic engagement
- Promote deliberative democracy
- Support open governance
- Foster inclusion and diversity
Our work also led us to think about tracking the performance of a platform through its lifecycle – recognizing that assessment priorities vary with stage of development, from early testing of a minimum viable product to later-stage scaling of a tested concept.
The result of this research: two guides to evaluating civic tech that summarize assessment best practices, including leading methodologies and metrics that can help innovators monitor progress towards their goals and evaluate the impact of their efforts. Some of these assessment best practices focus on connections between users, both online and off-line, with an important network dimension.
Assessing Civic Tech: Case Studies and Resources for Tracking Outcomes is a publication of the Knight Foundation with Network Impact that focuses on measuring the impact of civic tech platforms on people, places, and processes.
How To Measure Success: A Practical Guide to Answering Common Civic Tech Assessment Questions is a Network Impact publication that offers examples and advice for monitoring a platform’s ongoing performance using tools and approaches that are effective and practical.
Additionally, the Knight Foundation wrote up their key lessons from investing in civic tech that are also worth a read.
How Code for America is using the Assessing Civic Tech guide
The release of this guide is coming at the right time. Demonstrations of what is possible are up in running in communities of every size across the United States. Now we need to find out not only what works, but what works best over time.
At Code for America, the guide will be particularly helpful for Fellowship teams and volunteer Brigades who are thinking about the questions they need to ask and the changes in attitudes they need to measure to assess progress towards increasing civic engagement and open governance. The process and case studies documented in this guide will be useful for structuring these assessments.
At Code for America, we believe that it is critically important to identify the residents, community groups, or government staff who will be using the particular public service program or benefit, then work with them early in the assessment design process. This guide provides important examples of how to frame an evaluation to include and work with intended beneficiaries. It offers sample questions and resources that will be very helpful to organizations and individuals who are beginning to explore how they can include measures of civic engagement and changing attitudes in their assessment of their efforts.
Connecting to Change the World builds on an earlier resource that Pete and I developed called Net Gains. This latest collaboration with John Cleveland includes examples and lessons that have emerged from our work with social impact networks over the last decade or so. During that time, we’ve been introduced to many new networks and deepened our work with others. As a consequence, we have a better understanding of what makes some networks highly “generative.” By generative, we mean networks with a renewable collaborative capacity to generate numerous activities simultaneously. These are networks that activate members’ connections on an emergent basis as need and opportunities arise.
Examples in the book include RE AMP – more than 165 nonprofit organizations and foundations in eight Midwestern states working together on climate change and energy policies, Reboot- a network of young Jewish American “cultural creatives” who are exploring and redefining Jewish identity and community in the U.S. and the U.K., ten regional networks of state agencies and nonprofit providers that have organized to end homelessness in Massachusetts, and five regional and two national networks of rural-based organizations that are promoting public policies that benefit rural communities in the U.S. In all of these networks, members have been very deliberate about creating, strengthening and maintaining network ties in order to establish a base of connections from which many activities can arise at the same time or over time. This foundation is the starting point for the progression from connecting to aligning to production or joint action that we also discuss in the book.
Net Gains provides practical advice for the growing community of network builders developing networks for social change. The handbook draws from the experiences of network builders, case studies covering a diversity of different networks, and emerging scientific knowledge about “connectivity.” The guide is divided into four parts, each focusing on a specific element of network building and offering strategies for successful development of networks at different stages in their evolution, from the moment of their inception, to the management of their ongoing production.
The handbook can be downloaded here.
Connecting to Change the World is an informative guide to creating collaborative solutions to tackle the most difficult challenges society faces. Drawing from the authors’ depth of experience with more than thirty successful network projects, the book provides the frameworks, practical advice, case studies, and expert knowledge needed to build better performing networks. The book aims to give readers greater confidence and ability to anticipate challenges and opportunities.
When you’re evaluating a network, what are you looking for?
We recently submitted an evaluation proposal for a 7-year old network with more than 120 organizations spread across more than a half-dozen states. Without knowing much about the network we had to describe what we’d be evaluating, our analytic framework. It had 12 components, many of them specifically about a network, rather than an organization. It’s a framework we’d apply for assessing the condition and performance of any network.
Purpose: What is the network’s purpose? Is it being fulfilled? Has it changed over time? What other purposes are emergent among network members?
Value Propositions: What are the reasons that members participate in the network? Which reasons are most important to the members? How well do members feel their value propositions are being fulfilled by participating in the network?
Membership & Engagement: Who has been attracted to the network and who hasn’t that it would be desirable to have? What are the types of engagement in the network and to what degree do members engage in the network? Are the network’s rules/incentives for member engagement effective? Are there barriers that prevent/reduce member engagement?
Network Connectivity: What are the relationships among members? What level of reciprocity and trust has been built? What is being transacted between members? How has member connectivity evolved over time? What is the connectivity “shape” of the network (different patterns of connectivity—e.g., super hubs; multiple hubs; clusters) and how does the shape enable or block network efficiency and effectiveness?
Network Alignment: How well are network members aligned around ideas, goals, strategies, standards, and other guideposts? To what extent does alignment in the network influence members’ actions?
Network Production: To what extent has the network’s connectivity and alignment created conditions for collaboration/co-production by network members of, for instance, usable knowledge, policy change, services, or innovations. How well do network production processes function?
Other Network Capabilities: Which other network capabilities (e.g., network reach and resilience) matter to the network’s health—and what is their condition?
Governance: Does the network’s structure for decision-making enable members? Is it efficient and effective? Does it promote member confidence in and loyalty toward the network? What are the network’s monitoring and feedback loops and how well are they being used? What is the network’s resonance to members’ interests/actions? What is its adaptive capacity?
Business Model: What is the value chain within the markets and other contexts within which the network operates? What products and services—value creation– does the network offer? What is the network’s business model—revenues and costs—and how will it be sustained?
Operations: How well does the network enable members to benefit from the network through coordination of and communications among members, access to shared resources, working group leadership, and peer-to-peer exchange and learning? What staffing, mechanisms, and resources are in place? Which members do/don’t use them?
Strategic Communications: How is the network positioned with external audiences/stakeholders to achieve its goals? In what ways can the network’s external connections, capacities, and brand be leveraged for greater impact or to attract more resources?
Impacts: What measurable impact is the network having in achieving its purpose and goals? What impact is participating in the network having on the way members think and act? How can the network effectively measure its impact on a continuing basis—and use the information for improving its performance?
Taking your network’s temperature regularly is easy—and helps to inform continuous improvement of the network’s effectiveness.
When the 14 organizations in the Southwest Rural Policy Network met in November 2010 to discuss how well their network was doing, they didn’t just share their latest impressions. They had data stretching back nearly a year and a half. Since June 2009, as part of their formal work plan, they had self-assessed the network five times using a Network Health Scorecard. The assessment covered four essential categories: the network’s purpose, performance, operations, and capacity. The process only takes a few minutes after the network’s quarterly meeting—but reveals a great deal about how network members judge the network.
In November, Joyce Hospodar, the network member who chairs the network’s evaluation committee, summarized the scores—on a scale of 1-5, with 1 being low/5 being high—for the past five quarters: Purpose scores were holding steady. Performance scores peaked the previous spring. Operations and Capacity hovered around 4.0, but dipped recently.
Interestingly, when network members also scored where they thought the network’s health was compared to a year earlier, the ratings were all substantially higher than at the outset.
The scorecard is a tool, one source of evaluative feedback a network can use to gauge how well it’s doing and what sort of improvements might be useful. “Note,” says network coordinator Mikki Anaya, “this assessment only measures one aspect of the SWRPN’s effectiveness—the capacity/organizational efforts of the network.” A different evaluation will look at the network’s policy advocacy activities.
The one we developed has a total of 22 questions divided into the four categories. Several networks have adapted the questions to better reflect the specifics of their network. But in any case, the evaluative process is the same:
- Identify key indicators of the network’s well-being
- Regularly collect data from the members
- Analyze the data and share it with members
- Determine what changes are needed
Kudos to the members of the SW Rural Policy Network for picking up on this tool and incorporating its use into their network practice.
Intentionally managing members’ connections can strengthen your network.
A network’s connectivity–the number and quality of links between nodes, and the structure of those links–changes over time. To support a network’s development, network stewards intentionally manage this evolution, instead of just letting it happen.
A year ago, we started working with a start-up national network with about 60 members. The connectivity among members, which we measured and then, using special software, mapped graphically, was fairly low–not a surprise since it was a young network. But there was a core of about 11 members who were more densely and intensely connected to each other. The network maps, which place the most connected members at the center of the map, revealed this core of members, as well as those members at the periphery with few connections to others. As a result of the connectivity analysis the network stewards initiated activities aimed at increasing connectivity.
A year later–we just reported in a “state of the network” presentation at the network’s annual meeting–the connectivity building efforts have been a great success. The average number of links among members more than doubled. The intensity of links–what members transact with each other–also increased substantially. And new network maps revealed that the core of highly connected members also more than doubled–even thought there had been a 33% turnover in network membership. Now 25 members form the core or central hub of the network. All of these changes indicate strengthening of the network, revealed and made visible to the members through the use of network mapping.
Clay Shirkey, a champion of network approaches, sees a new revolution coming.
Here is Shirkey’s fascinating insight, offered in an interview in the June 2010 issue of WIRED:
“People have had lots of free time for as long as there’s been an industrialized world. But that free time has mainly been something to be used up rather than used, especially in postwar America, with the rise of suburbanization and long commutes. Suddenly we no longer lived in tight-knit communities and therefore we spent less time interacting face-to-face. As a result, we ended up spending the bulk of our free time watching television…
Someone born in 1960 has watched something like 50,000 hours of television already–more than five and a half solid years…”
Somehow, watching television became a part-time job for every citizen in the developed world. But once we stop thinking of all that time as individual minutes to be whiled away and start thinking of it as a social asset that can be harnessed, it all looks very different. The buildup of this free time among the world’s educated population–maybe a trillion hours per year–is a new resource. It’s what I refer to as the cognitive surplus.”
Shirkey further argues that as watching television, a solitary activity, is replaced by the use of technologies that promote social connection, there is a growing demand and ability for shared and productive activity.
“When someone buys a computer or mobile phone, the number of consumers and producers both increase by one. This lets ordinary citizens, who’ve been previously locked out, pool their free time for activities they like and care about. So instead of free time seeping away in front of the television set, the cognitive surplus is going to be poured into everything from goofy enterprises like lolcats, where people stick captions on cat photos, to serious political activities like Usahahidi.com, where people report human rights abuses.”
In short, the cognitive surplus will feed the process, already begun, of social networks of various sorts using technologies that support/enhance/ease connectivity to align around particular ideas and identities and then produce value. An idea that Shirkey explores in his new book, Cognitive Surplus: Creativity and Generosity in a Connected Age.
Building an Internet platform that links rural networks into a force for public policy change.
When the Kellogg Foundation started investing in getting grassroots networks in rural areas to work together to develop and advocate for rural-friendly government policies, it envisioned the creation of “a shared platform that helps coordinate research, learning, tool and resource development, expertise, and communication.” In other words, using the Internet to build a new collective capacity by connecting, learning, and aligning thousands of rural-based organizations, and helping them to collaborate and organize at vast scale.
To help turn that vision into reality, the Innovation Network for Communities developed a concept paper about what such a user-driven digital platform might look like and how it might be built. Although the idea focused on the particulars of supporting rural policy networks, we suspect that it has application to other network building situations.
Our starting point for the concept was what the platform is supposed to help people do. We imagined that “RuralUS.net” (a working name for the Web site) would help individuals, organizations, and networks interested in and engaging in rural policy change to:
• Find, learn about, and connect with each other, and then strengthen links.
• Align and organize in digital communities of varying size, shapes, and duration around shared characteristics (e.g., geography, age), interests, and identities.
• Access expert resources (digital modules, news feeds, webinar training, etc.) for developing policy change capacities and efforts.
• Collaborate in the production of policy change (e.g., identify opportunities; develop policy agendas; develop policy proposals; generate advocacy; etc.)
• Evaluate ideas and information by assembling collective points of view digitally (based on voting, Web page rankings, member- or user-community recommendation “engines” like Amazon.com’s “others like you have purchased…,” etc.)
The platform, we proposed, ”will combine the functions of a more traditional ‘trusted source’ Web site that provides users with reliable information, tools, and expertise with those of a ‘user-sourced’ site that engages users in peer-to-peer accessing, generating, sharing, using, and archiving of information.” Users would be able to:
• Build online personal social networks nationwide relevant to their concerns.
• Engage in “rapid learning communities” focused on learning about rural policy issues of immediate importance to them.
• Search for and access a wide range of expert materials, tools, and advice concerning specific rural policies; how to build policy change networks; and other topics;
• Generate peer-to-peer knowledge about and tools for policy-change processes and specific policy issues.
• Archive and share their experiences as rural policy change makers.
• Digitally access relevant meetings and conferences (e.g., regional planning meetings).
• Readily access and share news from the rural policy frontlines: news feeds, blogs.
• Build their policy-change capacities, such as in communications for policy advocacy.
• Create a user-generated “early warning” system that identifies and alerts others about changes in local, state, regional, tribal, federal policy opportunities.
Of course, what capabilities should be built into the platform depends on what the potential users say they need and would use. To get some sense of the potential look and functions of the user-driven dimension of RuralUS.net, we looked closely at a number of Web sites, including:
www.govloop.com– online community for government employees created by Steve Ressler, a federal IT employee, about 5 years ago. “I wanted an online forum to connect all my various groups and to connect with government employees across agencies. I wanted an informal place where people could gather, share their ideas, and ask other questions. A place that could serve as a repository for both current and future government employees as they start and grow in their career.”
www.techsoup.com – technology resources for nonprofit organizations. “TechSoup.org offers nonprofits a one-stop resource for technology needs by providing free information, resources, and support. In addition to online information and resources, we offer a product philanthropy service called TechSoup Stock. Here, nonprofits can access donated and discounted technology products, generously provided by corporate and nonprofit technology partners.” (Site is 10 years old; unique monthly visitors: 109,880.)
www.changemakers.net — Supported by Ashoka Foundation. “Changemakers is an initiative of Ashoka: Innovators for the Public that focuses on the rapidly growing world of social innovation. It provides solutions and resources needed to help everyone become a changemaker and presents compelling stories that explore the fundamental principles of successful social innovation around the world. Changemakers is building the world’s first global online “open source” community that competes to surface the best social solutions, and then collaborates to refine, enrich, and implement those solutions. Changemakers begins by providing an overarching intellectual framework for collaborative competitions that bring together individual social change initiatives into a more powerful whole.” (Unique visitors monthly: 25,797)
www.socialedge.org – supported by Skoll Foundation. “Check what our expert bloggers have to say about social entrepreneurship: practical advice, inspirational stories, name dropping… Every week is a new story on Social Edge.” (Unique visits monthly: 21,662)
Point us to other examples that you know about. Let us know your thoughts about designing digital media to promote social change.