OpenNews is a growing network of developers and designers who are collaborating on open source technologies that journalism needs to thrive. An OpenNews “hub” supports a core set of network resources including Source, a community site that showcases open technology projects, code convenings that assemble small groups of members to work together on last mile solutions and a fellowship that recruits early- and mid-career technologists who are working outside of journalism and places them in newsrooms. These and other network activities, including an annual gathering SRCCON, aim to attract technologists to the field of journalism while strengthening network connections among journo-technologists so they can increasingly work, learn and solve problems together, within and across journalistic settings.
Our research over 12 months – including an analysis of network participation data describing the engagement of more than 1,100 journo-technologists over 5 years, a survey to 514 participating journo-technologists and social network analysis of member connections – yielded insights into conditions that have contributed to the evolution of this highly generative peer learning and action network. Among them:
- Many doors. Designers and developers can enter the network in many ways with options for participation that match the variety of members’ initial value propositions for becoming involved.
- Transparent discussion of network culture. Network leaders and organizers explicitly promote network principles of openness, generosity and inclusion. This has had an impact. Eighty-eight percent of respondents to our OpenNews survey reported that they advocate more for gender balance and diversity in the tech community because of participation in the OpenNews network. One journo-technologist started a journalist of color Slack community inspired by the ethos of OpenNews activities.
- Flexible base of connections: OpenNews provides a foundation for different “nodes” to light up quickly. This flexible base of connections is ideal for peer learning and collaboration in a rapidly evolving field like journo-technology. Per one technologist we interviewed, “With the constant disruption and change that is characteristic of the industry, capacity to adapt is paramount. This places a high premium on access to diverse people and ideas in real-time with low transaction costs.”
Luz Gomez, Knight Foundation’s director for learning and impact explains “…lessons from the project highlight the strength of networks in connecting people and advancing innovation, while providing journalists with a view into opportunities and gaps as they work to advance newsroom change,”
Because this has implications for the many fellowship and awards programs that funders invest in, we included this important finding in our report to the Knight Foundation:
- OpenNews fellows and fellowship alums serve as network “bridge-spanners.” Their exposure to many parts of the network allows them to connect otherwise isolated network clusters.
(click image to enlarge)
Our survey was co-released with a census of “News Nerds” developed by OpenNews. True to the network’s open and inclusive culture, members have been invited to dig into census results and contribute to the learning agenda and planning for a follow up OpenNews survey next year.
The more information you have about the engagement patterns of network members or users of an online platform, the more tempting it is believe that these data alone can tell you everything you need to know. But, until you explore what type of engagement is valuable and why, and what kind of impact that engagement has on people, organizations and communities, your hypotheses about what actually drives outcomes remain untested.
Organizations often assess their network building efforts or technology interventions (or a combination of the two) to be able to come to more definitive conclusions about what works so that measures and indicators can be adjusted and an organization can learn from its experiences. With new technology that tracks people’s behavior (or even old technology like years of paper attendance records from different types of events) you can integrate actual behavior data on engagement over time with survey and other research to get a more comprehensive picture of how value and impact are created through engagement. You can also compare how engagement and other measures, such as number or type of connections in a social network, relate to impact.
With two recent projects, we were able to integrate engagement data with survey and other data to probe the value of different levels and types of engagement. The results offered insights into how impact was achieved and helped both organizations refine their network engagement strategies.
More Engagement on the Community Commons Means More Impact on Users
The Community Commons provides public access to thousands of meaningful data layers that allow mapping and reporting capabilities for people and organizations to explore community health and policy data interventions and best practices.
- Data Collection and Analysis of Engagement – We worked with the Institute for People, Place and Possibility (IP3), the organization that stewards the Commons, to implement an online system to track user-centric data in a searchable, cloud-based relational database. This provided us with data to establish categories for a ladder of engagement based on engagement with core platform activities, such as building maps and reports, connecting to others, or reading tutorials to build capacity for using data.
- Survey Data Collection on Outcomes – After a year of collecting platform data, we launched a user survey to explore what impact platform use and tool engagement had on users.
- Results – Across key measures, the combined data showed greater impact for users who were more engaged. One of the core hypotheses in the Commons’ Theory of Action was that increased engagement with the platform’s tools would increase users’ knowledge, skills and capacity, a hypotheses that was supported by our research. A sample of the findings from this integrated analysis below.
Different Patterns of Engagement in Mozilla Science Lab Correspond with Different Views on Network Health and Outcomes
The Mozilla Science Lab is a network of researchers, developers, and librarians making research open and accessible and empowering open science leaders through fellowships, mentorships, and project-based learning.
- Data Collection and Analysis of Engagement – In order to build a full database of people who had engaged with Science Lab over the years, we used event records, call attendance records, and GitHub data on code contributions and study group participation to create categories for both the level of engagement and the type of engagement of network members. This allowed us to compare diversity of participation – those people who participated in more than one way – to level of participation – those people who participated a specific number of times — as part of our analysis.
- Survey Data Collection on Outcomes – As part of an existing cross-program survey conducted by the Mozilla Foundation, Mozilla Science participants were asked about their engagement in the networks that the Mozilla Foundation supports. Respondents were asked questions about the network’s health, and how they benefited from their participation in the network.
- Results – We found that an individual’s levels of engagement and diversity of engagement correlated in slightly different ways with their reporting on network health and benefits (see results below for an example). Connecting the dots between patterns of engagement in a network and a range of network outcomes continues to be an important part of how we approach our network evaluation work.
In 2008, Lisa Watson was the executive director of the Downtown Women’s Center (DWC), an organization dedicated to meeting the needs of women on Los Angeles’ Skid Row hoping to overcome poverty and homelessness. That year, Lisa received a Stanton Fellowship to investigate the viability of a co-located social enterprise retail store that would offer workforce training to homeless women and generate revenues for the center. Revenues would be used to subsidize housing and supportive services in the pricey Los Angeles real estate market.
For the past ten years the Durfee Foundation has awarded a select number of Stanton Fellowships to social change leaders in Los Angeles with the aim of fostering innovative solutions to some of the city’s most intractable problems. Lisa’s project became a reality in 2011 with the opening of MADE by DWC, a gift boutique and café that offers organic coffee and food along with one-of-a-kind vintage and contemporary women’s clothing, accessories, household accents, and their signature handMADE product line. One hundred percent of the proceeds support the residents of the Downtown Women’s Center, providing the kind of earned revenue that is a vital component of long-term sustainability for most nonprofits.
Prior to the fellowship, Lisa had met a handful of other Stanton alumni, all in the housing/homelessness space. Over the course of her fellowship, however, she expanded her connections to include Stanton fellows with expertise in urban planning, health, education, the environment, and economic development, as well as contacts in the L.A. Mayor’s Office. The interactions with other fellows significantly affected her project’s design as well as its resulting success. “By bringing together smart people from various disciplines in Los Angeles,” she notes, “problems can be viewed through various prisms rather than through a telescope. Solutions and strategies are developed by looking more richly at the problem from various perspectives and disciplines.”
The Stanton Fellowship provides funds over two years for each fellow to think deeply about a specific challenge related to their work and to tease out solutions that will improve life in Los Angeles. The Durfee Foundation deliberately encourages connecting and knowledge sharing among fellows as a way to foster the cross-fertilization of ideas that might lead to new approaches. Stanton Fellows are intentionally selected to represent a wide-ranging spectrum of issues and sectors, with fellows coming from government and social enterprise as well as nonprofits. Key elements of the program include opening and concluding fellowship retreats that overlap with the next/prior cohort of fellows; quarterly get-togethers hosted by a fellow who provides a tour of the issue they are tackling and includes time for fellows to update the group on their projects; and foundation staff matching fellows with program alumni mentors. In addition, every other year the foundation hosts a retreat to which all alumni of the program as well as current fellows are invited.
Enhanced Peripheral Vision
In order to better understand the network dimension of the program, the Durfee Foundation asked Network Impact to assess the role that ties among Stanton Fellows play in contributing to the program’s goals. To that end, in the fall of 2014 we surveyed current fellows and alumni, and supplemented that work with focus-group interviews and Social Network analysis (SNA) to assess the nature of the connections among fellows over time. What we found has implications for funders who are supporting innovation in the social sector, particularly investors in fellowship or leadership development programs who are curious about the wider impact of these initiatives.
The Strength of Loose Ties
“I can ask any Stanton person for their support, and I have done so. Some I only see once a year and that’s fine….I know who to call if I need something.”
It is not uncommon for fellowship programs to cultivate close, trusting relationships among participants as a way to promote more alignment and coordination among participants. For example, bonds that have been deliberately fostered among community leaders who participate in the Barr Fellows Program in Boston have led to increased collaboration among leading nonprofit organizations in that city. In contrast, a social network analysis of the ties among Stanton fellows reveals a different pattern. In the Stanton case, loose ties among fellows resulting in an exchange of “information that leads to new thinking or framing” are more common (see the maps below). Our mapping also shows that participation in the program increases each fellow’s reach, creating pathways to advice and information from a variety of nonprofit leaders. The majority of connections among fellows are not regular, close, or personal. Instead, the overall effect is one of improved “peripheral vision.” With a view to engaging in transformational activity beyond their own “silos,” nonprofit leaders working on different issues can adopt lessons from other settings and better align their plans of action. Our observations offer a different take on the Strength of Weak Ties thesis put forward by Marc Granovetter. Moreover, research on networks suggests that high levels of trust in networks typically coincide with strong, often personal, bonds among members. In the Stanton network, fellows tend to seek advice from peers as trusted sources, not on the basis of strong personal ties.
Below are maps of connections that “provided information that led to new thinking or framing that has been useful in my work” before the Stanton Fellowship, and after. This type of connection saw a 308 percent increase compared to the intensity of connections before the start of the fellowship period.
A Culture of Trust
“I feel free to pick up the phone and call anybody who was a Stanton Fellow, because it’s sort of a common culture that we all share, and it’s a very different level of trust and access….”
Like other social changemakers, one of the things that Stanton fellows value most highly is their access to trusted information from peers. As in most networks, trust lowers transaction costs for peer exchange; in the case of the Stanton fellowship, this network “glue” is the product of a shared set of understandings that the Durfee Foundation fosters between itself and Stanton fellows and among the fellows themselves. As one fellow explained: “We refer to it as a ‘fellowship’, and that is a particularly relevant word for this experience. I think we all feel very strongly that this is a community that matters deeply and has added immensely to our lives, personally and professionally….Stanton calls are always taken and returned, and there is a warmth and common bond that immediately eases whatever else is happening in the day.”
Reflecting on how this culture of trust has been created and maintained, fellows repeatedly cited the efforts of Durfee Foundation staff to encourage them to take risks. One alumnus of the program explained, “They have created this culture that is incredibly healthy and vibrant and encouraging….They invite you to push the envelope and are not fearful of what other foundations might perceive as failure, which is a healthy environment in which to experiment.”
Equally important is fellows’ confidence in the foundation’s capacity to select candidates whose motivations are aligned with theirs: the desire to transform conditions in Los Angeles for the better and to help the most vulnerable in the city. As one fellow observed, “The staff set the standard for how much we all connect. That’s part of the leadership everyone trusts so much.” Another reflected: “I think my comfort harks back to the rigor and savvy with which the selection takes place. It does say something to me about who is in that group of people, that I can trust them, at least enough to reach out to them.” In other words, the mere fact of being selected as a fellow inspires confidence and trust.
Reflecting on her time as a Stanton Fellow, Lisa describes her experience of getting beyond traditional silos in an environment of loose ties but strong trust. “The whole idea of trying to change a city we all care about is a real connector point. It taught me to love L.A. more, and I now feel more a part of it. I was exposed to all these different things that were totally out of my world and experience. As a result, I am involved and engaged in the city in new ways, with different people than I would have been before I was selected to be a fellow.”
Lisa’s project evolved based on her fertile exchanges with other Stanton Fellows and has continued to grow, with more than fifty of the women involved in the training now gainfully employed, and MADE by DWC products now being sold online and in stores in Los Angeles, providing much-needed funds to advance the full spectrum of DWC programs. Our hope is that others looking to nurture innovative social change efforts will look at how their current activities to connect diverse leaders present opportunities to increase the visibility of their grantees’ efforts and will begin to track the value created through those network connections.
Network Impact is looking forward to 2016! As a team, we’ll be exploring new themes as well as deepening our work in some key practice areas. Here are a few questions we’ll be investigating in the coming year…
In what contexts are online communities most useful to networks?
What are best practices for creating and managing online communities that enhance network connectivity, alignment and action?
For several years we’ve been learning from practitioners about challenges and successes in this area. We’ve also developed some resources for tracking the effects of digital platform use on individuals, their organizations and the communities in which they work. (You can read more about these projects: Civic Tech Assessment Guide and Community Commons.) We are now compiling our top lessons learned and will have a blog post on the topic early this year.
How can funders track and improve the network impact of their efforts to connect people at convenings and retreats as well as in online environments that bring diverse groups into contact with one another?
Over the last year, we’ve fielded an avalanche of questions about social impact networks (of grantees, investees, fellows, awardees) that grantmakers hope to catalyze by creating environments in which people with related interests can connect. Our guest post for Philantopic discusses what we learned from a social network analysis of the Durfee Foundation’s Stanton Fellowship. This year, we’ll be looking at strategies including human-centered design that aim to boost positive network outcomes in this area. We’ll also be exploring frameworks and tools for evaluating these “mass-weaving” efforts focusing on the value they produce for individuals, organizations and at the field level.
As big data, and even “medium data” become ubiquitous, who is successfully leveraging data and technology for social change and how are they doing it?
In our work with Kaiser Permanente and the W. K. Kellogg Foundation, we spoke with dozens of grantmakers, innovators and change agents in multiple sectors to find out how funders can catalyze and spread the use of data and technology to advance social change. What we found is that most organizations, including funders, are struggling to use data and technology well. It will take a concerted effort, and strategic investment, to bring the social sector up to speed. In the coming year, we’ll be looking more closely at network initiatives in this domain such as strategies for connecting technologists to nonprofits, to each other and to each other’s open source projects.
Featured in the summer issue of The Foundation Review “Network Evaluation in Practice: Approaches and Applications” authored in partnership with Julia Coffman, Director or the Center for Evaluation Innovation.
We often hear grantmakers talk about leveraging dollars but rarely get to see year-to-year data showing a funder’s investment over time with the additional resources they attract to an issue. The Garfield Foundation agreed to let us review their investments in the RE-AMP network over the last ten years so we could take a closer look at a specific case. Our analysis shows that even foundations with a modest corpus can leverage their charitable dollars many times over by investing in networks. Our latest blog post on this topic is featured in The Stanford Social Innovation Review .
Developed with the Center for Evaluation Innovation this two-part guide to network evaluation includes a brief that outlines the frameworks, approaches and tools to address practical questions about designing and funding network evaluations and aCasebook that provides profiles of nine evaluations.
Download at: www.networkimpact.org/networkevaluation
The civic tech field has expanded so widely in recent years, it’s hard to think of a major city or an area of civic life that these technologies don’t touch. In this dynamic environment, the John S. and James L. Knight Foundation has been a field leader, investing over $25 million since 2010 in projects ranging from neighborhood forums, to civic crowdfunding platforms, to efforts that promote government innovation. For eighteen months, Network Impact worked with Knight Foundation grantees and other civic tech leaders to find out how they measure success, focusing on tools they’re using to track platform performance and assessment challenges they face along the way.
We started by identifying key outcomes related to these common civic tech objectives and gathered case examples of assessments from the field:
- Build place-based social capital
- Increase civic engagement
- Promote deliberative democracy
- Support open governance
- Foster inclusion and diversity
Our work also led us to think about tracking the performance of a platform through its lifecycle – recognizing that assessment priorities vary with stage of development, from early testing of a minimum viable product to later-stage scaling of a tested concept.
The result of this research: two guides to evaluating civic tech that summarize assessment best practices, including leading methodologies and metrics that can help innovators monitor progress towards their goals and evaluate the impact of their efforts. Some of these assessment best practices focus on connections between users, both online and off-line, with an important network dimension.
Assessing Civic Tech: Case Studies and Resources for Tracking Outcomes is a publication of the Knight Foundation with Network Impact that focuses on measuring the impact of civic tech platforms on people, places, and processes.
How To Measure Success: A Practical Guide to Answering Common Civic Tech Assessment Questions is a Network Impact publication that offers examples and advice for monitoring a platform’s ongoing performance using tools and approaches that are effective and practical.
Additionally, the Knight Foundation wrote up their key lessons from investing in civic tech that are also worth a read.
How Code for America is using the Assessing Civic Tech guide
The release of this guide is coming at the right time. Demonstrations of what is possible are up in running in communities of every size across the United States. Now we need to find out not only what works, but what works best over time.
At Code for America, the guide will be particularly helpful for Fellowship teams and volunteer Brigades who are thinking about the questions they need to ask and the changes in attitudes they need to measure to assess progress towards increasing civic engagement and open governance. The process and case studies documented in this guide will be useful for structuring these assessments.
At Code for America, we believe that it is critically important to identify the residents, community groups, or government staff who will be using the particular public service program or benefit, then work with them early in the assessment design process. This guide provides important examples of how to frame an evaluation to include and work with intended beneficiaries. It offers sample questions and resources that will be very helpful to organizations and individuals who are beginning to explore how they can include measures of civic engagement and changing attitudes in their assessment of their efforts.
Connecting to Change the World builds on an earlier resource that Pete and I developed called Net Gains. This latest collaboration with John Cleveland includes examples and lessons that have emerged from our work with social impact networks over the last decade or so. During that time, we’ve been introduced to many new networks and deepened our work with others. As a consequence, we have a better understanding of what makes some networks highly “generative.” By generative, we mean networks with a renewable collaborative capacity to generate numerous activities simultaneously. These are networks that activate members’ connections on an emergent basis as need and opportunities arise.
Examples in the book include RE AMP – more than 165 nonprofit organizations and foundations in eight Midwestern states working together on climate change and energy policies, Reboot- a network of young Jewish American “cultural creatives” who are exploring and redefining Jewish identity and community in the U.S. and the U.K., ten regional networks of state agencies and nonprofit providers that have organized to end homelessness in Massachusetts, and five regional and two national networks of rural-based organizations that are promoting public policies that benefit rural communities in the U.S. In all of these networks, members have been very deliberate about creating, strengthening and maintaining network ties in order to establish a base of connections from which many activities can arise at the same time or over time. This foundation is the starting point for the progression from connecting to aligning to production or joint action that we also discuss in the book.
Drawing on cases of nonprofit networks, this article makes a case for widespread use of networks in the civil sector and examines the practical uses of the knowledge developed by “network science.” Individual chapters focus on the formation and structure of networks for philanthropy and non-profits from the most essential stages of development – framing and defining the term “network” itself – to the dynamics within existing networks and the interactions that sustain them over time. Written by Peter Plastrik and Madeleine Taylor for the Barr Foundation.
Download the article here