Case Study of Public Safety Networks

10, 2013

What is success for an organization like Nlets? How do you know if Nlets’ investment in time and money is really worth it? 

Public Safety Networks (PSNs) provide a shared technology platform for supporting information sharing, computing interoperability and interagency interactions involving policing, criminal justice, homeland security and emergency response. Measuring the success of a public safety network organization like Nlets is challenging because doing so requires that you put a value on the tangible and intangible benefits that result from improvements in the quality and quantity of data sharing among diverse stakeholders - states, federal agencies, and private strategic partners.  The additional challenge with measuring success for Nlets is figuring out just how much Nlets helped/contributed to the success of its users (versus all the other wonderful things the users are also doing on their own).  Nlets staff understood the complexity of success measurement, and so took the time to develop strategic goals, build consensus for those goals, and is actively monitoring the achievement of those goals. 

People who study networks tend to focus on one of three areas to measure success – the agents (individual people), the artifacts (technology), or the arrangements (relationships and governance processes). Agent success is generally measured by improvements in efficiency and effectiveness, and particular to law enforcement, in terms of lives saved, property secured, investigation time reduction, and suspects apprehended.  Artifact success is generally measured by technology use and availability rates – increases in number of end-users, increases in message traffic, advancement toward 99.9999% network availability. Arrangements success can be measured in terms of improvements in the collaborative relationships among state, federal, and private organizations (e.g., increase in trust so future interactions go more smoothly). 

In our research studying PSNs across the U.S., we have found some general characteristics of successful inter-agency collaborations. These PSNs are smaller scale than Nlets, but many have thousands of users and all need to balance the interests of a variety of stakeholders.  More successful PSNs tend to figure out how to show their across all three dimensions – agents, artifacts, and arrangements.  

Agent-oriented success:

Agent-oriented success measures are most challenging to track because it is statistically difficult to isolate the role information sharing technology plays in saving an officer or citizen’s life, securing property, etc. Improving public safety involves a myriad of factors – economic, social, enforcement, criminal justice, etc., so PSNs sometimes offer testimonials or individual stories of how technology improved the efficiency or effectiveness of law enforcement professionals. We took the testimonies of Nlets members (published on the Nlets website) and produced a “word cloud” (see below) showing what words members mentioned most frequently. 


Artifact-oriented success:

Network organizations focus a good deal of attention on artifact-oriented success because data gathering for it tends to be more straightforward. Success measures can be oriented around level of use (e.g., individual technology like radio, smartphone, state-wide technology like message switch volume, or network-wide technology like disaster recovery capability or Portal).

For example, Nlets currently…

  • … supports interfacing for over 1 billion messages,
  • … provides 99.95% system availability (NJIN)
  • …serves as a data-sharing platform for over 1.2 million law enforcement professionals.

Arrangement-oriented success:

What does it mean to be successful in terms of “arrangements” (in terms of the value of the relationships being forged and advanced by the network organization)? There are two general ways of measuring the success of arrangements – tangible and intangible.  Tangible arrangement success can be assessed operationally in cost, quality, and speed outcomes.  Track the cost of decision-making and find a decrease in travel costs to build consensus across many different stakeholders.  Track the quality of decision-making and find resolutions are not revisited over and over again before action is taken. Track the speed of decision-making and find that agreements and consensus are achieved faster today than 10 years ago.  Has Nlets improved the cost, quality, and speed of state-level public safety data sharing decision-making?  We believe the answer is yes.

Intangible success can be observed based upon stakeholder satisfaction either with the relationship between them and the overall network (or additionally, in Nlets case, the central staff) or with specific relationship outcomes (perception improvements like increases in trust and commitment). 

-        Commitments of member states – e.g., state of Idaho agrees to host the initial Nlets disaster recovery site (which goes lives in   2002)…shows joint commitment, and possibly an increase in trust…when accommodation (i.e., foregoing self-interest to meet the needs of the network) becomes modus operandi for network-based interactions.

-        Evidence of perception of internal legitimacy of decision-making process…that individual voices matter (inclusion)

-        Evidence of perception of external legitimacy of decision-making (credibility)

-        Locus of leadership: Nlets moves from connector to consensus-builder to change-leader (e.g., see history of XML/NIEM related activities from 1998 to present)

-        Progression along collaboration continuum from joint communication, to cooperation, to coordination of actions (increase in grant projects, pilot programs, state-specific testing)

Worth it?

So given these three types of network success measures what is our assessment?  Our analysis of Nlets has found it is indeed successful in terms of agents, artifacts, and the arrangements it forges and facilitates. Nlets has been successful in helping to effectively and efficiently: meet the data-sharing needs of individual officers on the street [agent]; modernize the communication infrastructure for law enforcement [artifact]; and build a public-private partnership of public safety professionals that serves as a relationship base not only for law enforcement data sharing, but also terror and natural disaster response and recovery [arrangement]…and thus, the work continues.

About the blog series:

This entry is part of a series of entries on a National Science Foundation funded research project examining public safety networks in the United States.  Nlets was one of several case studies and we focused attention on it as a result of a tip from a Curt Wood of Massachusetts. Additional entries in the series will cover a variety of topics including – dealing with legacy systems, the advantages of being a network organization, challenges Nlets has faced, and some challenges Nlets faces going forward.

About the authors: 

Martin’s research examines the formation, operation, and evolution of IT-enabled public safety networks.  His expertise involves designing IT (from applications to architectures) that supports public safety inter-agency data sharing and collaboration. He is currently an Assistant Professor at Northeastern University in Boston, MA.

Dax is broadly interested in the management and use of technology in organizations, particularly in the public sector. More specifically his research focuses on how governance influences technology and technology influences governance as they interact over time. He is currently an Assistant Professor at California State University Channel Islands in Camarillo, CA.