This is the first in a series of articles that discuss the importance of measuring the effectiveness of fire prevention and public education programs. This process of evaluation will provide the stakeholders with information regarding a program’s (or project’s) effectiveness. This first article will discuss the reasons we incorporate evaluation as part of our programs and the resources available in the area of program evaluation. At the end of the article, the reader will be able to identify the need to incorporate program evaluation and available resources.
All aspects of fire service deliveries should be evaluated: emergency response, emergency medical services, specialty operations and fire prevention. We will look at the application of evaluation to fire prevention programs, specifically in the areas of fire inspection, code compliance, fire and life safety education, fire plan review and fire investigation. These programs have the greatest impact on community risk reduction because they are identified as preventative rather than reactive.
The Why
Evaluation has been a part of many industries over the past 50 years. Private industry, primarily in the business sector, started this practice in the late 1940s and was at its height in the 1980s with the introduction of Management by Objectives (MBO). This was the time that the healthcare industry began to incorporate evaluation into all phases of prevention program development. This was started by large hospital groups to answer questions by funding sources, especially business executives who were looking to qualify and quantify program results. It was then adopted by law enforcement in the 1990s for similar reasons.
The concept was presented to the fire service at the same time through benchmarking and initial performance measurements, but may not have been widely accepted or sought after because of limited funding sources outside of local government. However, at the time, the International City/County Management Association incorporated this information into their book, Managing the Fire Service. This book was developed not only for city managers, but also for fire service managers to assist with management decision-making.
The concept was furthered by the incorporation of program evaluation as part of the accreditation process for fire departments by the Center for Public Service Excellence in 2000. Currently, the Insurance Service Organization has incorporated evaluation of prevention and education programs as a component of their rating process for the fire department. Lastly, the Department of Homeland Security uses program evaluation as a component of any funding of prevention and education programs through the Assistance to Firefighters Grant (AFG).
So, why do we spend the time to evaluate our programs? The program manager needs to do so to show not only our command staff the impact of the program to the community, but also to provide information as feedback to the personnel involved in presentation. We also do this as the command staff to make the budget decisions; yes those tough decisions as to what gets funded and for how much. We are held accountable to funding sources, whether private or public, and are expected to prove the benefit to the community. We must pass this information onto the community, through our Police and Fire Commission and elected officials. The community needs to know what resources are available and the impact of these programs on the reduction of risk of injury, death and property loss.
The “why” will be discussed further in future articles through specific examples. The fire service needs to understand that evaluation will become a large part of the fire service in the coming years and we need to learn more about this area. While it is not expected that we embrace this area in its entirety initially, we must recognize its need to be incorporated a little at a time. The good thing is that we already incorporate some measure of evaluation through monthly or annual run statistics.
Resources
There are many resources available to learn about evaluation. A primary resource is the manual published by the Center for Disease Control in 2000 titled Demonstrating Your Program’s Worth. This manual teaches program managers how to demonstrate the value of their program to the community. While the manual was developed for health services, it can easily be applied to fire service prevention programs. As a matter of fact, this manual served as the basis for the development of the National Fire Academy’s six-day course titled “Demonstrating Your Fire Prevention Program’s Worth.” The manual will help you understand how to develop the tools for the various stages of evaluation. It is available for download at the CDC website – www.cdc.gov/ncipc/pub-res/demonstr.htm. It is no longer in print and only available online.
Another valuable resource is Managing Fire and Rescue Services (2002) from the International City/County Management Association. This book was originally published in 1988 and contains a chapter on performance measurements. Performance measurements are one of the methods to evaluate programs or service delivery. This book discusses the application of performance measurements to fire service delivery, specifically emergency response times. These measurements come from standards. This group is one of the largest associations and provides city and county managers with information for investigating and evaluating the reasons for high or low performance levels. The book is available at their website, www.icma.org, along with other recent documents on performance measurements.
The National Fire Protection Association’s Research Foundation published “Measuring Code Compliance Effectiveness for Fire-Related Portions of Codes” (2008). This paper addresses measuring the effectiveness of fire prevention programs through 11 specific measures that will identify whether inspection services are effective. They not only outline the measures, but show how they were created and applied within the fire service. This paper goes a long way in establishing a national performance measurement for prevention programs. This has lead to the creation of a new NFPA committee to establish a standard on performance measurements for the fire service. This will aid departments in not only applying these performance measurements within their department, but also in creating a way to measure against other fire departments of similar size. The paper can be downloaded from www.nfpa.org/assets/files/PDF/Research/CCEReport.pdf.
As mentioned earlier in this article, the National Fire Academy established a new six-day course encompassing the basics of evaluation, how evaluation is an essential component of community risk reduction, and the six stages of a program evaluation model. The course emphasizes the information presented by applying it to an existing program for understanding and then applying it as a final project to a new program. This course is the only one available to the fire service on the creation of evaluation methods. It is offered at least four times a year. Information on course offerings and applications can be found at www.usfa.dhs.gov/nfa/nfaonline.
The Home Safety Council has developed a web-based presentation on evaluation through their Expert Network Academy. This module provides the reader with an overview on the types of evaluation and the program evaluation model. These presentations provide affordable training along with a certificate of completion. This can be accessed at www.homesafetycouncil.org/Academy/index.html.
Lastly, Vision 20/20 presented a Model Program Symposium in 2010 and hopes to provide another symposium later this year. The symposium provided attendees with information on existing fire departments evaluating fire prevention programs. These presentations are provided for review at their website www.strategicfire.org/main.cfm?pageID=18. This organization was created in 2008 to work on national strategies for fire loss prevention. They established five strategies and the area of model performance measurements. This organization is comprised of major fire service organizations: The Institution of Fire Engineers – U.S. Branch; United States Fire Administration; National Fire Academy; International Association of Fire Chiefs; Insurance Services Organization; Center for Public Service Excellence; International Code Council; International City/County Management Association; and Center for Disease Control.
The next article will discuss the road to effective evaluation and the types of evaluation. The series will conclude with a discussion of the stakeholder’s responsibilities and evaluation examples.
MARTIN M. KING serves as assistant chief in charge of the fire prevention bureau for the West Allis, WI, Fire Department. He has been with the department for 24 years and a member of the fire service for 32 years. As an adjunct instructor for the National Fire Academy, he instructs numerous courses in risk management and he has served on two recent fire prevention manual material review committees for IFSTA. He is currently a member of the Vision 20/20 committee working on performance measurements for fire prevention and education programs. Martin is a contributing expert to the Home Safety Council's Expert Network Academy. He has a bachelor's degree in business administration and is a recent graduate of the National Fire Academy's Executive Fire Officer Program.
Martin King | For Firehouse.com
Martin King has over 32 years of experience in the field of fire and emergency services. He has been a member of the West Allis (WI) Fire Department for 24 years, serving as the Assistant Chief in charge of the Fire Prevention Bureau for the past 9 years. As an Adjunct Instructor for the National Fire Academy, he instructs numerous courses in risk management. He served on two recent fire prevention manual material review committees for the International Fire Service Training Association. He is currently a member of the Vision 20/20 committee working on performance measurements for fire prevention and education programs. He is a contributing expert to the Home Safety Council's Expert Network Academy. He has a bachelor's degree in Business Administration and is a recent graduate of the National Fire Academy's Executive Fire Officer Program.
