The Institute of Medicine (IOM) was asked to establish a committee to address the expressed need of the Department of Health and Human Services for a means of systematically assessing and evaluating the status of 72 or more Metropolitan Medical Response Systems developed by cities across the United States and understanding the effectiveness of the overall program approach. The committee's expertise included public health, medicine, emergency response, emergency management, emergency planning, mental health, hospital administration, community planning and evaluation, and program and systems evaluation.

The project was divided into two (2) phases. In Phase I the committee identified performance measures and systems to assess the effectiveness of, and identified barriers related to, the MMRS development process at the site, jurisdictional and governmental levels. In so doing, the committee considerded the following:

• How can Office of Emergency Preparedness (OEP) measure, at the program level, whether the strategies, resources, mechanisms, technical assistance, and monitoring processes provided to the NMRS development process are effective?
• How can OEP identify whether the performance objectives identified in the MMRS contract lead communities to preparedness?
• What modifications, additions and/or subtractions should be made to these performance objectives to assist communities throughout the development process?
• How can existing standards he used to validate these Performance objectives? If standards don't exist, how can new standards he created and/or the performance objectives be validated?
• What strategies have communities used to enhance their existing capabilities? What are the most effective means to measure these additional capabilities?
• Can relationships between traditional first responders / public safety officials and their supporting hospitals / public health offices be assessed? If so, how?
• What tools and/or models exist to measure preparedness for natural disasters?
• Do current Federal performance measures for natural disasters or other programs (mitigation and response) have application to WMD terrorism preparedness (e.g. FEMA Project IMPACT)?
• How can casualty assumptions, for communities of varying populations, be established (percent of population, historical data)?
• How can OEP measure the pre-existing systems, methodologies and plans that are used by public safety, public health and health services agencies to communicate during day to day operations? How can OEP measure the impact the MMRS development process has had on the level and/or expectations for this communication?
• How can financial barriers related to WMD preparedness be identified and measured?

In Phase II the committee used the performance measures developed from Phase I to recommend, then develop appropriate evaluation methods, tools, and processes to assess the MMRS development process. When developing these methods, tools and processes the committee addressed the following:

• What is the most appropriate approach or model for evaluating the MMRS development process (e.g. surveys, interviews, review of plans, peer review, operational tests, etc.)?
• Is there an appropriate sample size that would adequately represent the impact of the MMRS development process?
• Considering the variance in local health systems, how can OEP appropriately draw meaningful conclusions from the results of this evaluation?