IN.gov - Skip Navigation

Note: This message is displayed if (1) your browser is not standards-compliant or (2) you have you disabled CSS. Read our Policies for more information.

nps_1_newsticker_mgmt_plan

Appendix D: Evaluation Strategy

Introduction

In the fall of 2003, the U.S. Environmental Protection Agency (U.S. EPA) directed the states to develop an evaluation strategy for the statewide Nonpoint Source (NPS) Pollution Program. The state evaluation strategy is to measure the effectiveness of the NPS program at two levels: the overall statewide program level and individual NPS projects or activity level.

A. The evaluation is to be used as a multi-purpose tool to:

  1. build on existing data collection of state NPS programs and their partners;
  2. select methods and approaches that are appropriate within each state;
  3. support accountability at the program and project level; and,
  4. provide mechanism to improve and strengthen the states’ implementation approaches at the statewide and project levels.

The increased emphasis on environmental measures or outcomes will require the NPS Evaluation Framework to be closely integrated with the state’s monitoring and assessment program. The Indiana NPS program monitoring is conducted by the sub-grantees and IDEM staff. IDEM’s Assessment Branch data is requested by grantees for historical and supplemental data to be used in developing watershed management plans. The IDEM Assessment Branch data is the backbone for the 305(b) report and is also used to select streams added to the 303(d) list of impaired waters. Many of the 319 projects are selected in watersheds containing streams on the 303(d) list (an environmental indicator). Some projects are selected for education (a social indicator).

Historically, the greatest emphasis on measuring goals and final evaluations has been with administrative indicators, such as: funds spent, hours worked, and reports completed. This evaluation strategy will continue to use the administrative indicators, but with an increased emphasis on environmental and social indicators.

B. Timeline goals in this statewide NPS program Evaluation Strategy:

June 2005:
Identify additional administrative and environmental indicators for assessing the effectiveness of the overall NPS Program.
  • Results 2005: Additional indicators were included in the evaluation framework.
  • Results 2006: Rough drafts of both social and environmental indicators have been provided by Purdue University through the NPS Environmental Framework 319 project.
  • Results 2007: Rough draft of environmental indicators has been prepared by Purdue University. In addition to the final list of core social indicators, there is also an option to select additional social indicators that are selected on a project-by-project basis.
Submit 319 grant application to develop environmental and social indicators for both baseline results and post-BMP results.
  • Results 2005: The first two years of the 319 grant is expected to begin by October 2005.
  • Results 2006: This project is on target to be renewed for the third and fourth year.
  • Results 2007: An additional year was added to the Purdue Environmental Outcomes I with the addition of gathering more detailed information with social indicators and to establish an environmental council. The Environmental Outcomes II 319 proposal was submitted for contracting; still waiting for an executed contract.
October 2005
All NPS grant applicants will be required to supply information on project inputs, expected outputs, and short-term and long-term outcomes (or impacts) with their grant applications.
  • Results 2005: These elements were included in the 2005 grant applications and a much improved version was added to the 2006 grant applications.
  • Results 2006: These elements were also included in the 2007 grant applications and were part of the evaluation checklist.
  • Results 2007: These elements are included in the 2008 grant applications as part of the evaluation checklist.
Develop a data storage system compatible with Storage and Retrieval (STORET).
  • Results 2005: A 319 grant was given to the IDEM Data Management Section to assist in this project with an emphasis on the IDEM Assessment Branch AIMS database. Another option is to feed data through the Assessment Branch with 319 data in an acceptable format to be added to their AIMS database and the Indiana Water Quality Atlas.
  • Results 2006: The RFP for the new AIMS II database is almost complete. The project manager worked with personnel from IT, the Assessment Branch, and NPS/TMDL to provide detailed information on the NPS database that will be housed with the original AIMS database. It will be separated by programs and data requirements. The database for the NPS data will have versatility not exhibited in the older AIMS. AIMS and AIMS II data will be downloaded to STORET (Water Quality Exchange [WQX]).
  • Results 2007: Proposals were solicited by the state and a contractor was selected. Work is in progress and is slated to be completed by early 2009.
June 2006
Identify social indicators for assessing the effectiveness of the overall NPS program.
  • Results 2005: Purdue was awarded a 319 grant to complete this task.
  • Results 2006: The Purdue team, along with other Region 5 states, has developed a list of social indicators. They are in the process of developing a Web-accessible/Web-based tool box that will provide pre-project, middle project, and post-project surveys at the project level. They are also developing a database that will integrate with GRTS to house all the data collected from each 319 project.
  • Results 2007: The Purdue team is in the process of collecting baseline information using the developed indicators. Additional watershed projects are being selected for data collection and in-depth evaluation.
December 2006
Develop a long-term NPS monitoring plan to measure post-BMP results that will be updated annually.
  • Results 2005: Purdue University will be awarded a 319 grant to develop this plan.
  • Results 2006: The environmental framework project and the AIMS II project are addressing this issue and, with IDEM’s assistance, will develop the appropriate monitoring plans. Also, IDEM has begun to identify and work with a project by offering to supply equipment and additional funds in order to establish effectiveness monitoring with pre- and post-BMP implementation strategy and using a paired watershed to reduce the variability factor.
  • Results 2007: After AIMS II is completed, Implementation projects will submit strategy to measure the effectiveness of their BMP installations. Strategy will be provided with their proposals and added to their tasks list.
Incorporate environmental and social indicators into selected projects.
  • Results 2005: Purdue University will be awarded a 319 grant to complete this timeline goal.
  • Results 2006: Purdue, through the 319 grant, has already selected and collaborated with three projects throughout the state to complete this timeline.
  • Results 2007: Social indicator data is already being gathered for the projects selected last year. A clear plan for measuring the environmental indicators has not yet been finalized. Discussions for a few select projects selected for next year’s budget may have increased sampling strategy before and during implementation designed to measure BMP effectiveness and changes in water quality.
Develop a data storage system for additional indicators.
  • Results 2005: IDEM has redefined this timeline to, “Develop a procedure to store the results of the social indicators.”
  • Results 2006: The AIMS II project has a draft request for proposal and will be ready to hire a programmer to develop this data storage system. The social indicators will be stored in a database that is presently being developed by the Region 5 states. This database will be integrated into GRTS.
  • Results 2007: The data storage system to collect social indicators has been developed by Region 5, but is not yet released for use. The collection of environmental indicators has already begun with the individual 319 project. The quality assurance project plans require projects collecting environmental data to submit data to the project manager in electronic format, preferably in a Microsoft Access table with required spatial and temporal information.
Establish protocol for gathering baseline information for social indicators.
  • Results 2005: This timeline is expected to be included in the 319 Purdue University Environmental Framework project.
  • Results 2006: Purdue, with the 319 Environmental Framework project, is presently developing an online toolbox that will assist project coordinators in collecting baseline social indicators.
  • Results 2007: Toolbox has been developed, but is not officially online at this time. The first training to IDEM 319 and watershed specialist staff was presented by the Purdue team. The current contract was extended one year to add more in depth study of social indicators and 319 projects. An additional three year contract, “Demonstration of Outcomes Phase II,” was submitted for approval.
December 2007
Develop a system to accurately measure trends or environmental improvements, quantitatively, as a result of implementation of BMPs.
  • Results 2005: A basic system will be incorporated in each project’s Quality Assurance Project Plans.
  • Results 2006: This system is being developed through the Purdue Environmental Framework project and also by incorporating the strategies used by a large project that were presented at the 2006 National NPS Monitoring Conference.
  • Results 2007: Five projects were submitted to the Region 5 U.S. EPA Accountability Project. Each year, the estimated load reductions are submitted to Region 5 U.S. EPA through the required database. Two of the projects have submitted first year results. The other three will be implementing BMPs next year. A “W” team has been assembled and will be selecting additional projects to evaluate changes to water quality. A proposal was submitted to the 319 project selection team to add additional monitoring capabilities for the purpose of showing improvements as a result of BMP installations. This may result in a decrease in the number of projects selected and an increase of budgets to the selected projects. Estimated number of projects to be selected are three to four total.
Incorporate all indicators into selected 319 projects.
  • Results 2005: This timeline is expected to be completed on time.
  • Results 2006: Three projects throughout the state have already been selected and data is being gathered through a collaborative effort between project coordinators and Purdue University that is funded by the 319 Environmental Framework project.
  • Results 2007: Social indicators have been incorporated into selected 319 projects and environmental indicators are being measured by several planning and implementation projects through environmental sample collection or models.
December 2008
Begin tallying the environmental quality improvements.
  • Results 2005: This timeline is expected to be completed on time.
  • Results 2006: Results from the three test projects will be summarized and the toolbox with training materials will be close to being launched.
  • Results 2007: Region 5 database systems for the collection and evaluation of effectiveness, using social indicators, have not yet been released. The database system to collect environmental indicators has found a contractor to complete the work but this project has not started and is awaiting an executed contract.
September 30th, 2009
Develop a formal report to provide feedback to the NPS program.
  • Results 2005: This timeline is expected to be completed on time.
  • Results 2006: This timeline is expected to be completed on time.
  • Results 2007: Report will depend on the establishment of a format to collect indicators and to measure their effectiveness. Updates in GRTS and the Social Indicator Toolbox, along with the NPS database with a means of uploading data into STORET (WQX), will provide results of the NPS program to Region 5 U.S. EPA.

Appendix D: Table of Contents

Table of Contents