Planning is an integral part of the evaluation process. Through careful planning, collaboratives are more likely to be able to measure the indicators that will best show collaborative progress in achieving the desired outcomes for children and families in the community. Here are answers to questions that collaboratives are asking about developing an evaluation plan.

Collaboratives want to know...

How do the FY 2006 Evaluation Plan Guidelines differ from previous guidelines?

Do I have to use the worksheet provided?

I notice the Evaluation Plan Guidelines no longer suggest attaching instruments. Do you no longer want instruments attached?

Are there any suggestions for collecting and managing process information from our partners?

Can the Local Coordinator serve as the Local Evaluator for the collaborative?

Is it a conflict of interest to have the Collaborative Chair serve as the paid Evaluator?

How much should our collaborative attempt to accomplish related to evaluation?

Who should develop the Evaluation Plan?

What are the core elements of an FCP Evaluation Plan?

What does an Evaluation Plan look like?

How is the Results Evaluation Plan reviewed or evaluated?

How do the FY 2006 Evaluation Plan Guidelines differ from previous guidelines?

The overall intent of the guidelines is similar to previous years - to provide a framework for the development of an evaluation plan that supports the community strategic plan. The Guidelines have been focused on an Annual Results Evaluation, as opposed to a Three-Year Evaluation Plan, and they include an optional Results Evaluation Plan Worksheet. It is hoped the inclusion of a sample worksheet will lead to a more focused and streamlined community evaluation plan.

top

Do I have to use the worksheet provided?

No. You may organize your plan in any reasonable way that includes the five core elements discussed in the Evaluation Plan Guidelines for FY 2006. The Results Evaluation Plan Worksheet may assist you in developing and presenting the Annual Results Evaluation Plan in a streamlined, focused way. Rather than spending valuable resources "planning" more evaluation than is actually implemented, this worksheet will help you focus resources and planning on the required evaluation efforts.

top

I notice the Evaluation Plan Guidelines no longer suggest attaching instruments. Do you no longer want instruments attached?

DO NOT attach quarterly worksheets for collecting process information from partners. DO cite or provide copies of standardized instruments designed to collect knowledge, attitude, skills, aspiration, or behavior change information on target groups. DO provide copies of locally developed tools that assess knowledge, attitude, skills, aspiration, or behavior change information on target groups.

top

Are there any suggestions for collecting and managing process information from our partners?

Yes. Collecting process information: Most of your collaborative partners already submit process reports to their funders on a regular basis. Forwarding the same reports to the collaborative may help keep additional paperwork to a minimum. If you need a quarterly worksheet for partner reports, you may go to the FCP website/Evaluation Toolkit/Tools and Resources where you will find a Sample Worksheet for Partner Data.

Managing process data: A sample summary worksheet, Quarterly Summary of Partner Data is located in the Evaluation Toolkit/Tools and Resources and can be used to summarize the information from each partner for each separate program or activity in the three-year Strategic Plan. There is also an Example Quarterly Summary of Partner Data. The format allows you to summarize the information needed for the Electronic Quarterly Report. While "# of events" and "total # of individual contacts" are not required fields on the Electronic Quarterly Report, this information can be inserted in the Progress/Challenges text field. When you print out your county copy of the quarterly report, this additional information helps your collaborative have a better picture of how the programs and activities that comprise strategies are going. This information can provide early indications that the Strategic Plan is progressing and if not, where the difficulties might exist.

top

Can the Local Coordinator serve as the Local Evaluator for the collaborative?

Family Connection Partnership has no stated policy prohibiting collaboratives from handling their own evaluation or from having their local coordinator serve as the evaluator. However, given the Family Connection Partnership evaluation expectations and local expectations of a Family Connection collaborative coordinator, a coordinator may have difficulty completing a quality job with all of the tasks related to the two positions.

top

Is it a conflict of interest to have the Collaborative Chair serve as the paid Evaluator?

Again, there is no Family Connection Partnership stated policy saying it is a violation to have a chair as a paid evaluator. HOWEVER, there are very strong reasons for NOT doing it. There is potential for conflict of interest. Specifically, it often would be hard for members to know what "hat" the Chair/Evaluator is wearing at a given time. For example:

• If a partner fails to submit promised data, it could be very confusing to have the Chair/Evaluator follow up. The lines of authority may be hard to interpret and maintain if the Chair is serving in a leadership capacity and also providing a service to the collaborative as Evaluator.
  
  
• Since the evaluator technically works for the collaborative that is lead by the Chair, who would be working for whom if the roles were embodied in the same person?

In some cases, the Chair in one county is the evaluator in a different county. In these cases it is less problematic.

top

How much should our collaborative attempt to accomplish related to evaluation?

There are four types of activities the local collaborative is expected to engage in:

1. Develop and implement an annual results evaluation plan for at least one strategy or component program that is a part of the Three-Year Strategic Plan. The most recent Guidelines are posted on the FCP website in the Evaluation Toolkit. Always check the Evaluation Toolkit for the most recent version. Changes to the Guidelines and suggested tools are incorporated in the Guidelines for the Annual Community Plan. Evaluation Plan reviews will be in accordance with "Collaborative Standards."

2. Complete the annual Self-Assessment. This has been part of the 4th quarter contract deliverable since FY99. Communities submitting the quarterly electronic report for all four quarters will be excused from several sections of the annual Self-Assessment.

3. Ensure that a basic process evaluation is being conducted on all programs/activities included in the annual operating plan and submit an implementation report on a quarterly basis. The newly revised Quarterly Narrative Report (Annex C) is an easy way to accomplish this. To assist collaboratives, an electronic version is available containing the information from their FY05 annual operating plan. Use of the electronic form continues to be optional in FY05, but is strongly encouraged. One benefit for collaboratives is that information from the electronic version will be used to update the database that is frequently used to identify collaboratives who may qualify for funding opportunities.

4. Ensure that a results evaluation of at least one strategy or component program (s) is being conducted and submit a report no later than September 15 each year. This evaluation will include a process and outcome component that will assess changes in individuals, families, and/or the community resulting from the strategy being evaluated. Findings reported must be based on data where the end point is no older than the previous fiscal year. Following submission of the Report, a Guide to Using Evaluation Reports will be sent to encourage and support collaborative use of the findings in their work. Reports will be reviewed against the Evaluation Report Guidelines. Report reviews have been designed in accordance with "Collaborative Standards."

top

Who should develop the Evaluation Plan?

Consider creating a working team of the people who represent the programs and activities included in each strategy. Let them develop the core elements for their strategy. This process exemplifies best practice evaluation, which is participatory. It should increase ownership of the evaluation process and cooperation in the collection of needed information and data. After working groups have prepared their own evaluation plan, they may become part of the collaboratives's annual results evaluation plan. As multiple strategy team develop plans and evaluate their strategies, the plans can be incorporated into the collaborative's annual results evaluation plan. Most communities hire an external evaluator who will guide this process for the collaborative.

top

What are the core elements of an FCP Evaluation Plan?

1. Strategy Identifying Information (for each strategy being evaluated): strategy description; component programs; target group(s)
   
2. Guiding Questions or Indicators (Process and Results)
   
3. Methodology (Design, Instruments/Other Data Sources, Data Collection/Management, Analysis and Reporting)
   
4. Collaborative Development and Systems Change
   
5. Reporting and Using Findings

top

What does an Evaluation Plan look like?

You may organize your plan in any way that seems reasonable, as long as all the core elements, discussed in the FY 2006 Evaluation Plan Guidelines, are present in some thoughtful sequence. While there is no intent to make all evaluation plans look alike, you may want to use a tool, Results Evaluation Plan Worksheet, to help you think through the required elements. A Worksheet, Instructions and Example are provided at this link.

top

How is the Results Evaluation Plan reviewed or evaluated?

All annual Results Evaluation Plans are reviewed by the Family Connection Evaluation Team along with selected coordinators and local evaluators who received a Strategy-Level Evaluation Incentives Grant. All team members use the same criteria as they review plans. They check to see if the five core elements of a results evaluation are present and meet the requirements outlined in the Evaluation Plan Guidelines. While the process and criteria continue to evolve, the intent has been to identify which plans Exceed, Meet, or Need Work to meet Evaluation Plan Guidelines.

top