Here are some key answers to questions that collaboratives have about hiring an evaluator.

Collaboratives want to know...

What should we look for in hiring an evaluator?

Where can we find an evaluator?

What specific steps should we follow in hiring an evaluation consultant?

Generally what should we expect of our contracted evaluator?

What deliverables might our collaborative negotiate with a potential evaluator?

How can we make sure our evaluator helps us get results data?

What should we look for in hiring an evaluator?

When an Evaluation Committee and the Collaborative have decided to hire an evaluator, the next important issue is candidate credentials. The following factors might be a part of a candidate review sheet.

• Formal Academic Preparation: What is the candidate's academic background? Has he/she been trained in social science research? Is he or she skilled in research design, instrument development, data analysis, reporting...?
  
  
• Evaluation Experience: Does the candidate have experience with project management? Is the candidate skilled in both qualitative and quantitative research? Does the candidate rely upon one research method or does he/she have a large repertoire from which to draw?
  
  
• Philosophy or orientation: Does the candidate operate within a participatory approach to evaluation (in which collaborative members are encouraged to participate in evaluation activities)? Or does he/she prefer a more traditional approach (in which he/she assumes sole responsibility for the evaluation)?
  
  
• Track record, recommendations: What do past and present clients of the candidate say about the candidate's quality of work, ability to meet their needs, etc.? Have other groups chosen to retain the candidate's services for extended periods of time?
  
  
• Personal style and characteristics: Does the candidate "click" with collaborative members or are there personality conflicts? Is the candidate someone with whom members of the collaborative could work easily?
  
  
• The Needs of the Collaborative: What services and deliverables do collaborative members want from the evaluation consultant? Do they simply need someone to act as a monitor or facilitator of the evaluation process? Do they need someone who will work with collaborative members on all evaluation activities? Do they need someone who will assume responsibility for the technical aspects of the evaluation, such as instrument design and data analysis?
  
  
• Ability to Commit to the Work: Does the candidate have sufficient time to work with the collaborative or is he/she already over-extended? Does the candidate have a flexible schedule so that he/she can be readily accessible during "crunch" times? Is the candidate able to commit to a minimum of one year (preferably longer)?

top

Where can we find an evaluator?

Below are some possibilities for looking for an evaluator with the skills that you need.

• First, look around your own table. Is there someone in your collaborative with evaluation expertise? Is so, is that person willing and able to conduct evaluation activities and meet the needs of the collaborative as you've defined them?
• Examine the resources within your own community. Are there individuals in local agencies or organizations with evaluation expertise?
• Is there a nearby college or university that may have faculty or staff with evaluation expertise?
• Find out who other Family Connection collaboratives are using. You may obtain a list of local evaluation consultants by contacting the state evaluation office at eval@gafcp.org.
• Finally, explore other options. Have you heard about an evaluation firm in Georgia with expertise in systems change or grassroots initiatives? Contact the Atlanta Area Evaluation Association, which is an affiliate of the American Evaluation Association. The contact person is identified on the Affiliates link of the American Evaluation Association Web site (www.eval.org). Also look for Tips for Hiring an Evaluator at
  http://www.the-aps.org/education/promote/tipevaly.html

top

What specific steps should we follow in hiring an evaluation consultant?

• Have discussion with Evaluation Committee and Collaborative.
  • advantages and disadvantages
  • desirable deliverables
  • available resources
• Search for available candidate(s).
• Bring in one or more candidates to interview with the collaborative.
• Discuss desired deliverables and available budget.
• Clarify who will own databases resulting from data collection efforts, where the databases will be located and who will manage them.
• Reach a decision.
• Working with fiscal agent, prepare a contract reflecting deliverables, deadlines and fee payment schedule.

top

Generally what should we expect of our contracted evaluator?

The responsibilities of a hired evaluation consultant are called deliverables and they are a matter of negotiation between the hiring party - the Evaluation Committee and the Collaborative - and the potential evaluator. It is the responsibility of the Evaluation Committee to decide which evaluation tasks they want to contract for, which ones they want to maintain and which ones they'd like to work on cooperatively. Keeping in mind that best practice evaluation for the Family Connection Collaborative calls for participatory evaluation, you would be looking for an evaluator who can facilitate the evaluation process, involving stakeholders and building local evaluation capacity while they are delivering the contracted products.

top

Specifically what deliverables might our collaborative negotiate with a potential evaluator?

When contracting with an evaluation consultant, the Evaluation Committee must determine the deliverables the evaluator will be expected to produce and specify these, along with deadlines in writing the contract. Consider the following:

• An Evaluation Plan: Will the evaluator be able to work with the members of the collaborative to create the plan document in a participatory way or will he/she be solely responsible for its production? How much detail does the collaborative desire in the plan? Will the evaluation consultant insure the evaluation plan meets the guidelines developed by the state? On what date will this deliverable be due to the collaborative?
  
  
• Data Collection Instruments: Will the evaluation consultant be responsible for locating existing instruments/tools for use in the collaborative's evaluation? Will the consultant be expected to make any necessary modifications to located instruments/tools? Will you expect the consultant to produce new tools/instruments? Will you ask the evaluation consultant to involve families and youth in the development or modifications of the tools in a participatory way? In what time frame is the consultant expected to locate, modify, and/or produce the data collection instruments?
  
  
• Data/Databases: What data will the evaluation consultant be expected to collect, if any? Will the evaluator be responsible for the creation of new databases? Will the evaluation consultant be responsible for data entry into new or existing databases? What is the expected time frame for the collection of data and/or the creation of databases?
  
  
• Data Analyses: What types of analyses will the consultant be expected to conduct? How will the results of analyses be presented? By what date(s) are analyses expected to be complete?
  
  
• Evaluation Reports: Will the evaluation consultant be responsible for preparing reports of evaluation findings? Will the findings be presented verbally and, if so, to whom? Will the evaluation consultant be expected to produce one or more written reports of findings? What level of detail is expected? Will different versions of the reports be prepared for different audiences? On what dates will reports be due to the collaborative?
  
  
• Training/Capacity Building: Will the evaluation consultant be expected to provide any form of evaluation training? If so, what is the desired length and number of sessions desired? What is the expected content of the training sessions? With whom will the consultant be expected to work?
  
  
• Other: Are there other deliverables the collaborative would like to request of the consultant that do not fall within the area described above?

top

How can we make sure our evaluator helps us get results data?

Results data is important because it tells us whether our efforts are leading toward positive outcomes for families and children. Since FY 2003, all counties have been expected to conduct results evaluations on one or more strategies and submit a results evaluation report. You and your evaluator can work together in the following ways:

• Work together to write the Evaluation Plan. Ensure that results questions are asked and that plans have been made for answering those questions (i.e., selection of appropriate methods; identification of data collection instruments/data sources; etc.)
  
  
• Brainstorm with your evaluator about the ways in which results data can be collected. Since it takes time to achieve long-term results, you should consider looking for short- and intermediate-term indicators of progress. Typical short-term results include changes in knowledge, attitude, skills and aspirations. Intermediate results include actions taken, behaviors changed or practices adopted. These interim indicators of progress are also valuable because of the necessary time lag in annual updates to the benchmark database.
  
  
• Determine the deliverables related to results data that the evaluator will be expected to produce. Clearly specify these in writing, along with deliverable deadlines. Remember, your deadlines should reflect both data availability and reporting needs (when reports must be submitted, when decisions must be made, etc.)

top