In addition to this guide, we have focus group proposals, moderators’ guides, presentations and reports, along with other manuals If not, why not? Introductions - Begin the session with a set of questions that will enable focus group members to get to know each other better. Are activities conducted with the target population? When respondents are relaxed, they will open up and talk more. e. Is the target population affected by the project/ program equitably or according to the evaluation plan? The evaluator consults with all accessible stakeholders to develop specific questions that the evaluation will seek to answer. * Visual Appeal How will the program work with the target population to sustain the project/ program? As the project manager and facilitator of a focus group, your job is to fully engage the members to get their honest feedback. Chapters 2 and 3. Are they satisfied? Process evaluation questions focus on how a program is working, program performance, and involve extensive monitoring. Are objectives met? Focus group question examples. Program Evaluation Tip Sheet: Constructing Survey Questions October 2011 . 86–105). Extension Specialists in Program and Staff Development 203 Scovell Hall University of Kentucky Lexington, KY 40546-0064 Web Design: Does Everyone Know Where to Click? I am providing my consent by leaving the opt-in checked. The benefits of doing business with company are clearly presented in this web page. Thinking of the answer to the previous question, what made you say that? For example, ask them to think of three words to describe their favorite hobby. Before we decide what types of data we need (qualitative or quantitative) we need to know the bigger question of the project: What specific problem does the project or program address? 5. Methodologically, evaluation questions focus on varied assessment types. It illustrates how to order questions within a topic of inquiry. Tracy (2013) explains that within a focus group, there is a cascading effect of conversation, as comments link into each other and participants share what can be rich, emic, qualitative, data with each other and the moderator. NFI nonfood item . When respondents are relaxed, they will open up and talk more. The program evaluation was conducted by UCLA’s Center for Study of Evaluation … New York: Kluwer Academic Publishers. Program evaluation is a systematic process of obtaining credible information to be used for program assessment and improvement. What three words would you use to describe the product? Focus groups are often used in concert with other product research methods to determine not only how the product could be improved but also how the product should be marketed. IR intermediate result . If I wanted to purchase the product, I know where to click. Use the guide to answer basic questions about the focus group process. conducting a process evaluation. Format - When choosing focus group questions, be sure to include a mix of multiple choice questions to provide structure and direction and open-ended questions to allow for further exploration and discovery. Closing - Remember to thank the participants for their assistance and candor, and then ask them if there is anything that could be done to improve their experience as a member of the focus group. M&E monitoring and evaluation . If the program has been in place for a while and is thought to be relatively stable, then it may be time for a summative evaluation. If you would not recommend this product, what would you say to convince your best friend not to buy this product? Reasonable and appropriate, or realistic in the given project or program. 5. Work Assignment B-16 . Are outcomes, objectives, and goals achieved? Evaluation questions may seem intuitive, and thus be quickly developed to get to the more detailed program planning. This question is for testing whether you are a human visitor and to prevent automated spam submissions. Many projects involve online marketing and gathering feedback on a website design or feature. Focus Group Questions to Engage Your Group. 6. What features of the product disappointed you? Project level (local, activity based, “little p”) – how does, or could, the evaluation effect project or local changes?**. In Program Evaluation (pp. Overall, how appealing is this web page to you? About the AuthorAlly Krupar is an Adjunct Instructor at American University’s School of Professional and Extended Studies where she teaches Qualitative Methods in Monitoring and Evaluation. But, without well-developed, relevant, and accurate evaluation questions, developed with stakeholders connected to the problem, projects can move around a problem without addressing the most appropriate issues. Ask them about their favorite hobbies and interests and what they enjoy the most about the activity. How best should the project/ program deliver those services (or activities)? Is there anything that you would like to add or remove from the design that you have seen? Determine participants By clearly defining the purpose and outcomes of the focus group, it is easier to identify who should be invited to participate in the focus group. • The focus group needs to be appropriate for the type of participants in the group. 1. How would you rate your outlook for the economy over the next six months? Is the target population adequately reached by and involved in activities? Programs that are already running should have a theory of change, or how they think their intervention will lead to the stated outcome, objective, goal, or impact they hope to see. How could those services be provided? If I wanted to contact the company, I know where to click. Often, however, we select individuals who are able to comment on only a few aspects of the eLearning course: they provide invaluable feedback on the course aesthetics, but may not have much to say about the content, inevitably … How does the target population interact with the program? They focus on the merit, worth, or significance2 of a program or particular aspects of a program. * Ease of Use ** Adapted from Owen, J. M., & Rogers, P. J. If the problem is not identifiable by stakeholder communities, projects addressing it will not be successful. (Use a likelihood scale). Are there alternative approaches that could have the same outcomes with less cost?*. These resources are sample evaluation forms and guides to adapt for your own use. Some sample questions are: 4. Are there other populations the program should be working with? There is a trainer’s competency checklist and trainer attributes competency self-assessment. London: SAGE Publications Ltd. Additional ResourcesMorra Imas, L. G., & Rist, R. C. (2009). Instead of asking “why” you should rephrase the question to ask respondents to provide some specific examples of what they liked or did not like about the item in question. It should always include less than ten questions. What are the target population’s characteristics? Step 1: Form a working group. Specificity - Avoid vague open-ended questions that simply ask “why.” The word “why” has a negative connotation because it makes the respondent think their first answer was not good enough. surveys and focus group questions for student participants, parents, teachers (survey only), program staff, and program partners (survey only). Image by Gerd Altmann from Pixabay, How to Develop a Change Communication Plan. Training Evaluation - Forms and Questionnaires. These consultations may be informal conversations, reviewing grant requirements and terms of reference (documentation review), or semi-structured individual and/or group interviewing. Learn more about how to file a complaint about a distance program or courses. Student Focus Groups for Program Assessment: ... away from teacher evaluation. Decide which questions should be included in an evaluation questionnaire. And what can be learned? (2012). Please fill out both sides of this form: OBJECTIVES This program met the stated objectives of: 1. (Use a more or less tendency scale), Are you more or less likely to make a large purchase in the next six months? See the article about moderator guide. The evaluation questions may also be used in summative evaluations which focus on what happened after the program or project completed, i.e., were goals achieved? Please rate on a scale of 0 to 10, where 0 is extremely dissatisfied and 10 is extremely satisfied the following features: It might also be useful to have some more specific questions under Policy level – how does, or could, the evaluation impact relevant policy? Consumer sentiment and consumer confidence are the forerunners to the future success of many businesses. Assessment Tool kit: Focus Group Template Focus Group Script . Do you have any suggestions on how we could improve the product to make it more appealing? "Focus groups aim to discover the key issues of concern for selected groups. What do they think of the services? 1. eLearning focus groups are collections of individuals who represent your main learner body and need to be actively involved in your course evaluation. Evaluation questions, similar to research questions in academic research projects, guide the methods and tools used to collect data to understand the problem under investigation. How much do you agree or disagree with the statements below? Needs assessment evaluation questions may include focus on the significance of the problem, drawing on literature, previous projects, and baseline data with potential stakeholders. Please indicate your rating of the presentation in the categories below by circling the appropriate number, using a scale of 1 (low) through 5 (high). No matter why a focus group is formed, you can help your members have a more productive session by asking questions that grab their attention and get them thinking beyond the obvious. 4. Is the problem that the project/ program intends to address alleviated? Dr. Beverly Peters has written recently about qualitative interviewing and conducting and using focus groups. (Use an agreement scale). Are the headlines/links easy to read and enticing for visitors to click on? (Use an agreement scale). Some sample questions are: 3. Community Food Security Coalition ii Overview The Community Food Project Evaluation Toolkit (Toolkit) is a hands-on workbook designed to help community food project (CFP) staff conduct evaluations of their programs to develop the kind of information that will be compelling not only to fellow staff but to funders, participants and The key to creating a positive experience for your participants is by asking questions that will awaken both their creative and critical sides. What is evaluation Focus groups are structured discussions among small groups of people. One of this group’s first tasks will be to determine whether to bring in an outside evaluator or have staff conduct the evaluation.17, 18! This tip sheet is intended for programs funded through the Centers for Disease Control and Prevention’s (CDC) Division for Heart Disease and Stroke Prevention. Other suggestions in developing questions for focus group sessions include: a. This article will give you many effective focus group questions that should spark discussion. Sample questions are: Through asking questions, M&E practitioners can identify what their project specifically should address. Questions should be carefully planned but appear spontaneous during the interview. As stated above, program ... taker for focus group questions and/or activities which we develop with your program in advance. Answerable, similar to the reasonableness of a question, good evaluation questions must be able to be answered to some degree of certainty. (extremely, somewhat, or not familiar at all), Overall how satisfied are you with the product? Washington, DC: The World Bank.Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. your program is in its early stages, you may wish to focus on formative evaluation information, which will help you revise the program before undertaking a summative evaluation. * Speed Also, check out the article about basic question tips. Evaluation involves a systematic collection and analysis of data to critically examine a program for better effectiveness. The Road to Results: Designing and conducting effective development evaluations. Quality Matters Rubric A6. Contract 68-W-02-048 . FGD focus group discussions . Product Reviews: Is Everyone Completely Satisfied? The order of these questions can be shifted around. The document offers suggestions on writing the best possible survey questions—it is . Broadening the Sharing the of Designs and Benefits of Trade Methods for Impact Evaluations: Report of a study commissioned by the Department for International Development. ! Generally, a focus group is comprised of six to twelve people (Morgan, 1998b; Stewart & Thinking of your previous response, what are some specific examples that helped you formed your opinion? Generally, a facilitator leads a group of 8-10 people in a discussion about selected topics with planned questions, while allowing for interesting, new or unplanned follow up questions. Designing and Conducting Focus Group Interviews Richard A. Krueger Professor and Evaluation Leader University of Minnesota 1954 Buford Ave. … Here are some sample questions to help a project manager evaluate a new website design or concept. 2 PRACTICAL GUIDE: FOCUS GROUP DISCUSSIONS CONTENTS The decision on when to conduct FGDs and—in case of a mixed-methods approach— their sequencing in relation to the other methods is influenced by the objective of the effort; the overarching assessment, monitoring or evaluation questions the FGDs seek Find the perfect sound in seconds. KEQs usually need to be developed and agreed on at the beginning of evaluation planning - however sometimes KEQs are already prescribed by an evaluation system or a previously developed evaluation framework. From Evaluation Questions to Evaluation Findings. As the project manager and facilitator of a focus group, your job is to fully engage the members to get their honest feedback. The results can then be included in the focus group report, and can be used as a London: Sage Publications. How easy or difficult was it to navigate the content on the web page? According to Rossi, Freeman, and Lipsey (1999), evaluation questions must be: Once we have developed the larger question of the project, M&E practitioners begin to consider what data they need to answer the question, using a Theory of Change and a LogFrame. Discovering these issues can help determine which of a number of options is the preferred way forward, or to determine what are the concerns that would prevent a proposal going ahead. Do they have negative effects? (Use a numeric or satisfaction scale), What is your impression of the product? The Various Types of Group Evaluation Forms. Use this product-focused checklist for focus group discussions or interviews. What, if anything, do you find frustrating or unappealing about the product? EVALUATION OF THE EPA INTERN PROGRAM . The questions and the manner in which the focus group is facilitated will vary based on the type of participants. The focus group may also be undertaken to discover preliminary issues that are of concern to a group or community, and Similarly, formative evaluation questions look at whether program activities occur according to plan or the project is achieving its goals while it is underway. Process evaluation, or how the program addresses the problem, what it does, what the program services are and how the program operates. When they feel uneasy, they will retreat into their shells and you will not get the input that you need. (Use a numeric or the degree of appeal scale). Yes, I do want to communicate with American University via SMS. (Use a numeric or degree of likelihood scale). Student Focus Group Questions A4. Course summary evaluations, focus group questions, and expert observation tools are included. Similarly, formative evaluation questions look at whether program activities occur according to plan or the project is achieving its goals while it is underway. The first order of business when considering conducting a process evaluation is to form a working group that includes key program stakeholders. We appreciate your help in evaluating this program. We give you the insight you need to write all types of focus group questions with the help of proven examples and expert advice.