Go on. Explore!

lvl 0 • 0 xp

lvl 1

Home Market Decks Interests

Program Evaluation

memorize.ai (lvl 286)

(0)

39 cards

()

Section 1

Preview this deck

2. Program Monitoring Questions

Front

1 / 39

Programming

0.0

0 reviews

5			0
4			0
3			0
2			0
1			0

Active users

All-time users

Favorites

Last updated

6 years ago

Date created

Mar 1, 2020

Cards (39)

Section 1

(39 cards)

2. Program Monitoring Questions

Front

1. Is the intended target population being reached? 2. Is the program being implemented as planned? 3. What are the most appropriate strategies for promoting program implementation? 4. What are the key elements of this program?

Back

2. Research Grand vs. Research Contract

Front

Research Grant: The researcher develops the idea and proposes it to the funding agency along with why the idea is significant and how the study will be implemented Research Contract: The funding agency specifies what they want done and how they want it done, and prospective contractors make the case for how well they could execute the project

Back

Placebo Effect

Front

Expectations of the participant causes an effect

Back

3. Research vs. Evaluation

Front

Research: Intended to develop theory: Focus on creating new knowledge Evaluation: Intended to make judgement about the worth of (summative) or improve the value of (formative) a program, product, material or other intervention

Back

2. Description

Front

Describe a phenomenon in depth. This may be in terms of quantitative methods (i.e., means and standard deviation) or qualitative methods (i.e., Case Study). -What are the main barriers to accessing health care in a denied population?

Back

13. Distinguish between internal and external validity of research design

Front

Internal Validity: Concerned with the extent to which 2 variable are causally related External Validity: Concerned with the extent to which causal relationships can be generalized to other people, places and times

Back

4. Hypothesis Testing

Front

To test if two variables are or are not related statistically -There is no difference in rates of reading remediation for participants who received the intervention vs. those who did not -There is no relationship between knowledge about HIV transmission and self report sexual behaviors

Back

12. and list at least 3 threats to this kind of design validity

Front

Threats: 1. Low statistical power 2. Violation assumption of statistical tests 3. Fishing and the error rate problem

Back

3. Identify and describe three abuses of evaluation and give on example of each

Front

1. Ingratiating: doing an evaluation with the intent to obtain positive results -designing an instrument with the following rating scale (Outstanding, Excellent, Very good, Good) 2. Coercive: Doing an evaluation with the intent to obtain negative results -designing an instrument with the following rating scale (Extremely Poor, Very Poor, Poor, Fair) 3. Bureaucratic: Doing an evaluation with the intent of accountability to the funding source -Devoting extensive amount of time to keeping track of all activities of all staff all of the time

Back

2. Describe three broad classes of factors that influence the outcome of an evaluation study

Front

1. Theory and Program 2. Adequacy of Design and Analysis 3. Implementation -Many programs fail because they are not implemented well.

Back

5. Causal Explanation

Front

To determine the causal mechanisms that exaplin the relatoinship between two or more variables (Ex: Why does A cause B?) -Why does cigarette smoking cause lung cancer? -Why does expressive writing reduce anxiety and depression?

Back

1. Conceptual Definition vs. Operational Definition

Front

Conceptual Definitions: Defined in a qualitative way intended to communicate interpretation and meaning Operational Definitions: Defined in terms of how it is measured

Back

17. Outline the major steps to be achieved in conducting a research or evaluation study

Front

1. Formulate the questions 2. Demonstrate the significance of the question: so what? who cares? -Synthesis current knowledge -Demonstrate gaps in current knowledge -Describe how the proposed study will fill the gaps -Describe the relationship between the specific aims and longer term objectives 3. Formulate the research design: -From whom and when will data be collected? 4. Formulate the sample size and sampling plan 5. Formulate the data collection plan 6. Formulate intervention methods, if applicable 7. Formulate the data analysis plan 8. Collect and Analyze data 9. Present results, conclusions, implications and recommendations

Back

18. Identify and describe 5 functions of research and give one example of each 1. Exploration:

Front

Improve understanding of a topic about which very little is known -What is the cholesterol level that most doctors think drug treatment should be initiated?

Back

8. Distinguish 2 definitions for process analysis and give an example of what kind of data would be collected for each

Front

1. Process of implementation: -What are the key aspects of implementing the program under study? -Which learning activities and materials were used? 2. Causal Processes Involved: -What are the variables that mediate the relationship between the independent (ex. program) and depending (ex. outcome) variable? -Did students who learned more content information and skills perform better on the outcome under study?

Back

15. Identify and define four threats to the external validity of research design

Front

External Validity: If there is an actual relationship, can it be generalized to other people, places, and times Four Threats: 1. Interaction of selection (people) and treatment 2. Interaction of history (time) and treatment 3. Interaction of setting (place) and treatment 4. Interaction of testing and treatment: -Is applying the treatment in the real world going to work as well as it did in testing? Did the testing/priming/etc have an effect on the success of the program? -is the program going to do just as well when it is not accompanied to the pretest?

Back

Experimenter Bias

Front

Preconceptions or other characteristics of the experimenter causes an effect

Back

John Henry Effect

Front

When people in a control group learn they are being deprived they sometimes react with compensatory rivalry

Back

9. List 10 preliminary questions to be considered when planning an evaluation study?

Front

1. What are the questions addressed by the study? 2. Is the study necessary? 3. Will the study provide information I need? 4. Can my study be incorporated into a larger project? 5. How much is already known about my questions? 6. What have previous researchers learned? 7. What have been the weaknesses of previous studies? What is: 8. The target population? 9. The overall design 10. The time and cost required? What are: 11. The political and practical considerations 12. Audiences you intend to serve? 13. Assumptions, theories and limitations? 14. The possible (good and bad) side effects? 15. Exactly what can be measured? 16. Can this be achieved with reasonable degree of reliability and validity?

Back

Law of the instrument

Front

When an instrument becomes so popular that it is used whether or not it is appropriate

Back

1. Impact Assessment Questions

Front

1. To what extent did the program achieve what it was intended to achieve? 2. Were the objectives achieved? --If so, did the achievement of the objective result in the expected impact? 3. How useful is the program and the theory upon which it is based?

Back

14. Identify and define seven threats to the internal validity of research design

Front

1. History (historical event; i.e. traumatic event) 2. Selection Bias (unfavored population selection) 3. Maturation (changes with age, i.e. attention span) 4. Testing (Hawthorne effect; behavior changes because of observation) 5. Instrumentation (i.e., different examiners with different levels of training) 6. Statistical Regression (extreme scores may be by chance; should be retested) 7. Attrition (i.e. people dropping out of a trial) 8. Diffusion of treatment 9. Compensatory Rivalry 10. Resentful demoralization

Back

11. and provide three reasons why this information is important.

Front

1. To evaluate the quality of the study and therefore determine the confidence that can be placed in conclusions 2. The enable others to replicate the study 3. To contribute to development of new methods

Back

6. Describe 10 barriers to conducting evaluations

Front

Lack of: 1. Money 2. Time 3. Administration Support 4. Technical Assistance 5. Reliable and valid instruments to measure the constructs of interest 6. Ethical Constraints 7. Neglect to measure program implementation Dilemmas: 8. Rigor vs. Significance 9. Experimental vs. Placebo 10. Internal vs. External Validity

Back

12. Explain the concept of statistical conclusion validity

Front

Concerned with determining whether 2 variables are related. Several threats to this type of validity that may result in a calculation: that 2 variables are related, which in reality they are not (Type 1 Error) Or that 2 variables are not related when in reality they are (Type 2 Error)

Back

Misplaced Precision

Front

Exercising an over amount of precision in some aspects of the research design while not paying sufficient attention to other aspects

Back

16. Demonstrate an understanding of the following sources of error when given definition or examples: Halo Effect

Front

Positive past experiences result in favorable future evaluations (i.e., good first impression)

Back

4. Efficiency Evaluation Questions

Front

1. Are the benefits derived from the program worth the costs? 2. Are other programs more efficient? 3. How could the program be run more efficiently?

Back

3. Program Planning Questions

Front

1. What is the nature, scope, magnitude, severity and urgency of a problem? 2. Who is primarily affected by this problem? 3. What are alternative strategies that can be used to solve this problem? 4. To what extent are each o f the alternative strategies acceptable to the target population? Questions tend to resolve around three issues: (Importance, Feasibility and Desirability)

Back

10. Compare and contrast the terms in each of the following sets:

Front

1. Conceptual Definition vs. Operational Definition 2. Research Grant vs. Research Contract 3. Research vs. Evaluation 4. Limitations vs. Deliminations

Back

Hawthorne Effect

Front

Participation in a study or novelty causes an effect. The reactive effects of testing.

Back

11. Outline the information that would be included when describing the methods section of an evaluation study

Front

Procedures: 1. Design 2. Setting and subjects 3.Inclusions/exclusion criteria 4. Sampling 5. Intervention methods (if applicable) 6.Measurements 7. Data collection methods 8. Data analysis methods

Back

Limitations vs. Delimitations

Front

Limitations: Factors that threaten the internal validity of the study (ex: threats to internal validity) .These factors are not created by research, but she or he attempts to identify and control for them. Deliminations: Ways in which the investigator intentionally narrows the scope of his or her research with respect to who and what is being studied as well as how it is being studied

Back

5. List five conditions that help to establish a causal relationship? (Does A cause B? What is the evidence?)

Front

1. A and B co-vary--they are relatable 2. There is a does relationship between A and B 3. If A causes B, A must precede B 4.Relationships between A and B are shown consistently by different researchers studying different groups of different people at different times 5. Rival Hypothesis (other possible causes of B) are ruled out 6. There is a theoretical reason(s) why A would cause B.

Back

1. Explain the concept of research design, including 3 distinct sources of variance:

Front

1. Maximize the experimental Variance -Maximize experimental variance: we maximize experimental variance by having groups being as different as possible with respect to the variable under study (ex. the program) -Experimental Variance: variance attributed to the independent variable under study 2. Minimize Extranous Variance -Minimize Extranous Variance: we minimize extraneous variance by identifying and controlling threats to design validity -Extranous variance: Variance attributable to factors not related to the variable under study 3.Minimize error Variance -Minimize error Variance: We minimize error variance by selecting appropriate measurement methods and applying them in an appropriate context Error Variance: Variance attributed to random fluctuations

Back

4. List three questions that might be addressed by each of the following four categories of evaluation:

Front

1. Impact Assessment Questions 2. Program Monitoring Questions 3. Program Planning Questions 4.Efficiency Evaluation Questions

Back

6. And an additional 5 that make evaluation of school health education particularly problematic

Front

1. Long term nature of objectives 2. Controversy about what should be evaluated School Health Education is: 3. Designed to influence health (rather than illness or sick role behavior) 4.Attempts to influence many different and unrelated behaviors 5. Addresses fairly large groups and focuses on youth only Public School Health Educators: 6. Are generally not trained in evaluation theory or methodology 7. Could feel threatened by adopting behavioral criteria 8. Are not trained in behavioral change theory or practice There is difficulty in: 9. Observing health related behaviors in school 10. Determine whether appropriate health related behaviors result from school health education

Back

3. Prediction

Front

Predict relationship between variables, or what will happen to one or more variable in the future? -What is the relationship between exposure to cell phone and incidence of brain cancer? -Will aggressive behaviors among adolescents in high school increase in the future?

Back

7. Distinguish between formative and summative evaluation

Front

Formative Evaluation: -Evaluating program during initial development and use for the purpose of improving the final product Summative Evaluation: -Evaluation programs to make judgments about their worth

Back