Research Design

Typical Research Process
The typical quantitative study involves a series of steps, one of which is the statistical analysis
step 1:
research questions and hypotheses
step 2:
operationalize and choose measures
step 3:
choose a research design
step 4:
analyze data
step 5:
draw conclusions
We will write a custom essay sample on
Any topic specifically for you
For only $13.90/page
Order Now
causal relationship
-the goal of virtually every quantitative research study is to determine between two variables
-ideally a causal relationship would be established
-that is, does a change in X definitively lead to a predictable change in Y
Confounding Variables
-Variables always have multiple causes
-we would like to isolate only the two variables of interest and exclude the effects of all other variables
Unwanted variables
are called confounding or nuisance variables
Internal validity
the validity of findings with the research study; the technical soundness of a study, particularly concerned with the control of extraneous influences that might effect the outcome
External validity
the degree to which the findings can be inferred to the population of interest or to other populations or settings; the generalizability of the results
Both internal and external validity
are import in a study but they are frequently at odds with one another in planning and designing a study
Internal validity is
considered the basic minimum for experimental research
Internal validity facts
-is the basic minimum without which any study is not interpretable
-particularly important in experimental studies
-did, in fact, the experimental treatment (X) produce a change in the dependent variable (Y)
-lends itself to highly controlled, laboratory settings
To gain internal validity
the researcher attempts to control everything and eliminate possible extraneous influences
Threats to Internal Validity
history, maturation, testing, instrumentation, statistical regression, selection bias, experimental mortality, interaction among factors
events occurring during the experiment that are not part of the treatment
biological or psychological processes within participants that may change due to the passing of time (age, fatigue, hunger)
the effects of one test upon subsequent administrations of the same test
changes in testing instruments, raters, or interviewers including lack of agreement within and between observers
Statistical regression
the fact that groups selected on the basis of extreme scores are not as extreme on subsequent testing
Selection bias
identification of comparison groups in other than a random manner
Experimental mortality
loss of participants from comparison groups due to nonrandom reasons
Interaction among factors
factors can operate together to influence experimental results
External validity facts
-generalizability of results to what populations, settings, or treatment variables can the results be generalized
-concerned with real-world applications
-random selection is the key to controlling most threats to external validity
External validity is generally
controlled by selecting subjects, treatments, experimental situations, and tests to be representative of some larger population
Two types of External Validity
Population validity and Ecological Validity
Population Validity
refers to the extent to which the results can be generalized from the experimental sample to a defined population
Ecological Validity
refers to the extent to which the results of an experiment can be generalized from the set of environmental conditions in the experiment to other environmental conditions
Common sources of error
many possible sources of error can cause the results of a research study to be incorrectly interpreted. the following sources of error are more specific threats to validity of a study than those described previously
Threats to External Validity
interaction effects of testing, selection bias, reactive effects of experimental setting, multiple-treatment interference
Interaction effects of testing
the fact that the pretest may make the participants more aware of or sensitive to the upcoming treatment
Selection bias is
when participants are selected in a manner so they are not representative of any particular population
Reactive effects of experimental setting
the fact that treatments in constrained laboratory settings may not be effective in less constrained, real-world settings
Multiple-treatment interference
when participants receive more than one treatment, the effects of previous treatments may influence subsequent ones
Additional Threats to External Validity
-Reaction effects: Hawthorne effect, placebo effect, John Henry effect, rating effect
-Experimental effect: experimental bias effect
Hawthorne Effect
-a specific type of reactive effect in which merely being a research participant in an investigation may affect behavior
-suggests that, as much as possible participants should be unaware they are in an experiment and unaware of the hypothesized outcome
Placebo effect
participants may believe that the experimental treatment is supposed to change them, so they respond to the treatment with a change in performance
John Henry Effect
a threat to internal validity wherein research participants in the control group try harder just because they are in the control group
Rating effect
variety of errors associated with ratings of a participant or group
-halo effect
-overrater error
-underrater error
-central tendency error
Experimenter bias effect
the intentional or unintentional influence that an experimenter (researcher) may exert on a study
Types of designs
-true experimental
Non-experimental designs/ Descriptive Research
-weak experimental designs in terms of control
-no random sampling
-threats to internal and external validity are significant problems
-many definite weaknesses
-example: one-group pretest/posttest design
True experimental designs
-best type of research design because of their ability to control threats to internal validity
-utilizes random selection of participants and random assignment to groups
-example: pretest/posttest control group design
Quasi-experimental designs
-lack either random selection of participants or random assignment to groups
-lack some of the control of true experimental designs, but are generally considered to be fine
example: nonequivalent group design
Research Design Basics
-research design, or blueprint, for a study can be drawn out using a simple set of symbols proposed by Campbell & Stanley (1963)
X =
an intervention was administered (i.e., a treatment, program, IV, etc)
O =
data were collected (i.e., observed, measured, etc.)
Between Subjects design
-also called between groups or cross-sectional
-each participant is in one (&only one) of the treatments/conditions
-different groups of participants are in each treatment/condition
-typically used to study “differences”– when in application, a participant will usually be in one treatment/condition or another
Characteristics of between-subject design
-the defining characteristic of a between-subjects design is that it compares separate groups of individuals
-it allows only one score per participant (every score represents a separate, unique participant)
-each score represents a separate participant, a between subjects design is often called an independent-measures design
More characteristics of between-subject design
-between-subjects experimental design requires a separate, independent group of individuals for each treatment condition
-individuals are assigned to groups using a procedure that attempts to create equivalent groups
Goal of between-subjects experiment
is to determine whether differences exist between two or more treatment conditions (a researcher may want to compare two teaching methods (two treatments) to determine whether one is more effective than the other)
Advantages of between-subjects design
-each individual score is independent of the other scores
-participant’s score is not influenced by such factors as:
-practice or experience gained in other treatments
-fatigue or boredom from participating in a series of treatments
-contrast effects that result from comparing one treatment to another
Disadvantages of between-subjects designs
-large number of participants (problem with special populations)
-individual differences:
-characteristic that differ from one participant to another are called individual differences
-individual differences can become confounding variables
-individual differences can produce high variability in scores
Confounding variables in between subjects design
Individual differences:
-participant characteristics differ from one group to another
-participants in one group may be older, smarter, taller, than the participants in another group
environmental variables:
-characteristics of the environment differ between groups
-one group may be tested in a large room and another group in a smaller room
Equivalent groups
in a between-subjects experimental design, the researcher does have control over the assignment of individuals to groups
-the separate groups must be:
– created equally
-treated equally (except for the treatment conditions
-composed of equivalent individuals
Limiting confounding by individual differences
-random assignment,matching groups (matched assignment), holding variables constant
Random assignment (randomization)
a random process is used to assign participants to groups
Matching groups (matched assignments)
involves assigning individuals to groups so that a specific variable is balance or matched across the groups (IQ)
Holding variables constant
simply hold the variable (restrict the participant to those with with IQs between 100-110)
Within and between treatments variability
-variability between treatments
-it can be increased by increasing differences between conditions (levels)
-variability within treatments
-it is caused by individual differences
-should be minimized
Within and between treatments variability (2)
minimizing variability within treatments
-standardize procedures and treatments setting
-limit individual differences by holding a participant variable constant
-random assignment and matching
-sample size:
-using a large sample can help minimize the problems associated with high variability
Other threats to internal validity of between-subject designs
assignment bias, differential attrition, on diffusion or imitation of treatment, compensatory equalization, compensatory rivalry, resentful demoralization
Assignment bias
-groups of participants are different before the treatments
-the group assignment process produces groups within noticeably different characteristcs
Differential attrition
-attrition refers to participant withdrawl from a research study before it is completed
-differential attrition refers to differences in attrition rates from one group to another and can threaten the internal validity of a between-subjects experimental (effectiveness of a dieting program)
Diffusion or limitation of treatment
refers to the spread of the treatment effects from the experimental group to the control group (new depression therapy)
Compensatory equalization
occurs when an untreated group learns about the treatment being received by another group and demands the same or equal treatment (watching Batman in violent TV group)
Compensatory rivalry
occurs when an untreated learns about the treatment received by another group and then works extra hard to show that they can perform just as well as the individuals receiving the special treatment
Resentful demoralization
-opposite of compensatory rivalry
-occurs when an untreated group learns about the treatment received by another group and is less productive and less motivated because they resent the expected superiority of the treated group
Applications and statistical analyses of between-subjects design
-comparing only two groups of participants
-comparing means for more than two groups
Comparing only two groups of participants
-this design is referred to as the single-factor two-group design or simply two group design
-an independent-measures t test is used to determine whether there is a significant difference between the means
Comparing means for more than two groups
-single factor multiple group design may be used and analysis of variance (ANOVA) would be used for statistical analysis
-adding extra groups to a research study tends to reduce the differences between groups
Within-Groups Designs
-also called within-subjects, repeated measures, or longitudinal,
each participant is in all (every one) of the treatment/conditions
-one group of participants, each one in every treatment/condition
-typically used to study “changes” when, in application, a participant will usually be moving from one condition to another
Characteristics of within-subjects design
-a within subjects experimental design compares two or more different treatment conditions (or compares treatment and control) by observing or measuring the same group of individuals in all of the treatment conditions being compared
-a within-subjects design looks for differences between treatment conditions within the same group of participants
-a within subjects design is often called a repeated-measures design because the research study repeats measurements of the same individuals under different conditions
-it is used in experimental situations comparing different treatment conditions and also to investigate changes occurring over time
Advantages of within-subjects designs
-it requires relatively few participants
-it essentially eliminated all of the problems based on individual differences that are the primary concern of a between-subjects designs
-within-subjects design has no differences between groups
-each individual serves as his or her own control or baseline
Disadvantages of within-subjects
-time related problems:
-participant attrition, history, maturation, instrumentation, statistical regression
Disadvantages of within subjects design
carry over effects, progressive error
Carryover effects
-changes in behavior or performance that are caused by participation in an earlier treatment condition
-carryover effects exist whenever one treatment condition produces a change in the participants that affects their scores in subsequent treatment conditions (e.g. new skill from treatment 1 can influence results in treatment 2)
Progressive error
changes in participant’s behavior or performance that are related to experience over time in a research study but not related to a specific treatment or treatments (e.g. practice effects and fatigue)
Dealing with time related threats and order effects
-controlling time:
if the different treatment conditions are scheduled over a period of weeks, the chances greatly increase that the results will be influenced by some outside event (history) or maturation or change in the measurement instrument
When a within subjects design is not a good idea:
-e.g. comparing two methods of teaching reading to first-grade children (carryover effects)
Dealing w/ time related threats and order effects
-involves changing the order in which treatment conditions are administered from one participant to another
-the goal is to use every possible order of treatment with an equal number of individuals participating in each sequence
-the purpose of counterbalancing is to eliminate the potential for confounding by disrupting any systematic effects from factors related to time or the order of treatments
-e.g. with two treatments one half of the participants begins in treatment 1, then moves to treatment 2 and the other half begins in treatment 2, then receives treatment1
Applications and statistical analyses of within-subjects designs
Two treatment designs, multiple treatment designs
Two treatment designs
-a repeated-measures t test or an analysis of variance can be used to evaluate the statistical significance of the mean difference
-if the data are measured on an ordinal scale, a Wilcoxon test can be used to evaluate significant differences
Multiple-treatment designs
-with too many treatment conditions, the distinction between treatments may become too small to generate significant differences in behavior
-statistical analysis – repeated-measures analysis of variance to test for any significant differences among the treatment means
Matched-subjects designs
-each individual in one group is matched with a participant in each of the other groups
-the matching is done so that the matched individuals are equivalent with respect to a variable that the researcher considers to be relevant to the study (e.g. IQ)
-maintains all the advantages of between-subjects and within-subjects designs without the limitations of either (e.g. eliminates individual differences, time-related factors and order effects)
Properties of True Experiments
-True experiments provide the highest degree of internal validity possible
-Virtually all confounding variables should be controlled via random assignment
-Because confounds are controlled, a causal link can only be claimed when using a true experimental design
Limitations of Random Assignment
-If random assignment is so advantageous, what prevents its use?
-It may not be ethical to expose subjects to or withhold from them a treatment
–e.g., What are the effects of alcohol use during pregnancy?
-Some research questions are not amenable to random assignment
–Comparisons of naturally existing groups (e.g., males and females, ethnicity, socioeconomic differences)
If only True experiments can be causally interpreted, why even bother running non-experiments?
1st Remember that we can’t always run a true experiment !

Lots of variables we care about can’t be RA & manip – gender, family background, histories and experiences, personality, etc.

Even if we can RA & manip, lots of studies require long-term or field research that makes ongoing equivalence (also required for causal interp) very difficult or impossible.

We would greatly limit the information we could learn about how variables are related to each other if we only ran studies that could be causally interpreted.

Categorical vs. Continuous Variables
-In practice, it is not usually necessary to make such fine distinctions between measurement scales
-To distinctions, categorical and continuous are usually sufficient
-Categorical variables consist of separate, indivisible categories
-Continuous variables yield values that fall on a numeric continuum, and can (theoretically) take on an infinite number of values
Nominal Scale
-Observations fall into different categories or groups
-Differences among categories are qualitative, not quantitative
Counseling method (cognitive vs. humanistic)
Retention (retained vs. not retained)
Ordinal Scale
-Categories can be rank ordered in terms of amount or magnitude
-Categories possess an inherent order, but the amount of difference between categories is unknown
Class standing
Letter grades (A,B,C,D,F)
Likert-scales (SD, D, N, A, SA)
Interval Scale
-Categories are ordered, but now the intervals for each category are exactly the same size
-That is, the distance between measurement points represent equal magnitudes (e.g., the distance between point A and B is the same as the distance between B and C)
Fahrenheit scale of measuring temperature
Chronological scale of dates (1997 A.D.)
Standard scores (z-scores)
Ratio Scale
-Categories are ordered, but now the intervals for each category are exactly the same size
-That is, the distance between measurement points represent equal magnitudes (e.g., the distance between point A and B is the same as the distance between B and C)
Fahrenheit scale of measuring temperature
Chronological scale of dates (1997 A.D.)
Standard scores (z-scores)