Activity 7 Working with Your Tentative Quantitative Research Question
This Section has Activities that require you to work with a tentative research question. As you know, you will almost definitely change (or at least refine) your research question(s) when you write your Concept Paper, and you may not even choose to do a quantitative study (in fact, your research question will probably change for Activity 8). These Activities will teach you key concepts in research design that will serve you regardless of your later choices
Researchers are concerned with who they study—their sample—for two very different basic reasons:
(1) in order to be able to say something convincing about a population, i.e. to be able generalize claims about a sample to a population;
(2) in order to be able to say something convincing and meaningful about relationships between constructs.
In research that attempts to contribute to theory, as dissertation research must, (1) is not usually of interest. If, for example, your literature review suggests that gender or IQ scores or some other variable moderates a relationship, you will, of course, need to be sure that you include participants with the necessary characteristics (gender, IQ score, etc.) and you will need to ensure that your sample has an identity (e.g., second grade public school teachers in Madison, Wisconsin) but you do not need worry about whether your participants are selected in sufficient numbers in a specific way (e.g., randomly, stratified randomly) from a specific population to allow you to say something about the population from which the sample is drawn.
On the other hand, an inadequate literature review or chance may lead you to select a group in which a moderating variable is at work (so that you may not find a relationship that really does exist, just not for your sample) or a sample that does not meet the assumptions (even relaxed assumptions) of the statistical test you planned to use (oops!). These will be matters to address by using non-parametric statistics or to speculate about in Chapter Five of your dissertation or for future researchers to discover. As a researcher and a doctoral student in education, you must, though, know about (1) and understand the concepts involved with it. The first part of this assignment addresses this.
Researchers who attempt to contribute to theory are primarily concerned with exploring relationships among constructs, not with seeing if something that is true of a group is true of a population. They are concerned with (2)—having a sample that allows you to find a relationship among constructs. This is a matter of both the size of your sample and the sensitivity of your design. As you know, sample size and other aspects of design are intimately related, and you must think about them together when developing your research questions and planning your study.
Unless you are lucky enough to be able to obtain all the participants you need in order to do your preferred study, you will have to work back and forth between design and sample size, adding covariates or blocking to reduce error variance or maybe even changing from a between subjects to a within subjects design or making some other serious modification in your design until you arrive at a viable design that has a good chance of answering your research questions.
Unless you design your study adequately and select a sample of sufficient size, your design may be a set-up for a Type II error—failing to find a difference or a relationship that is really there—and your study may be largely a waste of time! You want to have a large enough N to find a relationship among constructs that is really there and to be able to argue that the relationship is meaningful.
There are four factors involved in calculating sample size:
- Statistical test – Your sample size is partly a function of the statistical test you use. Some tests (e.g., Chi-squared) require larger sample to detect a difference than others (e.g., ANCOVA).
- Expected/estimated Effect size – The effect size is potency of your intervention or the strength of the relationship you are investigating. For example, a psychedelic drug has a very potent effect on number and vividness of hallucinations. You may only need a single subject design to detect them. The effect of a traffic safety class taken in 2nd grade on a group of high school students may take a very large sample to detect. In the language of statistics, an effect size is the difference between the mean scores of two groups divided by the pooled standard deviation. This is called Cohen’s d. The greater difference between groups on a measure after you factor in how spread out the scores are, the more potent the intervention. You will calculate an effect size as part of the analysis of your data in order to determine that you have found something meaningful (not merely statistically significant), but in advance of doing your study, you must estimate the effect size in your study. Lipsey and Hurley (2009) describe a way to estimate effect size that many Learners will find helpful: Review the literature on the same or similar relationships or interventions to find the range of relevant effect sizes to estimate the effect size for your study.
- The alpha level is the probability of a Type I error—of rejecting the null, no difference, hypothesis when it is true—that you are familiar with. By convention this is set at p=.05. Convention may not be your best guide. As you know from the readings in Activity 3, the null hypothesis is always false and can always be rejected with a large enough sample, so a .05 level may unnecessarily require you to have a larger sample than you need. Better to use the literature and your judgment to justify an alpha level that makes sense for your study. This justification will involve looking at the danger of a Type I error versus the cost in resources of avoiding it.
- The beta level is the probability of a Type II error—of accepting the null, no difference, hypothesis when it is false, in other words, of failing to detect a difference when it is there. The main point of a power analysis is to have enough subjects and no more to detect a difference. As with alpha, you set beta based on a judgment. The convention is .2, which yields a power of .8 (1-beta).
- Trochim, W. M. K., & Donnelly, J. P. (2008): Chapter 2
- Fritz, A. E., & Morgan, G. A. (2010).
- Houser, J. (2007).
- Acheson, A. (2010).
- Piasta, S. B., & Justice, L. M. (2010).
- McCready, W. (2006).
- G*Power 3 software and documentation
- Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007).
- Mayr, S., Erdfelder, E., Buchner, A. & Faul, F. (2007).
Download G*Power and play around with it. See how changes in assumptions and parameters affect sample size estimates.
- Compare and contrast internal and external validity. Describe and give examples of research questions for which external validity is a primary concern. Describe and give examples of research questions in which internal validity is a primary concern. Discuss strategies researchers use in order to make strong claims about the applicability of their findings to a target population.
- Compare and contrast random selection and random assignment. Be sure to include a discussion of when you would want to do one or the other and the possible consequences of failing to do random selection or random assignment in particular situations.
- Explain the relationship between sample size and the likelihood of a statistically significant difference between measured values of two groups. In other words, explain why, all else being equal, as sample size increases the likelihood of finding a statistically significant relationship increases.
- Compare and contrast probability and non-probability sampling. What are the advantages and disadvantages of each?
If you do a quantitative study for your dissertation, you must estimate the sample size you will need in order to have a reasonable chance of finding a relationship among the variables stated in your research hypotheses (should one exist), given your statistical analysis(es) and assumptions/calculations of factors 2-4 above. You must do this, even if you plan to use a convenience sample (see below). There are a number of sample size calculators available. Northcentral uses G*Power, which is required in this Activity. You will use G*Power’s “a priori power analysis” function to calculate a sample size. If it yields an unrealistically large size sample, you will rethink your design and assumptions and, perhaps, use G*Power’s “compromise power analysis” to estimate a workable sample size that makes sense. If you plan on using a convenience sample, you would use both analyses as part of your argument that your convenience sample is large enough.
Main Task: Submit the Following
- Calculate the sample size needed given these factors:
- one-tailed t-test with two independent groups of equal size
- small effect size (see Piasta, S.B., & Justice, L.M., 2010)
- alpha =.05
- beta = .2
- Assume that the result is a sample size beyond what you can obtain. Use the compromise function to compute alpha and beta for a sample half the size. Indicate the resulting alpha and beta. Present an argument that your study is worth doing with the smaller sample.
- Calculate the sample size needed given these factors:
- ANOVA (fixed effects, omnibus, one-way)
- small effect size
- alpha =.05
- beta = .2
- 3 groups
- Assume that the result is a sample size beyond what you can obtain. Use the compromise function to compute alpha and beta for a sample approximately half the size. Give your rationale for your selected beta/alpha ratio. Indicate the resulting alpha and beta. Give an argument that your study is worth doing with the smaller sample.
- In a few sentences, describe two designs that can address your research question. The designs must involve two different statistical analyses. For each design, specify and justify each of the four factors and calculate the estimated sample size you’ll need. Give reasons for any parameters you need to specify for G*Power.
Include peer-reviewed journal articles as needed to support your responses to Part I.
Use the order calculator below and get started! Contact our live support team for any assistance or inquiry.[order_calculator]