Obsidian/Zotero/001_artiklid/shaverwhatstatisticalsignificancetesting1993.md

---

category: literaturenote citekey: shaverwhatstatisticalsignificancetesting1993 title: "What Statistical Significance Testing Is, and What It Is Not" authors: "Shaver, James P." year: 1993 date: 1993-00-00 1993 publication: The Journal of Experimental Education url: "https://www.jstor.org/stable/20152383" zotero_key: 4CZFKK4B zotero_storage: TEKH4T6T collections: transpordi_planeerimine folder: 001_artiklid firstAuthor: "Shaver, James P."

status: converted

What Statistical Significance Testing Is, and What It Is Not

Author(s): James P. Shaver

Source: The Journal of Experimental Education , Summer, 1993, Vol. 61, No. 4,

Statistical Significance Testing in Contemporary Practice (Summer, 1993), pp. 293-316

Published by: Taylor & Francis, Ltd.

Stable URL:https://www.jstor.org/stable/20152383

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at https://about.jstor.org/terms

Taylor & Francis, Ltd. is collaborating with JSTOR to digitize, preserve and extend access to The Journal of Experimental Education

What Statistical Significance Testing Is, and What It Is Not

JAMES P. SHAVER Utah State University

ABSTRACT. A test of statistical significance addresses the question, How likely is a result, assuming the null hypotheses to be true. Randomness, a central assumption underlying commonly used tests of statistical significance, is rarely attained, and the effects of its absence rarely acknowledged. Statistical significance does not speak to the probability that the null hypothesis or an alternative hypothesis is true or false, to the probability that a result would be replicated, or to treatment effects, nor is it a valid indicator of the magnitude or the importance of a result. The persistence of statistical significance testing is due to many subtle factors. Journal editors are not to blame, but as publishing gatekeepers they could diminish its dysfunctional use.

THE USE OF TESTS OF STATISTICAL SIGNIFICANCE in educational and psychological research has been under attack for over 30 years. Among the critics have been Skinner (1956), Bakan (1967), Meehl (1967), various authors in Mor rison and Henkel (1970), Seeman (1973), Signorelli (1974), and Cronbach (1975). In 1978, Carver's excellent critique, "The Case Against Statistical Sig nificance Testing," was published. I touched on the matter as it related to the lack of productivity of educational research in 1979, participated in an American Edu cational Research Association symposium in 1980 with a title similar to the one for which this paper was prepared?"Tests of Statistical Significance: Read dressing Their Role"?and then wrote a two-part article cautioning educational practitioners about misinterpretations of statistical significance (Shaver, 1985a, 1985b).

Thus, much of the present paper is a story told before. That has presented a quandary in regard to how extensively to develop various concepts and to cite supporting sources for ideas that seem well established, if not well accepted. I have probably been both overly frugal and excessive on both accounts at different

points in the paper. At any rate, Carver (1978, p. 379) noted that all of the criticisms of tests of statistical significance appeared to have had little effect. The situation has not

changed since then. A quick perusal of research journals, educational and psycho logical statistics textbooks, and doctoral dissertations will confirm that tests of statistical significance continue to dominate the interpretation of quantitative data in social science research. Surely one characteristic of statistical significance test ing is that it is an enduring?in the face of the devastating criticism, perhaps it would be better to say, relentless?phenomenon in educational and psychologi cal research.

The thrust of this paper, like so many written before it, is that the dominance of statistical significance testing is dysfunctional, because such tests do not pro vide the information that many researchers assume they do. Statistical signifi cance testing also diverts attention and energy from more appropriate strategies, such as replication and consideration of the practical or theoretical significance of results. In the hope that the accumulation of criticism will have an effect, I respond again in this paper to the question of what statistical significance testing is and what it is not. Possible reasons for the persistence of statistical significance testing are also discussed briefly, and proposals are presented for action by journal editors to moderate the negative effects of statistical significance testing, if not eradicate their inappropriate use.

What Statistical Significance Testing Is

A test of statistical significance is, at its very simplest in the hybrid (Fisher fused with Neyman-Pearson) hypothesis-testing model dominant in social science research (Gigerenzer et al., 1989, pp. 207-209), a procedure for determining how likely a result is assuming a null hypothesis to be true. Somewhat more precisely, our commonly used tests of statistical significance (z ratios, t ratios, and F ratios, such as in the analysis of variance or covariance) are procedures for determining the probability (usually at a prespecified level called alpha) of a particular result, assuming the null hypothesis to be true, given randomization and a sample of size n (i.e., the sample size used in the study). Here I use randomization to include both random sampling and random assignment, although some authors use ran domization to refer only to assignment. Much of the discussion that follows is in terms of random sampling; however, as I point out later, random assignment also

meets the randomness assumption. Individual elements of the above statement about tests of statistical significance are important, although often overlooked. First, the result of a test of statistical significance is a probability statement, often expressed as a dichotomy in terms of whether the probability was less or more than the alpha level. Second, the test is based on the assumption that the null hypothesis is true. That is, the theoretical sampling distributions against which results are compared (the normal distribu tion, the t distributions, the F distributions, the chi-square distributions) are gen erated by assuming that sampling occurs from a population, or populations, in which the null hypothesis is exactly true. Third, despite some claims to the con

trary (e.g., Thompson, 1987), randomization is a fundamental assumption under lying the use of these tests of statistical significance. Fourth, sample size is a crucial consideration, because the statistical significance of a result will depend on the number of cases on which it is based. Each of these elements will be al luded to in the discussion that follows.

Randomness as an Assumption

As Glass and Hopkins (1984) stated, "Inferential statistics is based on the as sumption of random sampling from populations" (p. 177). Elsewhere, they refer to "random samples" as one of the "building blocks for hypothesis testing" (p. 202), and they specify random sampling as an essential assumption for the use of the one-sample z test for a mean (p. 205), the t tests for two independent means (p. 231) and for the difference between means from correlated observations (p. 241), and the F ratios for differences between variances (p. 261) and in the

analysis of variance (e.g., pp. 342, 445). Randomization is important because it helps to ensure the independence of observations, or, equivalently, errors (Glass & Hopkins, 1984, p. 350). Despite what is commonly assumed, however, randomness does not guarantee indepen dence. For example, observations almost certainly will not be independent when treatments have been delivered to subjects in a group setting, as is common in educational research.

In addition, randomization is essential to the typical tests of statistical signifi cance. Randomness (i.e., random error) is the basis for the sampling distributions against which results are compared. Use of, for example, a t distribution to answer the question "How likely is this particular result under the null hypothesis?" will not yield a meaningful probability statement if the sample or samples are not random. Repeated random sampling (or assignment) yields known sampling dis tributions. Nonrandom sampling does not, nor does the comparison of a nonran dom sample to a randomly generated sampling distribution, provide a valid state

ment of probability of occurrence. The indispensability of randomness may be more evident when the question being addressed in tests of statistical significance is stated as "How representative is this sample (sample statistic) of the population (population parameter) specified in the null hypothesis?" Without randomness, that question cannot be answered validly using the common tests of statistical significance. As Glass and Hopkins (1984) put it:

The method of random selection of samples will ensure, within a certain known mar gin of error, representativeness of the samples and hence will permit establishing lim

its within which the parameters are expected to lie with a particular probability. The ability to estimate the degree of error due to chance (sampling error) is an important feature of a random sample. ... It is not possible to estimate the error with accidental sampling and many other sampling strategies since they contain unknown

types and degrees of bias in addition to sampling error, (p. 177, emphasis in the original)

In that context, I found it baffling that Thompson (1987) would assert that "sig nificance testing imposes a restriction that samples must be representative of a population, but does not mandate that this end must be realized through random sampling" (pp. 8-9) and then go on to discuss "comparing known sample charac teristics with known population characteristics to build some warrant for an as sumption of representativeness" (p. 9). The description of sample characteristics in order to allow generalization to populations from which a random sample was not drawn is an important, and often neglected, element of research reporting (Shaver & Norton, 1980a, 1980b). In fact, such description of sample characteris tics is crucial, even if a random sample was used, to assist readers in making generalizations?both because the sample may not have been drawn from a popu lation in which a research user is directly interested and, equally important, be cause a random sample may not represent well the population from which it was drawn.

Such descriptions are not, however, a substitute for random sampling; the pur pose of randomness is not to ensure representativeness (if that could be done, there would be no need for an inferential test) but to allow the specification of the probability that a sample came from a population with a hypothesized parameter (or, conversely, to estimate a range of probable values for a parameter). In short, random sampling addresses solely the representativeness of samples in the long run; it does not ensure that all of the characteristics of a particular sample, includ ing the dependent variable(s) under investigation, will be the same as those of the population, only that (whether assessed or not) they will differ only by chance from the population characteristics. Of course, in conventional significance test ing with random sampling from a population in which the null hypothesis is true and with alpha set at .05, 5% of the time the researcher will incorrectly conclude that the sample did not come from the population specified in the null hypothesis; however, a conclusion that the sample was not representative of the specified pop ulation (with a as the criterion) would be correct.

In essence, the mistake is in viewing randomization as an outcome (i.e., repre sentativeness) rather than as a process (i.e., sampling in which every member of the population has an equal chance of being selected for the sample). This error is not uncommon and can even be found in statistics books. For example, Fergu son and Takane (1989) provided an example of the use of chi-square "to test the representativeness of a sample where certain population values are known" (p. 218). They analyzed a set of data composed of 200 individuals drawn (the process is not specified) from the population of Montreal. The difference between popula tion and sample frequencies for three levels of national origin?French, English, and other?was statistically significant at the .01 level. Ferguson and Takane concluded, erroneously, "that the sample . .. cannot be considered a random sample" (p. 219). Of course, the question of randomness is not a matter of a chi

square goodness-of-fit test, but of the process by which the sample was drawn. Had they said that the sample could not be considered a "representative" sample, their logic would have been correct, although statistical significance has dubious validity as the criterion for such a decision.

Random assignment. The term randomization is used somewhat ambiguously in discussions of experimental design and tests of statistical significance. Some (e.g., Hays, 1973, p. 562) use randomization to refer generally to the application of random processes in designing experimental studies, encompassing both random sampling and random assignment. On the other hand, there are those who use randomization to refer only to random assignment to treatments, (e.g., Fergu son & Takane, 1989, p. 245; Winer, Brown, & Michels, 1991, pp. 7-8). Focusing on random sampling, and not discussing random assignment, is also common. That is the case with Glass and Hopkins (1984), who discussed tests of statistical significance in terms of random sampling but not random assignment to treat ments.

Although random assignment is not common in educational research, it is more common than random sampling (Shaver & Norton, 1980a, 1980b). As Berk and Brewer (1978) pointed out, with random assignment, researchers can appropri ately compare their results against the sampling distributions frequently used in tests of statistical significance (also see Hays, 1973, p. 562; Winer et al., 1991, p. 8). Whereas random sampling ensures chance sample differences from the source population on all characteristics, random assignment ensures that differences be tween the groups on all variables, assessed or not, are nonsystematic. Again, there is no assurance that the groups are not different on any important variable. In fact, a test of statistical significance may indicate, even after random assignment, that the groups are sufficiently different on the variable(s) under analysis that, follow ing the logic of the inferential test, one should conclude that they did not come from the same population.

Repeated random assignment to groups from the same population will result in a sampling distribution of mean differences with a mean equal to zero. The z, t, or F distributions can be used in tests of statistical significance to determine the probability of a particular mean difference (or difference in some other statistic) under the null hypothesis of no difference. Of course, with only random assign ment, a test of statistical significance provides no basis for generalization to a specific population, although it can be regarded as addressing the question of whether the groups under analysis can be regarded as samples from the same

undefined population. Just as random sampling will not ensure that a particular sample is representa tive of the population from which it is drawn, random assignment does not provide assurance that the resulting groups are identical to one another or, put alterna tively, that the groups are equivalent splits from the same hypothetical infinite population (McHugh, 1964). Random sampling into treatment groups addresses

both the estimation of population parameters and the likelihood of associations between treatment group membership and preexisting characteristics; random

assignment addresses only the latter. It should be noted that, without random assignment, an exact probability test of statistical significance can be based on a sampling distribution generated by randomly splitting the available sample into all possible combinations of the size of the groups in the study and computing the relevant statistic for each combina tion. The researcher can then ask, using that distribution, how likely it is that an obtained result would have occurred by chance (e.g., Berk & Brewer, 1978; Winch & Campbell, 1969). Such probability tests are rarely reported in the litera ture, however. Traditional tests of statistical significance are typically applied, often ignoring the assumption of randomization.

Violations of randomness. Unfortunately, statistics textbook authors tend to ignore the effects of violating the randomness assumption. For example, Glass and Hop kins (1984), who were explicit about the importance of random sampling if not random assignment, discussed the effects of violating the assumptions of normal population distributions and homogeneous population variances, but not the ef fects of lack of randomness on the appropriateness of drawing a conclusion about a particular result using the theoretical sampling distribution. However, they often mentioned the effect of lack of randomness on the independence of scores (e.g.,

Glass & Hopkins, 1984, p. 353). One reason that randomness is often ignored may be that the examination of the effects of violating that assumption is a formidable task because it involves all sample-population characteristics, not only the dependent variable, as with the normality and homogeneity of variance assumptions. As I pointed out earlier:

To enumerate every potentially relevant variable and specify its relationship to the dependent variable(s), being certain that no crucial variable was overlooked, in order to investigate the effects of nonrandomness on probability statements presents in superable difficulties. (Shaver, 1980, p. 6)

Nevertheless, the general conclusion that lack of randomness can be overlooked, as is common in the reporting of educational research, must be challenged. As Winer et al. (1991) stated in discussing analysis of variance assumptions:

Violating the assumption of random sampling of elements from a population and ran dom assignment of the elements to the treatments may totally invalidate any study, since randomness provides the assurance that errors are independently distributed within and between treatment conditions and is also the mechanism by which bias is removed from treatment conditions, (p. 101)

However, their table summarizing the "consequences of violation of assumptions of the fixed-effects ANO VA" (p. 102) includes randomization only in terms of the independence of observations (errors), perhaps because the other conse quences are unknown and unknowable in practice.

An analogy. To sum up, the commonly used tests of statistical significance provide the researcher with limited information: "How likely is this result, assuming the null hypothesis to be true and with randomization (random sampling and/or as signment) and a sample of size nV Without randomness, the result of the test of statistical significance is meaningless or, at best, its relevance to a statement of probability is indeterminate. Frequently in educational research, the researcher goes into a school or schools, obtains available groups (with neither random sam pling nor assignment), collects data (sometimes with, and sometimes without, a treatment), and then conducts tests of statistical significance. The results of such inferential tests are essentially meaningless, unless one is interested in compari sons to an abstract standard of probability as indicated by the question "What would be the probability of the obtained result if random samples had actually been drawn?"

Consider an analogous situation: A person walks into a room and sees 10 coins lying on a table. He observes 9 heads and 1 tail and wonders if the coins are biased. So he asks if this particular arrangement of coins is a likely chance occur rence. Having a statistics book handy, he turns to Pascal's triangle and finds that the probability of obtaining 9 or more heads out of 10 coins is 11/1024 or .011. Because that probability is less than the traditional .05 alpha level, he concludes that his result is not a likely chance occurrence under the null hypothesis of a 50 50 split in tails and heads and that he has evidence that the coins are biased. However, what he clearly should conclude is that if he had flipped each coin?or, alternatively, had flipped 1 coin 10 times?the probability of the particular result or one more extreme occurring by chance is less than 5%. But he has no evidence as to the bias in the particular set of coins because he does not know if they were all flipped. That is, the process by which they arrived in their positions is un known. The theoretical (binomial) distribution provides only an abstract standard of little relevance because the data were not produced in such a way as to meet a basic assumption for use of the distribution. (Describing the physical properties of the coins vis-?-vis biasedness would be a substitute for flipping and use of the

binomial distribution, not proof that the binomial distribution was applicable.) Just as application of the binomial distribution could not provide valid informa tion about possible bias in the observed coins, so educational researchers who use nonrandomized groups cannot obtain valid information about the probability of a group difference under the null hypothesis using a common test of statistical significance. The t distribution, for example, has no more relevance to differences between available groups than the binomial distribution does to the possible bias in coins found lying on a table.

What Statistical Significance Testing Is Not

A test of statistical significance used without randomization, then, does not yield valid information about the probability of a result under the null hypothesis. The following brief listing of what tests of statistical significance cannot do for the researcher is, therefore, based on the assumption that data come from a design that includes randomization?either random sampling or random assignment. As noted above, with randomization, a test of statistical significance provides a re searcher with information on the probability of a result assuming the null hypoth esis to be true and given the sample size. On the other hand, a test of statistical significance does not provide information on a number of matters of interest to researchers, even though it is often presumed to do so.

What About This Sample?

A test of statistical significance provides a statement of probability of occur rence in the long run, with repeated random sampling (or assignment), under the null hypothesis. As Carver (1978) argued, it is a fantasy to believe that such a test speaks to whether a particular result is a chance occurrence. That is, a test of significance provides the probability of a result occurring by chance in the long run under the null hypothesis with random sampling and sample size n\ it provides no basis for a conclusion about the probability that a particular result is attribut able to chance.

Even with random samples drawn from a population in which the null hypothe sis is true and in which scores are distributed according to the assumptions for the statistical model, with alpha set at .05, 5% of the time the researcher will conclude that a result is not a likely occurrence under the null hypothesis, thus making a Type I error. But the researcher has no way of knowing for any particular result whether an error is being made or if the sample was drawn from a population in which the null hypothesis was not true. That is why, according to Tukey (1969), R. A. Fisher's "standard of firm knowledge was not one very extremely significant result, but rather the ability to repeatedly get results significant at 5%" (p. 85). Replication is essential to confidence in the reliability (reproducibility) of a result, as well as to conclusions about generalizability (external validity) (e.g., Camp bell & Jackson, 1979).

What About Hol

A test of statistical significance does not indicate the probability that the null hypothesis is true or false. It provides the researcher with information in regard to the likelihood of a result given that the null hypothesis is true; it does not indicate the likelihood that the null hypothesis is true given a particular result. Carver (1978) and J. Cohen (1990) are among those who have cautioned against

that fallacy. Unfortunately, authors of statistics books and research reports frequently make statements about rejecting the null hypothesis based on one statistically signifi cant result. That is too absolute a conclusion. If any inference about the null

hypothesis is to be drawn from one test of statistical significance, it should be stated in terms of evidence for the plausibility of the null hypothesis, not an abso lute rejection. Even though the "rejection" decision may be set implicitly in a probabilistic context?with, for example, a .05 chance of a Type I error?that

qualification is, typically, quickly ignored. Conversely, a test of statistical significance also does not provide information on the probability that an alternative hypothesis is true or false (see, e.g., Carver, 1978). The sampling distribution is based on the null hypothesis; no evidence is provided as to the likelihood of the result occurring under alternative hypotheses.

What About Treatment Effects?

One of the more egregious errors is to conclude that a test of statistical signifi cance indicates whether a treatment being studied had an effect. Clearly, a test of statistical significance addresses only the simple question of whether a result is a likely occurrence under the null hypothesis with randomization and a sample of size n. At most (as noted above), a statistically significant result has to do with the probability of the result in the long run (i.e., over repeated samples), not with

whether the particular result did or did not occur under the null hypothesis. It seems terribly obvious that a test of statistical significance does not speak directly to causality. Even if a researcher were willing to conclude, following a test of statistical significance, that a low probability result did not come from a population in which the null hypothesis is true (i.e., the result was not a chance occurrence under H0) and the researcher did not make a Type I error in doing so, the test of statistical significance provides no evidence as to the cause of the result. That is a matter of design, not of statistical inference. Not even the selection threat to internal validity is perfectly controlled by random sampling or assignment; simply by chance pretreatment groups can be significantly (statistically and/or

practically) different on one or more relevant variables. The fallacy of concluding that a statistically significant result indicates a treat ment effect (as often seen, for example, in statements such as "The statistically significant difference between means indicates that Treatment A was more effec tive than Treatment B") is likely perpetuated by statistics books in which it is suggested that tests of statistical significance will address questions such as: "Is the treatment effective? Does Drug X reduce hyperactivity more than a placebo?

Does anxiety level influence test performance?" (Glass & Hopkins, 1984, p. 230). Statements such as "The mean practice effect was highly significant" (Glass & Hopkins, 1984, p. 242), following a t test comparing pre- and posttest means, can mislead as well, as can a statement that a confidence interval for a difference between means indicates the range of values that encompass "the true treatment effect (ix, - |ul2)" (p. 236). The ambiguity of the term effect as used in inferential statistics, especially analysis of variance, to refer to a comparison (e.g., a main effect) does not help. As J. Cohen and P. Cohen (1983) pointed out, the "causal

implication of the term effect" can obscure the fact that "causal interpretations are never warranted by statistical results, but require logical and substantive bases" (p. 210).

Nontextbook discussions to inform researchers about inferential statistics can misinform, as well. For example, J. Cohen (1990) commented that "everyone knows that... all [statistical significance] means is that the effect is not nil..." (p. 1307). And, in a generally sound piece, Berk and Brewer (1978) said: "If this null hypothesis [p,, = |x2] is rejected, it can be concluded that presence or absence of the specific diploma treatment contributes to group differences in income" (p. 209). Such statements create and perpetuate an erroneous view of the relationship between tests of statistical significance and causality.

What About Magnitude and Importance?

Despite frequent conclusions to the contrary in research reports (as noted, e.g., by Bracey, 1991; Harcum, 1989), statistical significance indicates neither the mag nitude nor the importance of a result. Statistical significance is only information in regard to the probability of a result under the null hypothesis, given randomi

zation and a sample size of n. Sample size is, of course, a primary concern in this particular instance of what a statistical significance test is not. For example, with n = 10 and a = .05 for a nondirectional test, a correlation of .63 is statistically significant; with n = 50, an r = .28 is needed; with n = 100, an r = .20; with n = 500, an r = .09; with n = 1,000, an r = .06; and, with n = 10,000, an r = .02 is statistically significant at the .05 level (Glass & Hopkins, 1984, p. 549). Alternatively, with a standard devi ation of 10 and n = 20, a difference of 9.4 between two independent means is necessary for statistical significance at the .05 level in a nondirectional test; with n ? 100, a difference of only 4.0 is required, and with n = 1,000, a difference of

only 1.2 is required. Obviously, very small and trivial results as well as important ones may be statis tically significant. As Meehl (1967), along with a host of other writers, has pointed out, with a large enough sample and reliable assessment, practically every association will be statistically significant. Conversely, with a very small sample, very few results will be statistically significant. Therefore, to know only whether a result is statistically significant tells one virtually nothing about the magnitude

or importance of the result. It is so commonly stressed that the statistical significance of results is directly a function of sample size that one can only wonder at the number of articles in which results are either interpreted as important because of statistical significance or in which the probability level appears to be taken as an indication of magnitude, as suggested by the use of terms such as highly significant when the probability is .01 or less. Even statistics textbook authors make the latter mistake, as in Glass

and Hopkins's (1984) references to a "mean practice effect [that] was highly sig nificant" (p. 242) and a "multiple correlation coefficient [that] is highly signifi cant" (p. 314) when the probabilities were .001 or less.

Effect sizes. Statistical probability is not, then, a useful indicator of magnitude of a result because it is dependent on sample size. This deficiency became especially clear in efforts to prepare quantitative research-literature summaries. Glass (1976) proposed that effect sizes?metrics for the magnitude of results that are indepen dent of sample size and scale of measurement?be used in reporting results. (Re sult size would be a better term to avoid cause-effect implications, but effect size is probably too firmly embedded in social science research usage to change now [Shaver, 1991]).

Effect sizes, too, may be misinterpreted, however. Despite J. Cohen's (1988, pp. 12-13) cautions to the contrary, researchers have ignored the arbitrariness of his conventions for low, medium, and large effect sizes (e.g., .2, .5, and .8, respec tively, for standardized mean differences). Yet, as has been made clear by a num ber of authors (e.g., Glass, McGaw, & Smith, 1981, p. 104; Shaver, 1985b, 1991), an effect size of 1 or larger may reflect a trivial result. The dependent variable may lack value (benefit), the construct or characteristic may not have been validly assessed (Messick, 1989), or the result may be too costly to produce or its reliabil ity may be in doubt. Substituting sanctified effect size conventions for the sancti fied .05 level of statistical significance is not progress.

Power analysis. One reaction to the relationship between sample size and statisti cal significance has been the call for statistical power analysis. J. Cohen (1988), among others, has indicted the low power?that is, the probability of obtaining a statistically significant result if there is a difference in the population?of much psychological and educational research, due to the small sample sizes typically used and the small-to-moderate effect sizes that are common.

In conducting a power analysis, the population value is not known (otherwise, a test of statistical significance would be irrelevant); an effect size must be esti mated, either as the population value or the minimum result that would indicate practical significance. Once this is done, the researcher can manipulate sample size, alpha level, whether the alternative hypothesis is directional or nondirec tional, and even the magnitude of the estimated effect size, to obtain a desired level of power. All seem to be rather meaningless exercises, an intellectual game, once the effect size of interest has been specified (a point to which I will return). The concern should be whether an anticipated effect size is obtained and whether it is obtained repeatedly, not how to manipulate design and analysis elements so that the result, if obtained, will be statistically significant. To focus attention on the appropriate issue, S. A. Cohen and Hyman (1981) have insisted that their doc toral students specify an effect size, not just an alpha level, as a criterion against which to judge results.

How About Replicability?

Something else that a test of statistical significance is not is an indication of the probability that a result would be obtained upon replication of the study. A test of statistical significance yields the probability of a result occurring under the null hypothesis, not the probability that the result will occur again if the study is repli cated. Carver's (1978) treatment should have dealt a death blow to this fallacy, too.

With the randomization model, a researcher has no way of knowing how close any one result is to the actual population parameter. The more extreme a finding is in the true sampling distribution, the less likely it is to be reproduced upon replication. That is, with the continued drawing of random samples (and espe cially with continued random assignment when not conjoined with random sam pling, so there is lack of control of the hypothetical populations from which the assignments are being made), there will be considerable fluctuation in the statis tics obtained. Moreover, sampling (i.e., random) error is compounded by the ex perimental circumstances that make it difficult in educational settings to imple ment a previous design with fidelity, including the valid reproduction of a treatment, across replications. These difficulties, coupled with the limited infor mation from a randomness-based probability statement, are why statistical sig nificance does not indicate the reliability, or replicability, of a result. Unfortu nately, the contrary assumption has been common, encouraging a one-shot approach to research (Shaver, 1979).

Statistical significance not only provides no information about the probability that replications of a study would yield the same result, but is of little relevance in judging whether actual replications yield similar results. Similar probabilities could be based on quite different results, and different probabilities could be based on identical results (see Rosenthal, 1991). For that reason, a recommenda tion that statistical significance and the statistical power of tests of statistical sig nificance, in addition to effect sizes, be reported in replications (Rosenthal, 1991) makes little sense: The question of interest is whether an effect size of a magni tude judged to be important has been consistently obtained across valid replica tions. Whether any or all of the results are statistically significant is irrelevant.

The Persistence of Tests of Statistical Significance

Tests of statistical significance, then, yield little valid information pertinent to the questions of interest in most social science research. Projects, such as the National Assessment of Educational Progress, in which random sampling tech niques are used to estimate population parameters, are notable exceptions. As I noted 13 years ago (Shaver, 1980), the continued reliance on inferential statistics in social science research in the face of the many criticisms would surely make an excellent study in intellectual history and the sociology of ideas. There is, un

doubtedly, a complex web of causal factors; it is an oversimplification to claim that the fault lies with journal editors who discourage the submission of reports of statistically nonsignificant results and of replications and who do not accept manuscripts in which quantitative data are reported without tests of statistical sig nificance.

Errors of Omission

There are, undoubtedly, many subtle factors involved in the continued use and misuse of tests of statistical significance. For example, although I have yet to see an article in which it is argued that a test of statistical significance does more than provide information on the likelihood of a result occurring under the null hypothesis, an error of omission is common. That is, there is a tendency to not remind readers of the central position of random sampling in the logic of the commonly used tests of statistical significance (e.g., J. Cohen, 1990) or to men tion randomness at one point in the discussion but ignore it elsewhere (e.g., Berk & Brewer, 1978, pp. 190-191; Carver, 1978, pp. 381-382, 385; J. Cohen, 1990, e.g., pp. 1307, 1310; S. A. Cohen & Hyman, 1981). Although the authors may understand that randomness is central to the meaningful interpretation of tests of statistical significance, many readers will miss that point when it is omit ted from much of the discussion.

Statistics Courses

That researchers might not read with randomization into a statement such as "What [statistical significance] tells us is the probability of the data, given the truth of the null hypothesis ..." (J. Cohen, 1990, p. 1307) is not surprising in light of the training that many receive. Statistics textbooks are sometimes the "source of myths and misconceptions" that lead researchers to overinterpret inferential statistics (Brewer, 1985). Moreover, statistics courses, like textbooks, are fre quently geared almost solely to helping students recognize the types of analysis situations to which various inferential statistics are applicable and to interpreting the results in terms of statistical significance. They are not aimed at encouraging students to question the uses or usefulness of tests of statistical significance, much less to reflect on the role of random sampling in their inferential tests.

The tendencies toward statistical overanalysis and unthinking interpretation are being exacerbated by the growing availability, with personal computers in practi cally every office, of black-box statistical solutions: The researcher puts data in and receives output, often without understanding the assumptions underlying the analysis. J. Cohen's (1990) caution that "simple is better" (p. 1305) is refresh ing advice that will likely be ignored with complex data analyses being so easily accomplished. A possible cause is the education that many social science re searchers receive. Their statistics courses in particular are too often not philosoph

ical in orientation. They are trained to be mechanical appliers of statistical tech niques, rather than educated to be thoughtful, critical users (Dar, 1987).

Graduate Committees

Graduate students' training is often reinforced by their experiences with their graduate committees. Faculty often insist on the application of tests of statistical significance when they are not appropriate, such as in the situation where the student will collect data on a total population. As alleged by Brewer (1985), there is also an unthinking insistence by graduate committees on the importance of results being statistically significant at the .05 level. The myth persists that only results that are statistically significant at or beyond the .05 level are important, and that beyond means of greater magnitude and/or of greater importance (p. 246). Graduate students are frequently anxious that they might not obtain statisti cally significant results and, consequently, their research will not be acceptable to their committees. Misunderstanding of the meaning of statistical significance is perpetuated, along with misapprehension of the importance of negative (no differ ence) findings in the production of knowledge, a point for discussion on another day.

Being "Scientific"

Among the reasons for the insistence on tests of statistical significance is that they provide a facade of "scientism" in research. For many in social science re search, being quantitative is equated with being scientific. Perhaps numbers give a sense of security and of seemingly firm results, especially when researchers confuse the numbers with reality (Shaver, 1991, p. 94). An extension is the assumption that mathematics, including inferential statistics, is essential to sci ence?despite the fact that some scientists (for example, physicists) and many outstanding psychologists (such as Wundt, Piaget, Lewin, and Skinner) managed very well without inferential statistics (J. Cohen, 1990, p. 1311; Gigerenzer et al., 1989, pp. 206, 211). Unfortunately, attention is distracted from the serious consideration of educational and psychological research as science, including the role of replication (Shaver, 1979).

Ritualized Practice

The use of statistics has been criticized as ritualistic, because it is so often unthinking practice (e.g., J. Cohen, 1990; Seeman, 1973; Shaver, 1987) and has even been compared to religious ritual (Salsburg, 1985). What Salsburg noted in regard to the religion of statistics in the medical profession applies equally well to social science research. There are the high priests, those who have the secrets to the mysteries of God, salvation, and eternal life (i.e., of statistical inference),

and they are not lightly or easily challenged. The religious practitioners go to church (start up their computers) and go through their rituals (compute their tests of statistical significance) hoping for salvation (findings at the .05 level). Overall, a sense of security pervades; the practitioners have confidence in the high priests' access to the underlying mysteries. This confidence allows the practitioners to avoid dealing with the foreboding uncertainties of the meaning of life (or of the meaning of statistical analyses).

The term mystery is particularly significant in this discussion because, to many users, tests of statistical significance really are mystical; the probability level is a magical indication of whether results are important?that is, have achieved the sacred .05 level of statistical significance. And, just as people are often raised to accept religious affiliation and practice, rather than encouraged to question underlying tenets, so do statistics courses and other graduate education experi ences, as noted above, often foster the unthinking acceptance of statistical sig nificance practices and overinterpretations. The acceptance of the ritual diverts attention from important matters such as the role of replication in knowledge veri fication, the reporting of effect sizes to indicate the magnitude of results, and the weighing of benefits and costs in making judgments about the practical signifi cance of results.

Social Context

Considerable social pressure also contributes to the continued unthinking use of tests of statistical significance. I can attest, as I am sure many readers can, to the subtle as well as overt pressures that have led me to have my own doctoral candidates report tests of statistical significance when they were basically mean ingless additions to interpretation (a practice I have abandoned). Rejecting the use of statistics is likely to be particularly threatening?that is, it raises fears of rejec tion by, even humiliation at the hands of, academic peers?for those who do not feel comfortable with mathematics or with their philosophical understanding of

the logic of statistics. Such influence is present in the broader research domain as well. Glass et al. (1981, pp. 197-199) discussed why inferential statistics are not appropriate for meta-analyses in which the reviewers are analyzing data from populations or near populations of studies. They noted, however, that Tukey had "chided them for not presenting standard errors of the more important averages" (p. 199), despite their reasons for not doing so. Tukey maintained that "regardless of such complica tions, some rudimentary inferential calculations would be informative and use ful"(p. 199). After that revelation, Glass et al. presented a lengthy discussion of

the application of inferential statistics in meta-analysis. Another part of the social context is professors who teach inferential statistics courses and researchers whose reputations have depended on statistically signifi cant results. Both have psychological investments, as well as personal stakes, in

the continued legitimacy of tests of statistical significance. Carver (1978) was correct: The influence of statistical significance testing will not be easily dimin ished because "too many have a vested interest in it" (p. 397).

Paradigmatic Effects

Breaking out of ritual, then, is not only a problem for those at the lower levels of statistical practice. As Thompson (1987, 1989) noted, paradigms, including accepted ways of thinking about research, are difficult to examine. They come to be taken for granted as natural thought, and they carry normative implications for what is appropriate thinking. Paradigms dominate the thinking of the high priests as well as their followers, in science as well as in educational and psychological research, and are resistant to any change.

As Lightman and Gingerich (1992) noted, there is in science a strong prefer ence for "explanations that are mechanistic, logical, and calculable" (p. 694)? certainly an apt description of tests of statistical significance. Lightman and Gin gerich also suggested that even in the face of anomaly, scientists?and one might assume, educational and psychological researchers?are conservative, "reluctant to change their explanatory frameworks" (p. 694). The explanations for this in ertia include social and cultural factors?as noted above and as discussed by Bar ber (1961)?such as the force of shared beliefs and commitments, the influence of and desire for professional prestige, resistance to nonspecialists, and adherence to schools of thought?and psychological factors, such as being comfortable with the familiar and desiring to avoid the discomfort of cognitive dissonance (Fes

tinger, 1957). The resistance to paradigmatic change may be so strong that facts are not even acknowledged as anomalous and thus a reason for change. Tests of statistical sig nificance seem to fit in that category. Despite all of the critiques, the unpleasant facts about the limited applicability of tests of statistical significance seem not to be recognized as anomalous with their use; thus, the need to upset established practices is avoided.

As one example of a paradigmatic influence in social science research, Thomp son (1987, p. 4) cited the unthinking use of analysis of variance, which leads individuals to think in terms of the statistical significance of differences between or among means, instead of regression techniques that focus attention on relation ships. Other examples of what appear to be paradigmatic influence are not too difficult to find. I noted above the continued inappropriate application of inferen tial statistics in meta-analysis when the reviewer has not sampled from a popula tion of studies, but has done an exhaustive literature search. Sophisticated discus sion of the computation of standard errors and establishing confidence intervals in meta-analysis (e.g., Hedges & Olkin, 1985; Rosenthal, 1984; Wolf, 1986) and the use of those inferential statistics in reporting meta-analytic results (e.g.,

Schlaefli, Rest, & Thoma, 1985) seem to me to be the result of a relentless para digmatic influence (Shaver, 1991).

Power analysis again. Another example comes from power analysis. Even those who are vocal critics of tests of statistical significance seem unable to shake loose from the nagging thought that tests of statistical significance might have more meaning than they accord to them in their criticisms. S. A. Cohen and Hyman (1981) strongly attacked the use of statistical significance and advocated hypothe sis testing based on the prior specification of a critical effect size?"the minimum value ... that the researcher has defined as educationally, or scientifically or prac tically significant" (p. 60). Yet, they also recommended the continuation of power analyses and the reporting of probability levels. J. Cohen (1990) (to whom, of course, much of the emphasis on power analysis can be attributed, based on his 1969 book) laid out a devastating criticism of statistical tests of the null hypothe sis, but then went on to advocate power analysis. His discussion illustrated the notion that once one has accepted the lack of information provided by tests of

statistical significance, power analysis is a vacuous intellectual game. It is difficult to argue with J. Cohen's (1990) contention that researchers should plan their research or with his proposal that a "tentative informed judgment" (p. 1310) be made about the population effect size under investigation. What is ques tionable is his recommendation that the planning include the risk in regard to a Type I error that the researcher is willing to accept and the statistical power that is desired, so that once these items have been specified, "it is a simple matter to determine the sample size you need" (p. 1310). Moreover, if "the sample size is beyond your resources, consider the possibility of reducing your power demand or, perhaps, the effect size, or even (heaven help us) increasing your alpha level" (p. 1310). What is the purpose of power analysis and the arbitrary manipulation of criteria in order to help ensure that the researcher will obtain a desired level of probability, when statistical significance has so little meaning? In this context, concern over whether to state a directional or nondirectional alternative hypothe

sis (Pillemer, 1991) seems to me to be an equally empty exercise. S. A. Cohen and Hyman (1981) also recommended the same sort of intellectual game playing?that is, after "usually" setting a critical effect size, they "set n which is often a feasibility decision..., then juggle alpha and beta error risks to try to maximize power as close to 80% (beta error = 20%) setting alpha at .05, .06, .10, etc., depending on what the power tables tell us ..." (p. 53). Why should researchers "focus on Critical ES rather than on statistical significance without

giving up the latter" (p. 64, emphasis in original)? If effect sizes are important because statistical significance (probability) is not an adequate indicator of the magnitude of result (or much of anything else), why play the game of adjusting research specifications so that a statistically significant result can be obtained if a prespecified effect size is obtained? As Carver (1978) pointed out, "Researchers should ignore statistical significance testing when de

signing research; a study with results that cannot be meaningfully interpreted

without looking at the p values is a poorly designed study" (p. 394). Another possible illustration of paradigmatic influence was mentioned earlier: Rosenthal's (1991) recommendation that, along with effect sizes, statistical sig nificance and the statistical power of tests of statistical significance be reported in replications. That makes little sense. The question of interest is whether an effect size of a magnitude judged to be important has been consistently obtained across replications of adequate fidelity, not whether the result from a replication was statistically significant or whether the design had adequate power for a result to be statistically significant. Carver (1978) was correct: "Replicated results auto matically make statistical significance unnecessary" (p. 393).

Journal Editors

Another explanation for the persistence of statistical significance testing is the influence of journal editors. It is commonly claimed (see, e.g., Kupfersmid, 1988; Neuliep, 1991) that researchers tend not to submit results that do not reach the .05 level of statistical significance often because they deem such findings to be unimportant, but also because they believe that journal reviewers and editors are unlikely to accept the results, even with accompanying effect sizes. It is also be

lieved that journal editors have policies against publishing replication studies. Publication is crucial to success in the academic world. Researchers shape their studies, as well as the manuscripts reporting the research, according to accepted ways of thinking about analysis and interpretation and to fit their perceptions of what is publishable. To break from the mold might be courageous, but, at least for the untenured faculty member with some commitment to self-interest, foolish. As gatekeepers to the publishing realm, journal editors have tremendous power. As Carver (1978) and Kupfersmid (1988) have argued, a decline in statistical significance testing is not likely to occur until journal editors take a stand against it.

What Should Editors Do?

Editors are, of course, caught up in the tangled web of influences that have been sparsely discussed above. They are not immune to the allure and emotional power of ritual, and they may themselves be somewhat in awe of the high priests, perhaps because editors, too, may not be comfortable in their understanding of or their capacity to challenge tests of statistical significance. Like the rest of us, editors have difficulty separating themselves from their educational rearing and do not like to take public stances that might result in opprobrium from their academic

colleagues. It may be unrealistic, as Carver (1978) recognized, to expect journal editors to become crusaders for an agnostic, if not atheistic, approach to tests of statistical

significance. What editors can accomplish is certainly limited by the research ecology within which they work. They should, however, be familiar with the long standing criticisms of statistical significance testing, such as those cited at the beginning of this paper, and the implications for editorial policy and practice should be given serious consideration. The result should be policy that does not rely on the assumed epistemological validity of tests of statistical significance and the offshoot, power analysis (as Thompson's, 1987, proposed model does). Following are guidelines for such a policy and for editorial practice.

Statistical Significance

In general, authors should be encouraged, even required, to minimize statistical significance in their analyses and interpretations. Reports of tests of statistical significance that are not based on randomized samples (randomly selected and/or randomly assigned) should not be published. For those studies in which random ization was an element in the design, authors should be restrained from interpreta tions that go beyond the legitimate conclusion in regard to whether the result is a likely occurrence assuming the null hypothesis to be true. The use of tests of statistical significance to reject the null hypothesis, to declare a particular result to be nonchance under the null hypothesis, to indicate the probability that an alter native hypothesis is true, as indicators of the probability that results are replicable, as measures of the magnitude of results, or as indications of treatment effective ness should not be accepted.

Effect Sizes

Even with randomness, authors should be required to state expectations for (hypotheses about) quantitative results as critical effect sizes (S. A. Cohen & Hy man, 1981). (Effect sizes other than the standardized mean difference are avail able [see, e.g., J. Cohen, 1988; Shaver, 1991].) Effect sizes should be required in reporting the findings from quantitative research, along with descriptive statistics (and verbal descriptions) that will help readers to interpret results and to evaluate the authors' interpretations and conclusions about the populations and settings to which the study findings might be generalized. In short, studies should be pub lished without tests of statistical significance, but not without effect sizes. And, it should be made clear that, with effect sizes specified, power analysis is not relevant.

Equally important, authors should be required to provide justification for their interpretations of the practical significance of effect sizes. There already is a ten dency to use criteria, such as J. Cohen's (1988) standards for small, medium, and large effect sizes, as mindlessly as has been the practice with the .05 criterion in statistical significance testing. Consideration of the value of the outcome? including the validity of assessment?and the human as well as financial costs in

producing the outcome should be mandatory. In addition, authors should be asked to confront the reproducibility of the result as an element of its practical signifi cance (Shaver, 1991).

Replication

The mention of reproducibility leads directly to replication, widely agreed upon as a crucial element of science, but largely missing from reports of social science research (e.g., Shaver & Norton, 1980a, 1980b). Editors should not only actively encourage the reporting of replications, but in many instances demand replication before results can be published. In doing so, however, they should not confuse replication with reviews of research. Even if done in a meta-analytic framework, reviews are not equivalent to replications, despite implications to the contrary by some authors (e.g., Bangert-Drowns, 1986, p. 398; Carlberg & Miller, 1984, pp. 9, 10; J. Cohen, 1990, p. 1311; Fiske, 1983, p. 67; Hedges & Olkin, 1985, p. 3; Hunter & Schmidt, 1990, p. 37; Jackson, 1980, p. 445; Rosnow & Rosenthal, 1989, pp. 1280-1281). As I have noted elsewhere (Shaver, 1991), "The post hoc assembly of studies that have few, if any, planned connections and [that have] unknown differences among them is [not] an adequate substitute for pur posely designed replications" (p. 91). Reviews of the literature have their place,

but they do not have the explanatory power of planned replications. A proposal for a shift in editorial policy to an emphasis on replications raises several issues. One is the matter of space: It may take more journal space to pub lish study descriptions that are adequate to allow replication and to describe repli cations sufficiently to allow the evaluation of their fidelity. Moreover, with replica tions given higher priority, fewer original studies could be published, requiring more critical judgments about study validity and importance. In the long run, the outcome should be better studies, an increased number of replication studies, and greater knowledge productivity. However, careful attention to standards will be crucial, raising other issues to be addressed by journal editors.

All studies are not worth replicating; replication will not make a trivial study worthwhile. And not all replications of potentially important studies will be worth publication. Criteria must be specified by which to judge which initial studies are nontrivial, when a direct replication is adequately replicative, when a sufficient number of independent replications have been published to establish the reliabil ity of a finding, and whether a systematic replication makes a substantial contribu tion to generalizability (e.g., Rosenthal, 1991). Kupfersmid's (1988) proposal that manuscripts first be submitted to editors without the Results and Discussion sec tions, so that the publication decision would be focused on study justification and design quality, is as relevant to replications as to initial studies. The logistics in volved in handling acceptances and later full submissions might, however, place unrealistic demands on editors.

Type of Significance

An easily instituted editorial policy, proposed by Carver (1978, p. 395), is to insist that authors always use a modifier for the term significant. Authors should not be allowed to make statements such as "a confidence interval . .. tells you incidentally whether the effect is significant ..." (J. Cohen, 1990, p. 1310). As any reader of the literature is likely to have observed, it is more common to use

the term significant without a modifier than with one. Insisting upon a modifier would make it more evident to authors that they should contemplate whether their results have educational or practical signifi cance as well as statistical significance or, conversely, whether results lacking sta tistical significance might not have educational or practical significance. At the same time, readers would be alerted against overinterpreting a statistically sig nificant result as necessarily practically significant. Sensitivity to the important epistemological issues underlying the use of tests of statistical significance might also be increased.

Guideline Examples

Another rather modest editorial action that might have important outcomes would be to delete misleading examples from guidelines for authors. For example, as Becker (1991) pointed out, the Publication Manual of the American Psycholog ical Association (1983) does not represent a good model of reporting, with horren dous examples such as "The first grade girls reported a significantly greater liking for school" (p. 81) and "The analysis of variance indicated a significant retention interval effect" (p. 81). Moreover, the Manual discourages "the reporting of neg ative results" (p. 19), thus promoting the idea that only statistically significant results are important.

Conclusion

J. Cohen (1990, p. 1311) suggested that changes in the way people conceptual ize, conduct, and report research take time, as evidenced by the over 40 years that elapsed before Gossett's t test made its way into statistics textbooks. However, criticisms of statistical significance testing have been around longer than 40 years, and the practice seems to continue unabated, despite a slight increase in the re porting of effect sizes. It is probably too much to hope that 10 years hence the participants in an AERA or an APA symposium on tests of statistical significance will be able to cite evidence that the criticisms are having a noticeable impact on researchers' planning of analyses and reporting of quantitative results.

The reasons for the persistence of tests of statistical significance are complex. The blame for the continuation of their use cannot be laid on journal editors alone. Nevertheless, given the role of editors in determining what gets published, they

could be a powerful influence for more rational use of tests of statistical signifi cance. Possible actions range from a rather modest insistence on the use of a modifier with the term significance to the more far-reaching policy that reports of tests of statistical significance be restricted to those based on randomized sam ples. Researchers would be encouraged to analyze more carefully the importance of their results and to demonstrate empirically the reliability and generalizability

of study outcomes. There are sufficient logical grounds, laid out in authoritative criticisms, on which to base changes in editorial policy to minimize, if not eliminate, the use of tests of statistical significance. No doubt action to limit the inappropriate use of tests of statistical significance would be stressful for editors, given the strength of statistical significance testing in the dominant social science research paradigm and the social and psychological pressures that encourage the continuation of the inferential ritual. Journal publication is, however, the most focused point in the process of research production and reporting, and editors have a historic opportu nity to influence the course of educational and psychological research and the production of research-based knowledge in a way not available to researchers, textbook authors, or statistics professors. Modifications in publication policies have great potential for effecting the changes in the use of statistical significance that have been called for over and over but have been largely ignored. The ques tion is, who will lead the way?

NOTE

Kenneth E. Bell, David G. Gibson, Perry J. Sailor, and Matt Taylor provided helpful comments on the paper. D. Richard Cutler's incisive critique was especially provocative and useful.

REFERENCES

Bakan, D. (1967). On method: Toward a reconstruction of psychological investigation. San Francisco:

Jossey-Bass. Bangert-Drowns, R. L. (1986). Review of developments in meta-analytic methods. Psychological Bul

letin, 99, 388-399.

Barber, B. (1961). Resistance by scientists to scientific discovery. Science, 134, 596-602. Becker, G. (1991). Alternative methods of reporting research results. American Psychologist, 46(6), 654-655.

Berk, R. A., & Brewer, M. (1978). Feet of clay in hobnail boots: An assessment of statistical inference

in applied research. Evaluation Studies Review Annual, 3, 190-214.

Bracey, G. W. (1991). Sense, non-sense, and statistics. Phi Delta Kappan, 73(4), 335. Brewer, J. K. (1985). Behavioral statistics textbooks: Source of myths and misconceptions? Journal of Educational Statistics, 70(3), 252-268.

Campbell, K. E., & Jackson, T. T. (1979). The role of and need for replication research in social

psychology. Replications in Social Psychology, 7(1), 3-14. Carlberg, C. G., & Miller, T. L. (1984). Introduction. The Journal of Special Education, 18, 9-10.

Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48(3), 378-399.

Cohen, J. (1969). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.

Shaver _315

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

• Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12), 1304-1312. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
• Cohen, S. A., & Hyman, J. S. (1981, April). Testing research hypotheses with critical ES instead of statistical significance in educational research. Paper presented at the annual conference of the
• American Educational Research Association, Los Angeles. Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist,
• 30, 116-127. Dar, R. (1987). Another look at Meehl, Lakatos, and the scientific practices of psychologists. Ameri
• can Psychologist, 42(2), 145-151. Ferguson, G. A., & Takane, Y. (1989). Statistical analysis in psychology and education (6th ed.). New York: McGraw-Hill.
• Festinger, L. (1957). A theory of cognitive dissonance. Evanston, IL: Row, Peterson. Fiske, D. W. (1983). The meta-analytic revolution in outcome research. Journal of Consulting and
• Clinical Psychology, 51, 65-70. Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J., & Kr?ger, L. (1989). The empire of chance: How probability changed science and everyday life. Cambridge: Cambridge University Press.
• Glass, G. V (1976). Primary, secondary, and meta-analysis of research. Educational Researcher,
• 5(10), 3-8. Glass, G. V, & Hopkins, K. D. (1984). Statistical methods in education and psychology (2nd ed.).
• Englewood Cliffs, NJ: Prentice-Hall. Glass, G. V, McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
• Harcum, E. R. (1989). The highly inappropriate calibrations of statistical significance. American Psy chologist, 44(6), 964.
• Hays, W. L. (1973). Statistics for the social sciences (2nd ed.). New York: Holt, Rinehart and Winston. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.
• Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in re search findings. Newbury Park, CA: Sage. Jackson, G. B. (1980). Methods for integrative reviews. Review of Educational Research, 50,
• 438-460. Kupfersmid, J. (1988). Improving what is published: A model in search of an editor. American Psy
  • chologist, 43(8), 635-642.
• Lightman, A., & Gingerich, O. (1992). When do anomalies begin? Science, 255, 690-695. McHugh, R. B. (1964). Need the randomized block design be replicated? The Journal of Experimental Education, 33(2), 169-174.
• Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philoso phy of Science, 34, 103-115.
• Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5-11.
• Morrison, D. E., & Henkel, R. E. (Eds.). (1970). The significance test controversy: A reader. Chi cago: Aldine.
• Neuliep, J. W. (Ed.). (1991). Replication research in the social sciences. Newbury Park, CA: Sage. Pillemer, D. B. (1991). One- versus two-tailed hypothesis tests in contemporary educational research.
• Educational Researcher, 20(9), 13-17.
• Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage. Rosenthal, R. (1991). Replication in behavioral research. In J. W. Neuliep (Ed.), Replication research
• in the social sciences (pp. 1-30). Newbury Park, CA: Sage. Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological sciences. American Psychology, 44, 1276-1284.
• Salsburg, D. S. (1985). The religion of statistics as practiced in medical journals. The American Statis tician, 39(3), 220-223.

• Schlaefli, A., Rest, J. R., & Thoma, S. J. (1985). Does moral education improve judgment? A meta analysis of intervention studies using the Defining Issues Test. Review of Educational Research,

• 55,319-352.

• Seeman, J. (1973). On supervising student research. American Psychologist, 28, 900-906.

• Shaver, J. P. (1979). The productivity of educational research and the applied-basic research distinc tion. Educational Researcher, 8(), 3-9.

• Shaver, J. P. (1980, April). Readdressing the role of statistical tests of significance. Paper presented at the annual meeting of the American Educational Research Association, Boston.

• Shaver, J. P. (1985a). Chance and nonsense: A conversation about interpreting tests of statistical significance, Part 1. Phi Delta Kappan, 67, 57-60.

• Shaver, J. P. (1985b). Chance and nonsense: A conversation about interpreting tests of statistical

• significance, Part 2. Phi Delta Kappan, 67, 138-141. Erratum, 1986, 67, 624. Shaver, J. P. (1987, October). Mandates for curriculum and instruction from educational and psycho logical research. Keynote address presented at the annual meeting of the Northern Rocky Mountain Educational Research Association, Park City, UT

• Shaver, J. P. (1991). Quantitative reviewing of research. In J. P. Shaver (Ed.), Handbook of research on social studies teaching and learning (pp. 83-95). New York: Macmillan.

• Shaver, J. P., & Norton, R. S. (1980a). Populations, samples, randomness, and replication in two social studies journals. Theory and Research in Social Education, 8(2), 1-20.

• Shaver, J. P., & Norton, R. S. (1980b). Randomness and replication in ten years of the American Edu cational Research Journal. Educational Researcher, 9(1), 9-15.

• Signorelli, A. (1974). Statistics: Tool or master of the psychologist? American Psychologist, 29, 774-777.

• Skinner, B. F. (1956). A case history in scientific method. American Psychologist, 11, 221-223.

• Thompson, B. (1987, April). The use (and misuse) of statistical significance testing: Some recommen dations for improved editorial policy and practice. Paper presented at the annual meeting of the American Educational Research Association, Washington, DC. (ERIC Document Reproduction Service No. ED 287 868)

• Thompson, B. (1989). The place of qualitative research in contemporary social science: The impor tance of post-paradigmatic thought. In B. Thompson (Ed.), Advances in social science methodology

• (Vol. 1, pp. 1-42). Greenwich, CT: JAI Press. Tukey, J. W. (1969). Analyzing data: Sanctification or detective work? American Psychologist,

• 24(2), 83-91. Winch, R. F., & Campbell, D. T (1969). Proof: No. Evidence? Yes. The significance of tests of sig

• nificance. The American Sociologist, 4, 140-143. Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statistical principles in experimental design

• (3rd ed.). New York: McGraw-Hill. Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Beverly Hills, CA: Sage.