Hypothesis Testing: Formulas and Solved Examples

What is hypothesis testing?

A statistical hypothesis describes a population or a probability model. For example, a manufacturer might claim a mean bulb lifetime of 1,000 hours. A sample cannot reveal that mean with perfect certainty; instead, it shows how far the observed result lies from the claim relative to sampling variability.

The null hypothesis, H₀, states the reference claim. By contrast, the alternative hypothesis, H₁ or H_a, describes the departure you want to detect. A test then measures how unusual the evidence would look under the null model and its assumptions.

One-sided and two-sided hypotheses

In a two-sided test, the question is whether the parameter differs in either direction. In contrast, a one-sided test asks whether it exceeds or falls below a threshold. Choose the direction before examining the results, because selecting the favorable tail afterward changes the error properties of the procedure.

For a one-sided test, calculations usually use the boundary value μ = μ₀. Therefore, introductory solutions often show that equality as shorthand for the full null hypothesis. The broader inequality still matters when you explain the research question.

What a p-value tells you

A p-value measures how extreme the observed test statistic is under a specified null model, including outcomes at least as extreme in the relevant direction. However, it does not give the probability that the null hypothesis is true. Nor does it directly measure the probability that chance caused the result.

For example, a two-sided p-value of 0.014 means that the null model assigns about 1.4% probability to test statistics at least as extreme as the observed one. That interpretation depends on the test and its assumptions. Consequently, the number cannot replace a review of the study design or data quality.

The American Statistical Association also emphasizes that statistical significance does not measure effect size or practical importance. Therefore, report the estimated difference and its uncertainty alongside the p-value, rather than treating a threshold as a complete scientific conclusion.

Method reference: American Statistical Association: Statement on Statistical Significance and P-Values.

Rejecting versus failing to reject

When p ≤ α, the decision rule rejects H₀. Otherwise, it fails to reject H₀. Crucially, a non-significant result does not prove equality; the sample may simply lack enough information to detect a meaningful difference.

Imagine a scale that cannot distinguish a one-gram change. An unchanged reading does not establish that the true weight stayed identical. Similarly, a low-powered test can overlook an effect that matters to a business or researcher.

At α = 0.05, a valid continuous test rejects a true simple null in 5% of repeated applications under its assumptions. This is a long-run property of the procedure. It does not mean that 5% of your significant findings must be false.

A practical six-step workflow

1. Define the target: specify the population, parameter, and meaningful effect.
2. Write the hypotheses: choose the direction and significance level in advance.
3. Check the design: identify independent, paired, clustered, or repeated observations.
4. Choose and compute the test: retain full precision until the final display.
5. Quantify the result: calculate the p-value, effect estimate, and suitable confidence interval.
6. Explain the evidence: connect the result to the question and state its limits.

Which hypothesis test should you use?

The outcome type and sampling structure determine the first decision. For example, a mean requires a different model from a proportion. Likewise, comparing two independent groups differs from measuring the same people twice.

A sample size above 30 is not a universal permission slip. Strong skew, heavy tails, outliers, dependence, and selection bias can still create problems. Conversely, small samples can support exact t inference when the normal model and independence assumptions make sense.

Before calculating, inspect a plot of the observations whenever possible. Moreover, ask how the sample entered the study. A precise formula cannot repair a sample that systematically excludes the population you want to describe.

Hypothesis testing for one population mean

The z-test and t-test formulas

Both statistics compare the observed gap with a standard error. However, the z-test uses a known population standard deviation, while the t-test estimates it from the sample. With more degrees of freedom, the t distribution approaches the standard normal distribution.

If σ is unknown, a large sample does not suddenly make it known. Therefore, the t-test remains the natural mean-test calculation in many such settings. The difference from a normal approximation often becomes small as the sample grows.

Worked example 1: bulb lifetime

A sample of 100 bulbs has a mean lifetime of 980 hours and a sample standard deviation of 80 hours. The benchmark equals 1,000 hours. To detect departures in either direction, use H₀: μ = 1,000 against H₁: μ ≠ 1,000, with α = 0.05.

Because 0.01406 is below 0.05, reject the null hypothesis. The sample supports a departure from 1,000 hours, and the estimated difference is −20 hours. However, the result concerns the population mean, not a guarantee about every individual bulb.

The supplied chapter uses a large-sample z approximation, which gives p ≈ 0.01242. Both approaches reach the same 5% decision here. Nevertheless, distinguishing them makes the role of the estimated standard deviation clear.

The interval excludes 1,000, consistent with the two-sided t-test. In practical terms, the point estimate represents a 2% shortfall from the benchmark. Whether that gap matters depends on warranty costs, customer expectations, and production requirements.

Worked example 2: detergent package weight

A company samples 25 packages and finds x̄ = 520 g and s = 75 g. It wants evidence that the population mean exceeds 500 g. Assuming independent observations and a suitable normal population model, test H₀: μ ≤ 500 against H₁: μ > 500.

At the 5% level, fail to reject H₀. Although the sample average exceeds 500 g, the evidence does not establish that the population mean does. The one-sided critical t value is approximately 1.7109, so 1.3333 does not reach the rejection region.

This distinction prevents a common reporting mistake: the result does not show that the mean equals 500 g. Instead, it shows insufficient evidence for the specific claim that the mean exceeds 500 g. A larger, well-designed sample could narrow the uncertainty.

Hypothesis testing for population proportions

One proportion: use the null value in the standard error

Here, x counts successes and p₀ represents the null proportion. The denominator uses p₀ because the test evaluates variability under the null model. By contrast, some confidence-interval formulas use an estimated proportion or a different construction entirely.

Check the expected numbers of successes and failures before using a normal approximation. A common conservative teaching rule asks for at least 10 of each under H₀. When counts are sparse, an exact binomial calculation can avoid this particular approximation.

Worked example 3: graduation rates

Historically, a program reports a four-year completion rate of 60%. In a cohort of 36 students, 15 graduate within four years. The question is whether the completion probability has declined, so test H₀: p ≥ 0.60 against H₁: p < 0.60.

Here, the left-tail normal p-value is approximately 0.01237. Therefore, reject the null at 5%, but fail to reject it at 1%. The sample estimates a decline of 18.33 percentage points, which differs from an 18.33% relative decline.

This exact method reaches the same decisions at 5% and 1%, although its p-value differs. This comparison illustrates why a rounded z statistic should not masquerade as an exact probability. Moreover, the analysis treats the historical 60% as a fixed benchmark; uncertainty in an estimated historical rate could require a two-sample comparison.

Two proportions: pooled testing under equality

Pooling fits the null claim that both groups share one success probability. However, a confidence interval for the difference usually estimates uncertainty without imposing that equality. Consequently, the test denominator and a simple interval denominator need not match.

Worked example 4: acceptable electronic components

Supplier A has 90 acceptable components out of 100, while Supplier B has 56 out of 80. Test whether A has the higher acceptance probability at α = 0.01. The alternative is p_A > p_B, and the null boundary sets the two probabilities equal.

Reject H₀ at 1%. The observed acceptance-rate gap equals 20 percentage points. Relative to B’s 70% rate, A’s 90% rate is approximately 28.6% higher; those two summaries describe the same data on different scales.

The supplied chapter rounds the pooled proportion to 0.8 and reports z ≈ 3.33. Retaining precision produces z ≈ 3.406 instead. The decision remains unchanged, but avoiding premature rounding protects borderline analyses.

A supplier decision still requires more information than acceptance rates alone. For example, prices, delivery reliability, defect severity, and sampling procedures could change the preferred contract. Also, components from one production batch may share conditions that weaken the independence assumption.

Comparing two population means

Welch’s test for independent groups

Welch’s t-test compares independent means without requiring equal population variances. In addition, its degrees of freedom account for uncertainty in both variance estimates. NIST documents the formula and distinguishes independent comparisons from paired designs.

Method reference: NIST: Two-Sample t-Test for Equal Means.

Worked example 5: wages in two cities

The teaching dataset gives City A a mean hourly wage of $6.00, s = $2.00, and n = 40. City B has a mean of $5.40, s = $1.80, and n = 54. These historical exercise values illustrate the method; they do not describe present-day wages.

At α = 0.05, fail to reject equal population means. The approximate 95% Welch interval for the difference runs from −$0.196 to $1.396 per hour. Thus, the evidence allows both a small negative difference and a substantially positive one.

The chapter’s large-sample normal approach gives p ≈ 0.13361 and the same decision. However, neither version establishes that the cities have identical wages. Occupation mix, experience, and other confounding factors would also matter before making a causal claim about location.

Why paired data need a different calculation

Suppose the same workers appear before and after a training program. Their measurements form pairs, so analyzing the two columns as independent groups discards that structure. Instead, compute each worker’s difference and test the mean difference.

Pairing can improve precision when repeated measurements correlate. Nevertheless, the pairs must remain independent across workers for the simple paired test. Repeated observations within teams or households may require a richer model.

Chi-square tests: counts, fit, and association

Goodness-of-fit versus independence

A goodness-of-fit test compares one categorical distribution with specified probabilities. In contrast, an independence test asks whether two categorical variables show association. Both use observed and expected counts, but they construct expectations differently.

For a fully specified categorical goodness-of-fit model, degrees of freedom equal k − 1. If the fitting procedure estimates parameters, an appropriate adjustment may reduce that number. For an r-by-c independence table, degrees of freedom equal (r − 1)(c − 1).

Small expected counts can make the chi-square approximation unreliable. Therefore, use a suitable exact or simulation method when the table is sparse, or combine categories only when the scientific meaning supports that choice. Do not merge categories simply to obtain a desired p-value.

Method reference: NIST: Chi-Square Goodness-of-Fit Test.

Worked example 6: the TV-sales error that changes the conclusion

A store’s reference sales mix is 30% small TVs, 40% medium TVs, and 30% large TVs. Among 100 purchases, it observes 20 small, 40 medium, and 40 large TVs. The null hypothesis states that the reference proportions still describe the population of purchases.

Since 0.035674 is below 0.05, reject the null distribution at the 5% level. Equivalently, 6.6667 exceeds the 5% critical value of approximately 5.9915. The purchases provide evidence of a changed size mix under the stated sampling assumptions.

In this sample, purchases shift 10 percentage points away from small TVs and 10 points toward large TVs. However, the test does not reveal why this happened. Promotions, availability, or customer composition could all affect the pattern, so an inventory manager should investigate those factors before treating the shift as permanent.

Worked example 7: buyer age and car type

A dealership records 170 purchases and classifies buyers by age and car type. The question asks whether the two categorical variables are independent. Use the following observed counts, which reproduce the teaching example’s inputs.

First, calculate expected counts from the margins without rounding them to whole numbers. For example, the expected count for younger buyers purchasing foreign cars equals 70 × 50 / 170. The other cells follow the same rule.

At α = 0.01, reject independence. Foreign cars account for 30/70 = 42.86% of purchases among younger buyers, compared with 20/100 = 20% among older buyers. Therefore, the observed difference equals 22.86 percentage points.

Using exact expected counts matters here: the chapter reports 9.44 after working with rounded expectations. The correct uncorrected Pearson calculation gives 10.3619, although both reject at 1%. A continuity-corrected test would be a separately specified method, not a justification for rounding expected counts.

This effect-size measure complements the significance test. Still, association does not establish that age causes a purchase preference. Income, product availability, and other variables could contribute to the observed relationship.

ANOVA: comparing three or more means

One-way analysis of variance compares population means across groups. Its null states that every group mean is equal. By contrast, the alternative requires at least one difference; it does not require every pair of means to differ.

Classical one-way ANOVA compares between-group variation with within-group variation. Under the standard model, observations are independent, errors follow normal distributions, and groups share a common variance. NIST presents the resulting sums of squares and F-test structure.

Method reference: NIST: The ANOVA Table and Tests of Hypotheses About Means.

Worked example 8: soap packaging and sales

A company lists five sales observations for each of three soap wrappings. For the initial calculation, follow the chapter’s independent-groups model. Afterward, examine whether the description of monthly observations supports that assumption.

The F statistic equals 65 / 9.1667 ≈ 7.0909, with 2 and 12 degrees of freedom. Its p-value is approximately 0.009270. Therefore, reject equal means at 5%, and also at 1%, under the assumed model.

In this sample, between-group differences account for about 54.17% of the total sum of squared deviations. This is a descriptive sample effect size, not proof that packaging causes 54.17% of population sales variation. Moreover, the overall F-test does not identify which pairs differ after multiplicity adjustment.

The hidden design question: do the rows represent shared months?

The source describes sales over five months. If each row corresponds to the same month across all wrappings, seasonal conditions may connect the three observations in that row. In that case, a blocked or repeated-measures model may better reflect the design than an independent-groups ANOVA.

For illustration, treating rows as fixed month blocks in an additive two-way model gives a month sum of squares of 40. Therefore, the residual sum of squares becomes 70, with 8 degrees of freedom. The wrapping F statistic then becomes 65 / 8.75 = 7.4286, with p ≈ 0.0150.

That illustrative blocked analysis requires its own assumptions, including an adequate additive model. With one observation per month-by-wrapping cell, the table cannot separately estimate an unrestricted interaction and residual error. Consequently, the correct analysis depends on how the company collected the observations, not just on which formula produces a smaller p-value.

What to do after a significant ANOVA

Start with the group estimates and a plot of the data. Then use a planned contrast or an appropriate multiple-comparison procedure if you need to identify specific differences. For example, Tukey comparisons often accompany a classical all-pairs analysis, while unequal-variance settings require methods compatible with that structure.

A significant F-test does not automatically justify choosing the highest sample mean. Costs, uncertainty, and operational constraints still matter. Also, an observational sales comparison cannot by itself isolate packaging from promotion, shelf position, or customer traffic.

Nonparametric hypothesis testing

Nonparametric procedures can help when a normal mean model does not fit the question or data. However, they still impose assumptions and often test different features of a distribution. Choosing them only because a sample is small can therefore miss the real issue.

Worked example 9: Wilcoxon signed-rank test for subsidiary profits

The source lists profits of 20, 35, 10, −5, −50, 5, 0, and 13 million monetary units. Here, the benchmark is 5 million. Subtract 5, remove the zero difference under the conventional Wilcoxon rule, and rank the absolute nonzero differences.

The tied absolute differences of 5 receive the average rank 1.5. Under a symmetric null distribution for the independent differences, enumerate all 2⁷ = 128 sign assignments to these seven ranks. Doubling the smaller inclusive tail probability gives an exact conditional two-sided p-value of 0.84375.

Therefore, fail to reject the null location at 5%. However, the result does not prove that the population median equals 5. The large negative profit and the small number of observations also make the symmetry assumption worth examining.

For a simple median interpretation, the signed-rank test requires an appropriate symmetric location model. Without symmetry, a sign test may answer the median question more directly, although it uses less magnitude information. SciPy’s documentation also warns that ties and zero differences affect the null distribution and the choice of computational method.

Method reference: SciPy: Wilcoxon signed-rank test.

Worked example 10: Kruskal–Wallis on the soap data

Kruskal–Wallis compares independent groups through pooled ranks. For the same soap data, rank all 15 observations together and assign average ranks to ties. The rank sums equal 35.5 for Wrapping 1, 23 for Wrapping 2, and 61.5 for Wrapping 3.

Five tied pairs produce a correction factor of 1 − 30/3360 = 0.9910714. Therefore, the corrected H statistic equals 7.7705. With 2 degrees of freedom, the usual chi-square approximation gives p ≈ 0.02054.

At 5%, this calculation rejects the common-distribution null under independent sampling. A median-only interpretation needs additional shape assumptions. Moreover, replacing observations with ranks does not solve the possible shared-month dependence identified in the ANOVA example.

SciPy notes that the chi-square approximation usually requires at least five observations per group, which this example only just meets. Consequently, an appropriate permutation analysis can be useful when an accurate small-sample calibration matters. A significant overall result still does not identify the differing groups.

Method reference: SciPy: Kruskal–Wallis H-test.

Worked example 11: Kolmogorov–Smirnov goodness-of-fit

Consider the seven observations 2, 4, 8, 9, 11, 13, and 18. The reference model is a normal distribution with a prespecified mean of 10 and standard deviation of 3. A one-sample K–S test compares the empirical cumulative distribution with that fixed continuous reference.

Both sides of each empirical step matter. Therefore, calculating only the difference between i/n and F₀(xᵢ) can miss the true maximum. The supplied example happens to reach the correct maximum despite presenting only one side of the calculation.

At 5%, fail to reject the specified distribution. However, seven observations provide limited ability to detect many departures. The correct conclusion is insufficient evidence against this particular model, not proof of normality.

NIST highlights a further limitation: standard one-sample K–S calibration assumes a fully specified distribution. If you estimate the normal mean and standard deviation from these same observations, the usual p-value no longer applies. Use an appropriate adjusted or simulation-based procedure instead.

The continuous reference assumption also deserves attention because actual unit sales are discrete counts. Here, the normal model serves as the textbook’s approximation. A real count-data analysis should evaluate whether that approximation fits the measurement process.

Method reference: NIST: Kolmogorov–Smirnov Goodness-of-Fit Test.

Power, sample size, and decisions beyond 0.05

Why a non-significant result can hide an important effect

Power depends on a specific alternative, not just the name of a test. For example, a bulb test might detect a 40-hour lifetime shortfall quite reliably while missing a 5-hour shortfall. Larger samples generally improve precision, but the effect size and variability also determine detection probability.

For a two-sided z-test with known σ = 80 hours and α = 0.05, consider a true 20-hour departure from the null mean. The exact normal-model powers are approximately 23.9% at n = 25, 70.5% at n = 100, and 99.9% at n = 400. These are calculated design illustrations, not observed industry performance.

A sample-size planning approximation

Suppose you want roughly 80% power to detect a 20-hour difference with σ = 80 and a two-sided 5% test. The approximation gives [(1.96 + 0.8416) × 80 / 20]² ≈ 125.58. Therefore, round up to 126 independent observations as a starting point for this known-variance design.

Real planning may need a t-based calculation, allowance for missing data, or adjustment for clustering. In addition, the assumed variability may come from uncertain pilot estimates. A sensitivity analysis across several plausible values helps avoid an overly optimistic sample-size target.

Multiple testing: a surprising numerical consequence

If you run 20 independent tests at α = 0.05 and all null hypotheses are true, the probability of at least one false rejection is not 5%. Instead, it equals one minus the probability of avoiding false rejections in every test.

The independence condition matters for that exact calculation. Nevertheless, the example shows why trying many outcomes and reporting only the smallest p-value can mislead readers. Prespecify the main analysis and use a suitable multiplicity strategy when a family of claims requires joint error control.

For example, Bonferroni control at an overall 5% level across 20 tests uses 0.05/20 = 0.0025 per test. Other methods address different goals, such as controlling the expected proportion of false discoveries among rejections. Choose the method to match the inferential question.

Confidence intervals and practical equivalence

A two-sided test at α and its matching 100(1 − α)% confidence interval give consistent decisions when they use the same model and construction. However, that relationship does not mean the interval assigns a probability to a fixed parameter. Frequentist coverage describes how the interval procedure behaves across repeated samples.

To argue that a difference is practically negligible, define an equivalence margin before analyzing the data. Then use an equivalence procedure that asks whether the evidence places the difference within that margin. Simply obtaining p > 0.05 from a difference test does not answer the equivalence question.

A calculation audit of the supplied examples

The table below separates exact arithmetic, modern method choices, and interpretation issues. Therefore, it helps readers understand whether a discrepancy changes the numerical result, the decision, or only the wording. Every recalculated value comes from the inputs displayed in this article.

How to report hypothesis testing clearly

A good report starts with the research question and effect estimate. Next, identify the test, statistic, degrees of freedom where relevant, p-value, and uncertainty interval. Finally, explain what the study design allows you to conclude.

Avoid reporting “p = 0.000” when software rounds a very small value. Instead, provide enough digits or an appropriate upper bound. Likewise, keep the exact inferential method visible when corrections, exact enumeration, or asymptotic approximations affect the result.

Common mistakes and their practical fixes

Frequently asked questions about hypothesis testing

What is the simplest explanation of hypothesis testing?

It asks whether the sample looks unusually inconsistent with a reference claim, given a statistical model. However, an unusual result can reflect a false claim, a poor model, or a problem with the data. Interpretation therefore requires more than the final p-value.

When should I use a z-test instead of a t-test?

For a population mean, the classic z-test uses a known population standard deviation and an appropriate normal sampling distribution. When you estimate that standard deviation from the sample, a t-test commonly fits the problem. Large-sample approximations may look similar, but the distinction remains conceptually useful.

Does p = 0.06 mean there is no effect?

No. It means the result does not cross a 0.05 threshold for that particular test. Examine the estimated effect, interval, sample size, and study design before deciding what the evidence supports.

Can I compare p-values from different tests directly?

Only with care. Different tests may examine different null hypotheses, use different information, or rely on different assumptions. Therefore, the smallest p-value is not automatically the most appropriate answer.

Why does the chi-square test use squared differences?

Squaring prevents positive and negative deviations from canceling each other. Dividing by the expected count then expresses each discrepancy relative to the category’s expected scale. Consequently, a difference of 10 counts matters differently when the expectation is 20 versus 2,000.

Does ANOVA tell me which group performs best?

The overall test only evaluates whether all population means are equal. After rejection, examine suitable comparisons and their uncertainty. Practical decisions also require costs, constraints, and a design that supports the intended interpretation.

Are nonparametric tests always safer for small samples?

No universal rule makes them safer. They may offer useful alternatives, but their assumptions and target hypotheses still matter. For example, signed ranks require symmetry for a straightforward location interpretation, and rank methods cannot remove dependence between observations.

What should I do when a textbook answer disagrees with software?

First, check the data entry, tail direction, degrees of freedom, rounding, and any continuity correction. Then compare the formulas and the exact versus approximate methods. As the TV example shows, the printed answer itself can contain an arithmetic error.

Turn statistical evidence into a defensible conclusion

Hypothesis testing works best when you connect the numerical result to a clearly defined question. Begin with the study design, choose the appropriate model, and preserve precision during calculation. Then explain the estimated effect and its uncertainty before translating the evidence into a decision.

The examples above show why that discipline matters. A simple arithmetic correction reverses the TV-sales conclusion, while a design question changes how monthly soap sales should enter an analysis. Ultimately, reliable inference comes from combining correct calculations with careful reasoning about what the data represent.

Sources and calculation notes

The supplied chapter pages 87–95 provide the teaching inputs. This article paraphrases the explanations, independently recalculates the results, and adds methodological checks; it does not reproduce the chapter verbatim. The additional Kruskal–Wallis analysis, blocked-sales illustration, power curves, and multiplicity calculations are original extensions using explicitly stated assumptions.

• American Statistical Association: P-value statement
  https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
• NIST: Two-sample t-test
  https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm
• NIST: Chi-square goodness-of-fit
  https://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm
• NIST: ANOVA table
  https://www.itl.nist.gov/div898/handbook/prc/section4/prc433.htm
• NIST: Kolmogorov–Smirnov test
  https://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm
• SciPy: Wilcoxon
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html
• SciPy: Kruskal–Wallis
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kruskal.html

Numerical checks use standard normal, Student t, binomial, chi-square, F, and exact continuous K–S calculations. The signed-rank p-value uses enumeration of all sign assignments to the observed nonzero ranks. Display rounding may create tiny differences when readers recompute from the printed decimals.