Simple Regression Analysis: Formulas and Examples

This guide develops the method through two complete datasets, with calculations you can reproduce. Along the way, you will learn ordinary least squares, coefficient tests, confidence intervals, prediction intervals, correlation, and the Gauss–Markov theorem. In addition, you will see why rounding, notation, and annual trends can quietly change the conclusions you draw.

What is simple regression analysis?

Simple regression relates one response variable, Y, to one explanatory variable, X. The word “simple” refers to the number of predictors, not the importance of the question. In contrast, multiple regression includes two or more predictors.

For a straight-line model, the population relationship takes this form:

Here, β₀ is the intercept, β₁ is the slope, and u_i is the unobserved error. Meanwhile, the index i identifies a person, farm, firm, year, or other observation. Under the zero conditional mean assumption, the expected outcome satisfies E(Y | X) = β₀ + β₁X.

The model, the fitted line, and the residual are different

A population model contains unknown parameters. By comparison, a fitted line uses estimates from the available sample:

The hat on Ŷ marks a fitted or predicted value. Consequently, the residual e is the observed gap between an outcome and the fitted line. The unobserved error u refers to the gap from the population relationship, so you should not treat errors and residuals as identical objects.

How to interpret the slope correctly

In practical terms, the slope measures the change in the fitted outcome for a one-unit increase in X. For example, a slope of 1.66 in the fertilizer model means that one additional pound of fertilizer per acre corresponds to about 1.66 additional predicted bushels per acre. However, the calculation alone does not prove that adding fertilizer causes that increase.

Likewise, the intercept gives the fitted outcome at X = 0. Its practical meaning depends on whether zero belongs to a plausible range. In the corn data, fertilizer ranges from 6 to 32, so interpreting the intercept as observed output without fertilizer would go beyond the sample.

“Linear” does not always mean a straight line in the raw data

A regression can remain linear in its coefficients while including a transformed predictor. For instance, Y = β₀ + β₁ln(X) + u is linear in β even though its graph curves against raw X. Similarly, a polynomial model can contain X² without becoming nonlinear in its parameters. NIST explains this distinction in its guide to linear least squares.

How ordinary least squares finds the regression line

Ordinary least squares, or OLS, selects the intercept and slope that minimize the sum of squared vertical residuals. Squaring prevents positive and negative residuals from canceling. As a result, large misses receive more weight than small misses.

These are vertical distances in Y units. They are not perpendicular geometric distances to the line, and OLS does not minimize absolute deviations. For a concise methodological reference, see NIST’s least-squares definition.

The essential calculation formulas

First, compute the sample averages X̄ and Ȳ. Next, measure each observation’s deviation from its average. Finally, combine those deviations into three useful quantities:

The denominator S_xx must exceed zero. Otherwise, every X value is identical and the sample cannot identify a separate slope and intercept. An equivalent raw-sum expression is useful when a question supplies totals:

Why the fitted line passes through the sample averages

Differentiating SSE with respect to the intercept gives Σe_i = 0. Next, differentiating with respect to the slope gives ΣX_ie_i = 0. Therefore, the fitted values have the same sample mean as the observed outcomes.

Dividing the first equation by n produces Ȳ = b₀ + b₁X̄. Thus, an OLS line with an intercept passes exactly through (X̄, Ȳ), apart from numerical rounding. This property gives you a quick way to detect a transcription or calculation error.

Simple regression analysis example: corn and fertilizer

The first dataset contains ten annual observations from 1971 to 1980. X measures fertilizer in pounds per acre, while Y measures corn output in bushels per acre. Although both variables increase over much of the period, the dataset does not isolate fertilizer from weather, seed quality, technology, or other changes.

Step 1: Calculate the averages and slope

Adding fertilizer observations gives 180, while adding corn observations gives 570. Therefore, X̄ = 18 and Ȳ = 57. The centered products sum to 956, and the squared fertilizer deviations sum to 576.

Step 2: Calculate the intercept

Use the unrounded slope when calculating the intercept. Otherwise, early rounding slightly changes every fitted value that follows.

At X = 18, the prediction equals 57 exactly. Meanwhile, increasing fertilizer by five pounds per acre raises the fitted yield by 5 × 1.659722222 = 8.2986 bushels per acre. That is a prediction from the fitted association, not an experimentally established treatment effect.

Step 3: Calculate fitted values and residuals

For each row, substitute its X value into the fitted equation. Then subtract the fitted result from the observed Y value. A positive residual indicates that actual yield exceeds the prediction; a negative residual indicates the reverse.

For 1971, the fitted yield is 37.0833 and the observed yield is 40. Consequently, the residual is 2.9167 bushels per acre. Using full precision throughout, the residuals sum to zero and their squared values sum to 47.3055556.

A rounding detail that many worked examples overlook

The supplied pages display the rounded equation Ŷ = 27.12 + 1.66X. That version produces SSE = 47.3056, almost identical to the full-precision answer. However, rounding the slope’s standard error from about 0.101321 to 0.10 changes the reported t statistic from about 16.3808 to 16.6.

Standard errors, t tests, and coefficient confidence intervals

A fitted slope describes the sample. To discuss uncertainty about the population slope, you also need a standard error. The conventional formulas below assume the classical model with constant conditional error variance and uncorrelated errors; exact small-sample t inference additionally uses normal errors.

Residual variance and residual standard error

A straight-line model estimates two parameters, so its residual degrees of freedom equal n − 2. Accordingly, the corn example has 8 residual degrees of freedom.

The residual standard error has the same units as Y: bushels per acre. By contrast, s² has squared units. For this sample, s describes the residual scale under the model; it is not the mean absolute error or a guaranteed bound on future prediction errors.

Standard errors of the slope and intercept

Substituting the corn values gives SE(b₁) = 0.1013211 and SE(b₀) ≈ 1.9793. Notice that a wider spread in X increases S_xx. Consequently, for a fixed error variance and sample size, that wider spread improves slope precision.

Test whether the slope differs from zero

Suppose the null hypothesis is H₀: β₁ = 0 and the alternative is H₁: β₁ ≠ 0. Under the classical assumptions, calculate:

For eight degrees of freedom, the two-sided 5% critical value is approximately 2.306. Because 16.3808 exceeds 2.306, reject the zero-slope null under this model. The corresponding two-sided p-value is approximately 1.94 × 10⁻⁷.

Nevertheless, this small p-value does not establish causation or validate the error assumptions. It measures how unusual this test statistic would be under the zero-slope null and the assumed sampling model. A large t statistic cannot repair omitted variables or a flawed study design.

Construct a 95% confidence interval for the slope

Thus, the interval estimates a population slope between about 1.43 and 1.89 bushels per acre per additional pound of fertilizer per acre, subject to the model. In repeated samples under those assumptions, the interval procedure would cover the true slope 95% of the time. It does not mean that 95% of observed corn yields lie inside these two numbers.

R², correlation, and the ANOVA table

A significance test addresses uncertainty about a coefficient. R² instead summarizes how closely the fitted values track the observed outcome variation within the sample. Therefore, the two statistics answer different questions.

Separate total, explained, and residual variation

This decomposition holds for OLS with an intercept. Here, SST = 1634, SSE = 47.3055556, and SSR = 1586.6944444. Consequently, the model accounts for most of the sample’s variation around mean corn yield.

In this sample, the fitted line accounts for approximately 97.10% of the squared variation around average yield. However, that does not mean “97.10% prediction accuracy,” “97.10% causation,” or “97.10% of the observations are correct.” The remaining 2.90% is residual sample variation, which can reflect omitted factors, noise, measurement issues, and model mismatch.

The RSS and ESS notation trap

The supplied textbook uses RSS for regression sum of squares and ESS for error sum of squares. Other references use RSS for residual sum of squares and ESS for explained sum of squares. As a result, copying an acronym without checking its definition can reverse your R² calculation.

Correlation preserves the direction that R² loses

The positive slope gives a positive correlation. In contrast, a negative slope requires the negative square root of R². Therefore, the shortcut r = √R² only works when the relationship has a nonnegative direction.

For simple OLS with an intercept and nonzero variation in both variables, R² equals r². Pearson correlation has no units, while the regression slope does. Consequently, a slope of 1.66 and a correlation of 0.9854 describe different aspects of the same fitted relationship.

ANOVA connects the slope test to the F test

The F statistic divides the regression mean square by the residual mean square. In simple regression, the conventional F test of a zero slope equals the square of the conventional two-sided slope t statistic: F = t². Thus, 16.3808² ≈ 268.3312, giving the same test conclusion.

Adjusted R² introduces a degrees-of-freedom adjustment. For this two-parameter model, it equals 1 − [SSE/(n − 2)]/[SST/(n − 1)] ≈ 0.9674. Although useful for comparing some models, adjusted R² still does not measure causal validity or guarantee accurate forecasts.

Confidence intervals versus prediction intervals

A point prediction gives one number. However, predicting an average outcome differs from predicting a single new observation. A prediction interval must include the new observation’s unexplained variation, so it is wider than the corresponding confidence interval for the mean.

Penn State’s regression estimation and prediction lesson treats these two targets separately. For the formulas below, the new observation must follow the same model, and its error must be independent of the estimation sample.

Mean response at a chosen predictor value

A single future observation

The additional 1 inside the square root represents the new observation’s error variance. Consequently, even a precisely estimated mean does not eliminate individual uncertainty. At X = 20, the fitted yield is about 60.32, but the two interval targets differ substantially.

Why predictions become less precise away from the center

The term (X₀ − X̄)² increases as you move away from the sample mean. Therefore, the confidence band is narrowest at X = 18 and widens toward either end. Prediction intervals widen for the same reason, although they also contain the extra individual-error component.

At X = 40, the formulas still produce numbers, but the sample ends at 32. This is extrapolation. Moreover, the mathematical interval does not include unknown model failure beyond the observed range, such as diminishing agricultural returns or changes in soil conditions.

Second worked example: consumption and disposable income

The second dataset contains twelve annual observations from 1988 through 1999. Both consumption and disposable income use billions of US dollars in the supplied example. However, the photographed material does not name the economy or clarify inflation adjustment, so the series should remain a teaching example.

Calculate the consumption equation

The averages are X̄ = 145 and Ȳ = 127. Meanwhile, S_xx = 4812 and S_xy = 4144. Applying the same OLS formulas gives:

Within this fitted relationship, an additional $1 billion of disposable income corresponds to approximately $0.861 billion of predicted consumption. In a structural consumption model, economists may interpret the slope as a marginal propensity to consume. Nevertheless, a bivariate regression of annual observations does not automatically identify that causal behavioral parameter.

Predict consumption at income of $150 billion

The fitted consumption value is approximately $131.31 billion. Under the classical assumptions, the 95% mean confidence interval runs from about $129.06 billion to $133.56 billion. By comparison, the 95% prediction interval for one new observation runs from about $123.41 billion to $139.20 billion.

The hidden problem: both examples contain time

The corn rows describe successive years, and the consumption rows also form an annual sequence. Therefore, neither dataset automatically satisfies an independent-observations assumption. Shared trends or persistent errors can produce a strong-looking relationship without a reliable causal interpretation.

Before using such a model for economic forecasting, examine the time plots and residual autocorrelation. In addition, verify whether the monetary series use nominal or real values and whether a time-series treatment is necessary. Hyndman and Athanasopoulos discuss residual dependence and spurious regression in Forecasting: Principles and Practice.

Which assumptions does simple regression actually need?

You can calculate an OLS line whenever the data provide enough variation in X. However, its statistical properties depend on additional conditions. It helps to separate assumptions for unbiasedness, efficiency, consistency, and exact inference rather than treating them as one undifferentiated list.

Zero unconditional mean is not enough

The statement E(u) = 0 does not guarantee that the error averages zero at each value of X. In contrast, E(u | X) = 0 rules out systematic error patterns linked to the predictor. This distinction matters when omitted variables influence both X and Y.

For example, better-managed farms might use more fertilizer and better seeds. If seed quality remains in the error term, the fertilizer coefficient may absorb part of that association. Consequently, interpreting the slope as fertilizer’s isolated effect would require stronger evidence.

Normality is not necessary to draw the OLS line

OLS computation requires no normality assumption. Furthermore, the Gauss–Markov efficiency result does not require normally distributed errors. Normality instead supports exact small-sample inference when combined with the other classical conditions.

Also, the normality condition concerns errors conditional on X, not the marginal distribution of X. A skewed predictor therefore does not automatically invalidate linear regression. With only ten or twelve observations, however, you should avoid relying casually on large-sample approximations.

Fixed predictors are a framework, not a universal real-world requirement

Some textbook derivations hold the observed X values fixed across repeated samples. In observational applications, analysts can instead use an appropriate random-X sampling framework and condition on the predictors. The central issue is whether the error assumptions hold, not whether income or fertilizer is literally nonrandom in everyday life.

What BLUE actually promises

Under the Gauss–Markov conditions, OLS is the best linear unbiased estimator. Here, “best” means smallest variance within the class of linear unbiased estimators, while “linear” describes the estimator’s dependence on observed Y values. It does not mean OLS dominates every biased, nonlinear, or prediction-focused method.

A biased estimator can sometimes reduce variance enough to achieve a smaller mean squared error. Therefore, unbiasedness alone does not settle every modeling decision. Likewise, consistency means convergence in probability to the true parameter as sample size increases under appropriate conditions; one small dataset cannot demonstrate that property.

Regression diagnostics: inspect what the line leaves behind

A polished equation can hide weak assumptions. Start by examining a scatterplot, then inspect residual patterns and influential observations. Penn State’s model-assumptions lesson emphasizes residual analysis as a way to assess the model.

Patterns that deserve attention

For the corn sample, residuals range from about −3.64 to +3.72 bushels per acre. The plot alone offers limited evidence because it contains only ten points. Consequently, neither an apparently random pattern nor a single formal test can certify every model assumption.

NIST’s assumption-checking guidance describes how residual patterns can reveal changes in spread or other departures. However, residual checks cannot establish that an omitted confounder is absent.

High leverage and a large residual are different

Leverage measures how far an observation’s X value lies from the sample center. In simple regression with an intercept, its formula is h_i = 1/n + (X_i − X̄)²/S_xx. For the corn observation at X = 32, leverage equals approximately 0.4403, compared with average leverage 2/n = 0.20.

Nevertheless, its residual is only about −0.2361. A point can therefore have high leverage without a large residual. To assess actual influence, examine how coefficient estimates change when you omit it, while keeping substantive reasons for inclusion in view.

NIST’s regression diagnostics reference distinguishes diagnostic tools beyond basic residual plots. Never delete an observation merely because removing it improves R².

Useful insights that basic regression summaries often miss

Centering changes the intercept but not the fitted values

Define Z = X − 18 for the corn dataset. Then the fitted equation becomes Ŷ = 57 + 1.6597222Z. Consequently, the intercept now represents predicted yield at the sample’s average fertilizer level, rather than at an unobserved zero-fertilizer setting.

The slope, residuals, SSE, and R² remain unchanged. However, the centered intercept’s standard error becomes s/√n ≈ 0.76897. This smaller uncertainty reflects a different, better-supported prediction target.

Changing units changes the slope but not the relationship’s strength

Suppose you express fertilizer in tens of pounds per acre, using W = X/10. The fitted equation becomes Ŷ = 27.125 + 16.5972222W. Thus, the numerical slope increases tenfold, but the fitted yields and R² stay identical.

Its standard error also increases tenfold. As a result, the slope’s t statistic remains unchanged. This makes units a crucial part of coefficient interpretation, even when they do not affect significance.

Reversing X and Y does not simply invert the slope

Regressing Y on X minimizes vertical errors in Y. Conversely, regressing X on Y minimizes errors in X. For the corn data, the reverse-regression slope is S_xy/S_yy = 956/1634 ≈ 0.5851, while 1/1.6597222 ≈ 0.6025.

These values differ because the relationship is not perfect. In fact, the product of the two regression slopes equals r² for simple regressions with intercepts. Therefore, rearranging a fitted line is not equivalent to refitting the reverse prediction problem.

A regression’s error scale is not its training RMSE

The residual standard error divides SSE by n − 2 before taking a square root. Training RMSE instead divides SSE by n. Consequently, the corn model has residual standard error 2.4317 but training RMSE about 2.1750.

Neither number measures performance on unseen observations. To assess forecasting quality, use an appropriate validation design and report out-of-sample errors. For time-ordered data, preserve the temporal order when constructing training and evaluation periods.

Common mistakes in simple regression analysis

Practice problems with explained answers

1. Predict corn yield at X = 24

Substitute 24 into the unrounded fitted equation. The result is Ŷ = 27.125 + (956/576) × 24 = 66.9583. Because the observed yield equals 68, the residual is 1.0417 bushels per acre.

2. Estimate the fitted change when fertilizer rises from 14 to 22

The change in X equals 8. Therefore, the fitted change in Y equals 8 × 1.6597222 = 13.2778 bushels per acre. The intercept cancels when you compare two predictions from the same line.

3. Find the correlation when R² = 0.81 and the slope is negative

Take the square root of 0.81, which is 0.90. Then apply the slope’s negative sign. The correlation is −0.90, not +0.90.

4. Explain why n − 2 appears in the residual variance

Here, the model estimates an intercept and a slope from the sample. Consequently, residuals satisfy two fitted constraints and have n − 2 degrees of freedom. With ten observations, divide SSE by 8 to obtain the conventional unbiased estimate of error variance.

5. Decide whether a $1 billion income increase proves $0.861 billion of extra consumption

No causal conclusion follows from the fitted slope alone. The model associates that income difference with about $0.861 billion of predicted consumption. However, omitted variables, common trends, and the sampling structure can change a causal interpretation.

6. Choose an interval for one new corn observation

Use the prediction interval because the target is a single new outcome. At X = 20, that interval is approximately 54.42 to 66.22 under the classical model. By contrast, the narrower interval 58.49 to 62.15 concerns the mean response.

7. Check whether the rounded equation passes through the averages

Substitute X = 18 into 27.12 + 1.66X. The result equals 57, so this particular rounded equation still passes through the sample means. Nevertheless, its slope differs slightly from the exact OLS value and should not replace full precision in inferential calculations.

8. Interpret a residual of −3.64

The observed outcome is approximately 3.64 units below its fitted value. Therefore, the model overpredicts that observation. A negative residual does not mean the outcome itself is negative.

How to report a simple regression clearly

For clear communication, a useful report connects the equation to its units, data source, uncertainty, and limitations. For example, the corn results support the following description:

Using ten annual textbook observations, the fitted equation is predicted corn yield = 27.125 + 1.659722 × fertilizer. Fertilizer uses pounds per acre and yield uses bushels per acre. The model has R² = 0.9710 and residual standard error 2.4317. Under classical independent, constant-variance normal-error assumptions, the 95% slope confidence interval is 1.4261 to 1.8934. These observations show a strong fitted association, but the data do not independently establish a causal fertilizer effect.

Before submitting your own analysis, confirm the observation unit and sample size. Next, identify the outcome, predictor, and their measurement units. Finally, report both statistical uncertainty and the study-design limitations that numbers alone cannot resolve.

Frequently asked questions

What is the difference between correlation and regression?

Correlation summarizes the direction and strength of a linear association without units. Regression predicts one variable from another using a slope and intercept. Therefore, swapping the variables leaves correlation unchanged but generally changes the fitted regression.

Is a high R² always desirable?

A high R² indicates a close fit to the observed sample relative to its mean. However, it can coexist with omitted variables, extrapolation risk, or time-series problems. Always interpret it alongside diagnostics and the question the model should answer.

Can R² be negative?

For ordinary in-sample OLS with an intercept and a nonconstant outcome, R² lies between zero and one. However, an out-of-sample score defined as one minus an error ratio can become negative when predictions perform worse than its reference mean benchmark. Models without an intercept also require care because software may use a different definition.

Do X and Y both need normal distributions?

No. The classical normality assumption concerns the conditional errors, not both raw variables. Moreover, OLS estimation and the Gauss–Markov result do not require normality.

How many observations do I need?

There is no universal sample-size rule that guarantees a useful regression. More than two observations and variation in X allow a positive residual degrees-of-freedom count, but adequate precision depends on noise, predictor variation, dependence, and the intended use. Consequently, these small teaching datasets should not serve as evidence that ten observations are always enough.

Conclusion: turn a fitted line into a defensible analysis

Simple regression analysis starts with a clear question and a scatterplot. OLS then provides an equation, while residuals reveal what the line misses. Confidence intervals, prediction intervals, and diagnostic checks help you decide how much weight to place on the result.

The corn example produces a slope of 1.659722 and R² of 0.971049. Meanwhile, the consumption example produces a slope of 0.861180 and R² of 0.968711. Both fits look strong, yet their annual structure, small samples, and incomplete source metadata limit broader conclusions.

Ultimately, a strong analysis keeps full calculation precision, defines every sum-of-squares symbol, distinguishes average uncertainty from individual uncertainty, and separates association from causation. Those habits make the method useful long after you finish a textbook exercise.

References and further reading

• Supplied source photographs: Chapter 6, “Simple Regression Analysis,” pages 128–136, Tables 6.1–6.4. Full bibliographic metadata not visible. Source for the two instructional datasets, not for a claim about current real-world conditions.
• NIST/SEMATECH. Linear Least Squares Regression. Model definition, objective, and limitations.
• Penn State. STAT 501: Simple Linear Regression. Additional teaching material on the basic model.
• Penn State. STAT 501: SLR Estimation & Prediction. Mean estimation and prediction intervals.
• Penn State. STAT 501: SLR Model Assumptions. Residual analysis and model checks.
• NIST/SEMATECH. Check of Assumptions. Diagnostic residual patterns.
• NIST. Regression Diagnostics. Diagnostic measures beyond the basic fitted line.
• Hyndman, R. J., and Athanasopoulos, G. Forecasting: Principles and Practice, Chapter 7. Time-series regression, model evaluation, and spurious relationships.

Calculation conventions: all regressions include an intercept; reported intervals are two-sided 95% classical-model intervals; displayed values may differ slightly from calculations performed on rounded inputs. Figures and derived statistics are original recalculations from the supplied data.