Linear Regression in Dose–Response and Calibration Work
Fitting a straight line to points is taught in school statistics as y = mx + c, and it can feel like busywork. It is not. Least-squares regression is the arithmetic behind almost every quantitative measurement made in an analytical laboratory: you cannot read concentration off an instrument directly, you read a signal, and a fitted line is what converts one into the other. This article works a full calibration curve by hand and then states clearly where the line stops being trustworthy.
The formulas
m = [ n Σxy − (Σx)(Σy) ] / [ n Σx² − (Σx)² ]
c = ȳ − m x̄
R² = (SSxy)² / (SSxx × SSyy)
where the "sum of squares" shortcuts are SSxx = Σx² − (Σx)²/n, SSyy = Σy² − (Σy)²/n and SSxy = Σxy − (Σx)(Σy)/n. With those, m is simply SSxy / SSxx.
| Term | Meaning in a calibration |
|---|---|
| x | The known quantity you set — concentration of a standard, or dose |
| y | The measured response — absorbance, peak area, counts, effect size |
| m (slope) | Sensitivity: how much signal one unit of x produces |
| c (intercept) | Signal at zero concentration — blank, background or baseline offset |
| R² | Fraction of the variation in y explained by the straight line, 0 to 1 |
Worked example — a five-point calibration curve
Five standards of a coloured complex are measured in a spectrophotometer at fixed wavelength and 1 cm path length.
| Concentration x (mg/L) | 2.00 | 4.00 | 6.00 | 8.00 | 10.00 |
|---|---|---|---|---|---|
| Absorbance y | 0.104 | 0.201 | 0.312 | 0.405 | 0.508 |
Step 1 — the five sums. n = 5.
Σx = 2 + 4 + 6 + 8 + 10 = 30
Σy = 0.104 + 0.201 + 0.312 + 0.405 + 0.508 = 1.530
Σx² = 4 + 16 + 36 + 64 + 100 = 220
Σxy = 0.208 + 0.804 + 1.872 + 3.240 + 5.080 = 11.204
Σy² = 0.010816 + 0.040401 + 0.097344 + 0.164025 + 0.258064 = 0.570650
Step 2 — the sums of squares.
SSxx = 220 − (30)²/5 = 220 − 180 = 40
SSyy = 0.570650 − (1.530)²/5 = 0.570650 − 0.468180 = 0.102470
SSxy = 11.204 − (30 × 1.530)/5 = 11.204 − 9.180 = 2.024
Step 3 — slope and intercept.
m = 2.024 ÷ 40 = 0.0506 absorbance units per mg/L
x̄ = 30/5 = 6.00 and ȳ = 1.530/5 = 0.306
c = 0.306 − (0.0506 × 6.00) = 0.306 − 0.3036 = 0.0024
The calibration line is y = 0.0506x + 0.0024.
Step 4 — R².
R² = (2.024)² ÷ (40 × 0.102470) = 4.096576 ÷ 4.098800 = 0.9995
Step 5 — use it. An unknown sample gives absorbance 0.256:
x = (0.256 − 0.0024) ÷ 0.0506 = 0.2536 ÷ 0.0506 = 5.01 mg/L
Two useful checks on your working. First, the line must pass through (x̄, ȳ) — substitute x = 6.00 and you get 0.3060, which is ȳ to within rounding. Second, the residuals (observed minus predicted) must sum to zero: +0.0004, −0.0038, +0.0060, −0.0022, −0.0004 add to exactly 0.0000. If yours do not, the arithmetic is wrong.
Where this is actually used
This calculation is not a teaching exercise dressed up as an application — it is the application. Regulated analytical laboratories run a calibration curve alongside every batch of samples, and method validation guidelines across the pharmaceutical, food, environmental and clinical sectors require linearity to be demonstrated and documented before results are reported. A few of the routine uses:
- Instrumental analysis. UV-visible spectrophotometry, atomic absorption, HPLC, GC and ICP all quantify by fitting standards and back-calculating unknowns exactly as above.
- Detection limits. The slope is the denominator in the conventional estimates of limit of detection and limit of quantitation, computed from the standard deviation of the blank divided by the sensitivity. A shallow slope means a poor detection limit, whatever the R² says.
- Stability and shelf-life work. Assay against time is fitted to find the rate of loss and to project when the content will fall below specification. This is the basis of the stability testing programmes run across the industry.
- Dose–response and toxicology. Response is regressed against dose, or more usually against log dose, to estimate potency measures such as the concentration giving half the maximum effect.
- Kinetics and thermodynamics. Every "plot ln k against 1/T" or "plot ln P against 1/T" exercise is a regression; the activation energy or enthalpy is read out of the slope.
The honest limits
A high R² does not mean the result is correct. R² measures how tightly the points sit on the fitted line, and nothing else. A contaminated set of standards, a mis-set wavelength or a wrong stock concentration will still give R² = 0.999 — the line will be beautifully straight and systematically wrong. Precision and accuracy are separate properties, and regression only reports on the first. Recovery checks against an independent reference material are what test the second.
Correlation is not causation. In a designed calibration the analyst sets x deliberately, so the causal direction is genuinely known. In an observational dose–response study — exposure measured rather than assigned — a strong fitted slope establishes only that the two vary together. A third factor associated with both can produce the same line, which is why observational evidence is treated as weaker than a controlled experiment.
Extrapolation beyond the calibrated range is not valid. The line above was established from 2 to 10 mg/L. Feeding in 40 mg/L returns a predicted absorbance of 2.03, but at that level a real spectrophotometer departs from linearity: stray light, detector response and chemical effects such as association at higher concentration all bend the curve. The equation happily produces a number; the instrument will not reproduce it. If a sample reads above the top standard, dilute it and re-measure — never extend the line.
And most dose–response relationships are not straight at all. Biological response typically follows a sigmoidal curve: little effect at low dose, a steep middle region, then a plateau at saturation. Fitting a straight line across the whole range mis-describes both ends. Linear regression is legitimate there only within the roughly linear middle section, or after a transformation that genuinely straightens the data.
Common mistakes
- Forcing the line through the origin. Tempting, since zero concentration "should" give zero signal. Do it only if you have shown the intercept is not significantly different from zero; otherwise you build the blank error into every result.
- Swapping x and y. Ordinary least squares assumes all error is in y. Regressing concentration on absorbance gives a different slope and a wrong answer.
- Rounding the sums. Σy² carries six decimals here for a reason — SSyy is a small difference between two larger numbers, so early rounding destroys R².
- Reporting more significant figures than the data support. Standards known to three figures cannot give a concentration to five.
- Never plotting the residuals. R² can be high while the residuals show a clear curve — proof the model is wrong even though the fit "looks" excellent.
- Ignoring unequal scatter. If the spread of y grows with x, unweighted regression lets the top standards dominate and the low end suffers. Weighted regression exists for exactly this case, and it matters most near the detection limit.
Fit your own data. Paste in the x and y pairs and the Linear Regression tool returns slope, intercept and R² instantly — the same quantities computed by hand above.
Open the Linear Regression Calculator →Analytical chemistry and error analysis carry real weight in IIT-JAM, GATE, CSIR-NET and CUET-PG. ABC Chemistry runs coaching-centre and online chemical-sciences batches for students across India — abcchemistry.in.