Module 13 · Improve

Design of experiments

Everything before this module analysed data the process happened to produce. A designed experiment makes the process produce the data you need: a small, deliberate set of runs in which several factors are changed together, so that each factor's effect, and the way the factors interact, can be read off with arithmetic. This module builds the two-level factorial from the ground up, works a 2³ with replicates by hand and by script, shows what a half fraction costs and buys, and takes a fair look at the Taguchi controversy.

Learning objectives

• Explain with a worked case why changing one factor at a time can miss the best setting entirely, and what an interaction is.
• Lay out a 2^k factorial in standard order, run it in random order, and compute every main effect and interaction from the contrast columns.
• Analyse a replicated factorial with an ANOVA table, a standard error for the effects and a prediction equation, and an unreplicated one with Lenth's method and a Pareto of effects.
• Construct a half-fraction design from a generator, write its defining relation and alias structure, and state its resolution.
• Describe when a two-level design is not enough and how a response surface design (centre points, central composite) follows from it.
• Summarise the Taguchi debate fairly: what is worth keeping (robustness, noise factors), what the critics showed, and how to analyse a Taguchi array with standard methods.

Before you start

Three questions on Module 11. They do not count toward completion.

Why this matters

Designed experiments are the oldest part of this course. Fisher set out randomisation, replication and blocking, and the factorial principle of varying several factors at once, in The Design of Experiments in 1935, for agricultural field trials.^[1] Yates gave the arithmetic for two-level factorials in 1937,^[2] Plackett and Burman the screening designs in 1946,^[5] and Box and Wilson the response surface method in 1951, for a chemical process.^[6] None of it is new, and none of it is used as much as it should be. The engineering instinct, when a process misbehaves, is to change one thing, look, change it back, change the next thing. That instinct is wrong in a specific, demonstrable way, and worked example 1 demonstrates it.

The real example of this module is a controversy again, because it is documented and it shaped how industry does experiments. In the 1980s Genichi Taguchi's methods for "robust design", experiments laid out on orthogonal arrays and analysed with signal-to-noise ratios, spread rapidly through American manufacturing.^[11][12] Statisticians examined them. Box's 1988 Technometrics paper argued that the signal-to-noise ratios were poorly chosen performance criteria and that a transformation of the response achieved the same ends more simply,^[9] and the 1992 panel discussion edited by Nair brought Taguchi's proponents and critics into one room.^[10] The section near the end of this module sets out what each side established. The short version, which the NIST handbook also takes, is that Taguchi's arrays are fractional factorials under another name, that his emphasis on noise factors and robustness was a genuine contribution, and that the analysis is best done with the standard methods this module teaches.^[8]

The following is an illustrative failure with invented details, not a real program. A braze line has a leak problem. The engineer raises the furnace peak temperature by 10 °C: leaks go down a little. He tries more flux: no change. He tries a tighter fixture clamp: worse. He settles on the higher temperature and moves on. Six months later a designed experiment on the same line shows that the tighter clamp is by far the best change, but only together with the extra flux; on its own it starves the joint. One-factor-at-a-time never sees a combination it did not try, and the combination that matters is the one it did not try.

One factor at a time, and why it fails

Suppose two factors each have two levels. One-factor-at-a-time (OFAT) starts at a baseline, changes A and keeps the better level, then changes B at that level and keeps the better. It visits three of the four corners of the square and never the fourth. If the effect of B depends on the level of A, which is what an interaction means, the fourth corner can be the best by a wide margin and OFAT walks away from it. The factorial runs all four corners, and with the same number of measurements gives each effect twice the precision, because every run contributes to every effect.

Worked example 1: the corner OFAT never visits

The data below is a constructed example generated by a seeded script, not a real production run. Setting: the lap-shear strength in MPa of a structural adhesive; factor A is cure temperature (80 °C low, 120 °C high), factor B is cure time (30 min low, 60 min high); two specimens per combination, eight runs in random order, tester to 0.1 MPa.

The OFAT path. Start at the baseline, both low: 15.50 MPa. Raise the temperature: 11.65 MPa, worse by −3.85; keep 80 °C. Lengthen the cure at 80 °C: 13.55 MPa, worse by −1.95; keep 30 min. Conclusion of OFAT: the baseline is best, neither factor helps. The corner it never visited, 120 °C for 60 min, gives 22.05 MPa, 6.55 MPa or 42 % better than the baseline.

The factorial view. The main effect of A is the average change when A goes from low to high, over both levels of B: ((11.65 + 22.05) − (15.50 + 13.55))/2 = 2.325 MPa. The main effect of B likewise is 4.225 MPa. Both are positive on average, which is already something OFAT could not see. The interaction AB is half the difference between the effect of A at high B (8.50) and the effect of A at low B (−3.85): AB = ((15.50 + 22.05) − (11.65 + 13.55))/2 = 6.175 MPa, the largest effect of the three. Temperature hurts a short cure and helps a long one. The ANOVA (next section explains it) gives p = 0.00012 for the interaction. Chemically this is the ordinary story of a cure that needs both heat and time to complete; statistically it is the reason the factorial exists.

The 2^k factorial design

Each of k factors is set at two levels, coded −1 and +1 (written − and +). All 2^k combinations are run, each r times (r replicates), in random order. The runs are listed in standard order: A alternates fastest, then B, then C. Each treatment combination is labelled by the lower-case letters of the factors at their high level, with (1) for all low: for k = 3, (1), a, b, ab, c, ac, bc, abc. The columns of signs for the interactions are the products of the main-effect columns.

Worked example 2: a 2³ with two replicates on a moulded housing

Constructed data, not a real production run. Setting: the overall length of an injection-moulded housing, nominal 45.20 mm, measured on a CMM to 0.01 mm. Factors: A melt temperature (230/250 °C), B hold pressure (40/60 MPa), C cooling time (15/25 s). Two parts per combination, 16 runs in random order.

Effects by hand. Mean of the eight runs at high melt temperature (a, ab, ac, abc, both replicates): 45.1612 mm; at low: 45.2413 mm; effect of A = −0.0800 mm, that is −0.080 mm. For B: 45.2450 − 45.1575 = 0.0875 mm. The interaction AB from its sign column (Table 2 applied to Table 3): −0.050 mm. The contrast for A is 8 × effect = −0.640, and its sum of squares is contrast²/16 = 0.02560.

Reading it. The standard error of an effect is 2√(0.000575/16) = 0.0120 mm, so any effect beyond about 0.03 mm (2.3 standard errors, the 5 % t on 8 degrees of freedom) is unlikely to be noise. Three terms clear that bar: hotter melt shortens the part by −0.080 mm (F = 44.5 against a 5 % critical value of 5.32), higher hold pressure lengthens it by 0.0875 mm, and the two interact: at low pressure the temperature effect is −0.0300 mm, at high pressure only −0.1300 mm. Cooling time (p = 0.18) and the other interactions are within noise. The model explains 0.937 of the total variation; the residual standard deviation of 0.0240 mm is the part-to-part noise at fixed settings, consistent with the replicate pairs in Table 3.

Using it. The prediction equation in coded units is ŷ = 45.2013 + −0.0400·A + 0.04375·B −0.0250·AB (dropping the terms within noise). The target is 45.20 mm. At high temperature and low pressure the model predicts 45.1425 mm, −57 µm short; at low temperature and low pressure 45.1725 mm, −27 µm short; at both high 45.1800 mm, −20 µm long. None of the corners hits nominal, and because of the interaction the answer is not on a straight line between them: the next step is a confirmation run at an interpolated pressure, or a small response-surface study (below). Note what the experiment has also told you for free: the cooling time can be cut from 25 s to 15 s with no measurable effect on length, which is a cycle-time saving that nobody asked the experiment about.

Calculator: 2^k factorial effects

Pre-loaded with Table 3. Enter responses in standard order (A alternates fastest); the calculator prints the effects, the ANOVA when there are replicates or Lenth's method when there are not, a Pareto of effects with the noise reference line, and the cell means.

Randomisation, replication, blocking, and centre points

• Randomise the run order. Standard order is for the arithmetic; the runs happen in random order so that anything drifting during the experiment (tool wear, ambient temperature, operator fatigue) is spread across the effects instead of loading onto one of them. Fisher's insistence on this is the foundation of the whole subject.^[1] If a factor is genuinely hard to change (a furnace temperature that takes an hour to settle), say so and use a split-plot analysis; do not pretend it was randomised.
• Replicate. A second run of every combination doubles the cost and gives you an honest estimate of noise, F tests, and a check that the effects are larger than the part-to-part scatter. Without replicates the design is saturated and the noise must be estimated from the small effects (Lenth's method, below), which assumes most effects are small.
• Block. If the 16 runs of a 2³ × 2 cannot be made in one shift, put one replicate in each shift: the shift difference is then confounded with nothing but the replicate label, and can be removed from the residual. For an unreplicated 2³ run over two days, confound the day with the ABC interaction (runs 1, 4, 6, 7 on day 1, the rest on day 2, from the ABC column of Table 2), sacrificing the effect you least expect to need.^[3]
• Add centre points. Three or four runs with every factor at its middle level, spread through the sequence, estimate pure error even without replicates and test for curvature: if the centre-point mean differs from the average of the corners by more than the noise, the response is not planar and a two-level design cannot describe it.^[7]

Fractional factorials and confounding

A 2⁵ needs 32 runs, a 2⁶ 64. Most of those runs estimate three- and four-factor interactions that are almost always negligible. A fractional factorial runs a chosen fraction, half or a quarter, and gives up the ability to separate some effects from each other. The price is stated exactly.^[4]

To build a 2⁴⁻¹ (four factors in eight runs): write the full 2³ in A, B and C, then set the fourth factor's column equal to the ABC product column, D = ABC, the generator. Multiplying both sides by D gives the defining relation I = ABCD (I is the column of all +). Multiplying any effect by the defining relation gives its alias, the effect it cannot be separated from: A = BCD, B = ACD, C = ABD, D = ABC, AB = CD, AC = BD, BC = AD. Each contrast you compute is the sum of an effect and its alias. The resolution of a design is the length of the shortest word in its defining relation: here IV, meaning main effects are aliased only with three-factor interactions and two-factor interactions with each other. In a resolution III design (I = ABC for a 2³⁻¹) main effects are aliased with two-factor interactions, which is why resolution III designs are used only for screening many factors when interactions are assumed absent, and resolution V or higher when two-factor interactions must be estimated cleanly.^[4] Plackett-Burman designs extend the idea to 12, 20, 24 runs for screening up to 11, 19, 23 factors at resolution III.^[5]

Worked example 3: a half fraction on a plating bath

Constructed data, not a real production run. Setting: nickel plating thickness in µm measured by XRF to 0.1 µm. Factors: A current density (2/4 A/dm²), B bath temperature (50/60 °C), C agitation (off/on), D plating time (20/30 min). Eight runs, generator D = ABC, run in random order, no replicates.

Effects. Analysed as a 2³ in A, B and C, the seven contrasts estimate the following aliased pairs: A + BCD = 5.875 µm; B + ACD = 0.725; C + ABD = −0.225; AB + CD = −0.125; AC + BD = 0.425; BC + AD = 0.175; and the ABC contrast, which is D + ABC = 8.825 µm. Two stand out: current density and time, exactly what Faraday's law would predict for a plating bath, which is the kind of agreement between physics and arithmetic that you should always look for.

Lenth's method. With no replicates there is no residual. Lenth's method estimates the noise from the effects themselves, on the assumption that most of them are noise.^[13] Sort the absolute effects: the median of the seven is 0.425, so s₀ = 1.5 × 0.425 = 0.6375. Discard any effect larger than 2.5 s₀ = 1.594 (that removes A and the ABC contrast, 2 of the seven) and take the median of the rest, 0.225: the pseudo standard error is PSE = 1.5 × 0.225 = 0.3375 µm. The margin of error is t_0.975 on m/3 = 2.33 degrees of freedom, 3.764, times the PSE: ME = 1.270 µm. Effects beyond it: A (5.875) and D + ABC (8.825). The next largest, B at 0.725, is 0.57 of the margin and is treated as noise. (Lenth's stricter simultaneous margin, SME = 3.040 µm, protects the whole set of seven comparisons and reaches the same verdict.)

The alias caveat. The 8.825 µm attributed to plating time is strictly D + ABC. Attributing it to D rests on the assumption that a three-factor interaction of current, temperature and agitation is negligible, which is reasonable here and should be stated. Had the AC + BD contrast been large, the experiment could not have said whether current interacts with agitation or temperature with time; a follow-up half fraction with the opposite generator (D = −ABC) would separate them, and the two halves together are the full 2⁴. Prediction at the best corner (high current, long time, the others at centre): 25.162 + 2.9375 + 4.4125 = 32.51 µm, against 17.81 µm at the worst.

A published check: Montgomery's plasma-etch experiment

The harness behind this course reproduces the 2³ plasma-etch example from Montgomery's Design and Analysis of Experiments (gap, gas flow and power against etch rate, two replicates), as its data, effects and ANOVA are reproduced in a University of Washington lecture handout.^[14][15] The published effects are A = −101.625, B = 7.375, C = 306.125, AC = −153.625 Å/min, with residual standard deviation 47.46 and R² = 0.9661; the course's three independent routes match every published figure. That example also shows a large interaction, AC, 2 times the size of the gap main effect: power and gap do not act independently on etch rate, and an OFAT study would have reported whichever slice of it happened to be run.

Beyond two levels: response surface methods

A two-level factorial fits a plane (with twists for the interactions). It tells you the direction in which the response improves, not where it stops improving. Box and Wilson's method for reaching an optimum uses the factorial in sequence: fit the plane, move along the path of steepest ascent (the direction of the effects, in proportion to their sizes) with a few runs until the response stops improving, run a new factorial there, and when centre points show curvature, augment it into a design that can fit a quadratic.^[6] The standard augmentation is the central composite design: the factorial corners, the centre points, and "star" points along each axis; the Box-Behnken design is the alternative that avoids the extreme corners. Both need each factor at three or more levels and fit the second-order model ŷ = b₀ + Σ b_ix_i + Σ b_iix_i² + Σ b_ijx_ix_j, whose contours locate the optimum.^[7]

For a Green Belt the practical points are three. Use centre points in every factorial so that curvature announces itself. When it does, do not fit a two-level model to a curved response. And bring in someone who has run a response surface study before, because the design choices (axial distance, rotatability, the number of centre points) have consequences that the software's defaults hide.^[7]

Taguchi methods and the debate

Taguchi's contribution has three parts, and the debate treated them differently.^[11][12]

1. The idea of robust design. Choose the settings of the factors you control so that the product performs well despite the factors you cannot control (ambient temperature, material lot, wear), rather than trying to control the uncontrollable. This is universally accepted as important, and Module 15 builds on it with the loss function.
2. Inner and outer arrays. Run the control factors in one array and, for each row, run the noise factors in another, so that the variation across the noise array measures robustness. The NIST handbook describes the arrangement and notes that Taguchi's orthogonal arrays are the familiar fractional factorials under another name.^[8] The arrays are fine; they are mostly resolution III, so the alias structure of the previous section applies and interactions among control factors are confounded with main effects, which the method's users often did not know.
3. Signal-to-noise ratios as the response. Collapsing the noise-array results into an SN ratio ("larger is better", "nominal is best") and analysing that. This is where the critics concentrated. Box showed that the SN ratios confound location and dispersion effects and that analysing the mean and the log of the variance separately, or transforming the response, does the job more transparently.^[9] The 1992 panel recorded broad agreement on the value of the robustness goal and on the weaknesses of the SN analysis and of the accompanying analysis-by-marginal-means without interactions.^[10]

The position this course takes, which is the mainstream one: keep the goal, keep the noise factors, and analyse the experiment as a factorial with the noise factors as factors (a crossed or combined array), reporting the mean and a dispersion measure separately with the standard methods of this module. If a customer or a supplier hands you an L8 or L9 array, it is a fractional factorial; write its alias structure before you believe its conclusions.

Common mistakes

1. One factor at a time. Consequence: the best combination is never run and the interaction is never seen. Fix: a 2^k or a fraction of it, always.
2. Running in standard order. Consequence: a drift during the day lands on factor C, which happens to be the one that changed halfway through. Fix: randomise, record the actual order, and chart the residuals against it.
3. Ranges too narrow. Consequence: real effects sit inside the noise and the report says "no factor matters". Fix: set levels bold enough to produce a change you could see without statistics, within the safe operating window.
4. No replicates and no centre points. Consequence: no estimate of noise, no F tests, no curvature check; every effect looks either huge or negligible. Fix: replicate, or add three or four centre points, or accept Lenth's assumption and say so.
5. Ignoring the alias structure of a fraction. Consequence: a "main effect" that is really a two-factor interaction gets built into the process. Fix: write the defining relation and the aliases before the runs; state the assumption behind each attribution.
6. Reading main effects when there is an interaction. Consequence: "temperature raises strength by 2.3 MPa" when it lowers it at short cures. Fix: interpret an interaction with the two-way table of means, not the marginal means.
7. Stopping at the ANOVA. Consequence: a list of p-values and no setting. Fix: the prediction equation, the predicted response at the chosen setting, and a confirmation run before anything is changed on the line.
8. Extrapolating the plane. Consequence: a setting outside the ranges tested, where the response curved and the plane did not. Fix: predictions inside the box only; a response surface study beyond it.

Exercises

Exercise 1: spot-weld nugget diameter, 2² with three replicates

Constructed data, not a real production run. Setting: the nugget diameter in mm of resistance spot welds, measured by peel test to 0.01 mm. Factors: A weld current (8/10 kA), B weld time (10/14 cycles). Three welds per combination, 12 runs in random order.

Tasks. (a) Compute the three effects from the cell means. (b) Compute SS_A from its contrast, the residual mean square, the standard error of an effect and the F ratio for A. (c) Which effects are real? (d) The specification minimum is 5.6 mm. Which settings satisfy it, according to the model, and what would you do next?

Exercise 2: surface roughness, 2³ unreplicated

Constructed data, not a real production run. Setting: the surface roughness Ra in µm of a turned shaft, profilometer to 0.01 µm. Factors: A feed (0.1/0.2 mm/rev), B cutting speed (150/250 m/min), C depth of cut (0.5/1.0 mm). Eight runs, no replicates, random order.

Tasks. (a) Compute the seven effects. (b) Apply Lenth's method by hand: s₀, the trimmed set, PSE, ME. (c) Which effects are active, and what does the interaction mean physically? (d) Recommend a setting for minimum roughness and predict Ra there.

Quiz

Ten questions. Score 70 % or more to mark the module complete on this device.

Key takeaways

• Changing one factor at a time never runs the combination it did not think of, and cannot see interactions. A factorial runs every combination and gets every effect from every run.
• In a 2^k design each effect is the average response at the high level minus the average at the low level; interactions come from the product columns; SS = contrast²/N; the standard error of an effect is 2√(MSE/N).
• Randomise the run order, replicate or add centre points, and block what you cannot randomise. Report the prediction equation and a confirmation run, not only the p-values.
• A half fraction saves half the runs and aliases effects in pairs given by the defining relation. Write the alias structure before the experiment; resolution IV keeps main effects clear of two-factor interactions, resolution III does not.
• Unreplicated designs are analysed with Lenth's pseudo standard error and a Pareto or normal plot of effects; the method assumes most effects are noise.
• When centre points show curvature, a two-level model is wrong; response surface designs (central composite, Box-Behnken) fit the quadratic and locate the optimum.
• Taguchi's robustness goal and noise factors are worth keeping; his arrays are fractional factorials; analyse them with standard methods rather than signal-to-noise ratios.

References

All web sources accessed 2026-09-10. Sources marked "catalogue" or "secondary" were not read in full by the course author; the claim is taken from the part shown. Standards and books are paraphrased.

1. Fisher, R. A. The Design of Experiments. Oliver and Boyd, 1935. https://archive.org/details/in.ernet.dli.2015.502684 (catalogue; randomisation, replication, blocking and the factorial principle)
2. Yates, F. The Design and Analysis of Factorial Experiments. Imperial Bureau of Soil Science, Technical Communication 35, 1937. https://link.springer.com/rwe/10.1007/978-0-387-32833-1_429 (secondary: Yates' algorithm as described in later texts; the original was not read)
3. Box, G. E. P., Hunter, J. S., and Hunter, W. G. Statistics for Experimenters, 2nd ed. Wiley, 2005. https://www.wiley.com/en-us/Statistics+for+Experimenters... (catalogue; factorial design, effects, blocking)
4. NIST/SEMATECH. "5.3.3.4. Fractional factorial designs" and subsections (5.3.3.4.1 half fractions, 5.3.3.4.4 confounding, 5.3.3.4.5 design resolution). e-Handbook of Statistical Methods. https://www.itl.nist.gov/div898/handbook/pri/section3/pri334.htm
5. Plackett, R. L., and Burman, J. P. "The Design of Optimum Multifactorial Experiments." Biometrika 33(4):305 to 325, 1946. https://academic.oup.com/biomet/article-abstract/33/4/305/225377 (abstract read)
6. Box, G. E. P., and Wilson, K. B. "On the Experimental Attainment of Optimum Conditions." Journal of the Royal Statistical Society, Series B 13(1):1 to 45, 1951. https://academic.oup.com/jrsssb/article/13/1/1/7026650 (abstract read)
7. NIST/SEMATECH. "5.3.3.6. Response surface designs." e-Handbook of Statistical Methods. Centre points and curvature, central composite and Box-Behnken designs, rotatability. https://www.itl.nist.gov/div898/handbook/pri/section3/pri336.htm
8. NIST/SEMATECH. "5.5.6. What are Taguchi designs?" e-Handbook of Statistical Methods. Orthogonal arrays as fractional factorials; inner and outer arrays. https://www.itl.nist.gov/div898/handbook/pri/section5/pri56.htm
9. Box, G. E. P. "Signal-to-Noise Ratios, Performance Criteria, and Transformations (with discussion)." Technometrics 30(1):1 to 40, 1988. https://www.tandfonline.com/doi/abs/10.1080/00401706.1988.10488313 (abstract read)
10. Nair, V. N. (ed.). "Taguchi's Parameter Design: A Panel Discussion." Technometrics 34(2):127 to 161, 1992. https://www.stat.cmu.edu/technometrics/90-00/vol-34-02/v3402127.pdf (abstract read)
11. Taguchi, G. Introduction to Quality Engineering: Designing Quality into Products and Processes. Asian Productivity Organization, 1986. https://archive.org/details/introductiontoqu0000tagu (catalogue)
12. Phadke, M. S. Quality Engineering Using Robust Design. Prentice Hall, 1989. https://books.google.com/books/about/Quality_Engineering_Using_Robust_Design.html?id=TZoQAQAAMAAJ (catalogue)
13. Lenth, R. V. "Quick and Easy Analysis of Unreplicated Factorials." Technometrics 31(4):469 to 473, 1989. https://www.tandfonline.com/doi/abs/10.1080/00401706.1989.10488595 (abstract read; the s₀, PSE and ME definitions are the standard ones implemented in statistical software)
14. Montgomery, D. C. Design and Analysis of Experiments, 8th ed., Wiley, 2013 (the plasma-etch 2³ example, Section 6.3); 10th ed., 2019. https://www.wiley.com/en-us/Design+and+Analysis+of+Experiments,+10th+Edition-p-9781119492443 (catalogue; the example's data and results are taken from [15])
15. University of Washington, STAT 502 lecture notes. "The 2ᵏ Factorial Design (Montgomery chap. 6; BHH chap. 5)." https://sites.stat.washington.edu/pds/stat502/LectureNotes/2k.factorial.intro.pdf (secondary: reproduces Montgomery's plasma-etch data, effects and ANOVA, which the course harness matches)

Further reading

• NIST/SEMATECH e-Handbook, Chapter 5 (Process Improvement), for the complete treatment of design selection, analysis and response surface methods with worked examples.
• Module 15 for the Taguchi loss function and robust design; Module 11 for the F tests and p-values used in the ANOVA tables.