Quick answer: Possibly. When a new treatment is always tested after the existing process, changes in equipment temperature, material batches, operator behaviour or system condition can become confused with the treatment effect. Randomising test order, blocking known sources of variation or addressing carryover between runs can strengthen the comparison. Where the sequence cannot be changed, the conclusion should reflect what the experiment can and cannot separate.
Why Test Order Can Distort an Engineering Result
A manufacturing company is investigating whether a revised process can improve the consistency of a specialist coating.
The engineering team begins by testing the established process. It then installs the modified configuration and conducts another series of trials.
The new process produces more consistent results.
At first glance, the experiment appears successful.
But something else has changed.
The production equipment has now been operating for longer. Its temperature has stabilised. The coating materials come from a different preparation batch. The operators have also become more familiar with the test procedure.
The observed improvement may reflect the new configuration.
It may also reflect conditions that changed as the experiment progressed.
The problem is not necessarily poor measurement. Every observation could be recorded accurately while the comparison itself remains difficult to interpret.
This is an order effect: an influence associated with when or in what sequence experimental treatments are tested.
Order effects matter whenever experimental conditions can change over time.
In manufacturing, equipment warm-up, component wear or material preparation may influence later runs.
In software testing, system load, cache state or shared infrastructure may change between evaluations.
In environmental testing, temperature, moisture or other external conditions may vary as trials proceed.
If the existing process is always tested first and the proposed process always tested later, treatment and sequence become difficult to separate.
Research published through the Australian National University discusses how blocking and treatment structures can be used to accommodate and interpret extraneous variation in designed experiments. The important principle is that relevant variation should be considered in the experimental design rather than assuming every run occurs under equivalent conditions. The Australian National University
The practical question is therefore not simply:
Did the later configuration perform better?
It is:
Could something associated with being tested later have contributed to the difference?
Design the Test Sequence Around the Likely Sources of Variation
There is no single sequence that suits every engineering experiment.
The appropriate design depends on what is being tested, how easily treatments can be changed and which conditions are likely to vary during the work.
The important step is to consider sequence effects before interpreting the difference between treatments.
Randomise the order where practical
If treatments can be applied independently, randomising the order of experimental runs can reduce the risk that one treatment is consistently associated with earlier or later conditions.
Consider a materials investigation comparing an established formulation with a proposed alternative.
Instead of testing every specimen using the established formulation first and every alternative formulation afterwards, the researchers may vary the order in which the treatments are tested.
This can help prevent time-related changes from systematically favouring one treatment.
Randomisation does not make every run identical.
It does not correct an unsuitable measurement method or guarantee that all important sources of variation will balance perfectly.
Its value is that treatment order is less likely to be systematically aligned with other changing conditions.
NIST's engineering experimental-design guidance describes randomisation in terms of randomly assigning the run sequence and contrasts it with an unrandomised design in which all runs at one treatment level are completed before moving to the next. NIST
Use blocks for known sources of variation
Sometimes researchers already know that a particular condition is likely to affect the result.
A production experiment may use several material batches.
Equipment behaviour may differ between shifts.
A field trial may span meaningfully different environmental conditions.
In these situations, blocking can help organise the comparison around a known source of variation.
For example, rather than testing the established process only on earlier material batches and the revised process only on later batches, the experiment may compare both processes within each suitable batch.
The treatment effect can then be assessed without automatically treating batch-to-batch variation as part of the treatment difference.
NIST describes blocking as a technique for dealing with important nuisance factors so that the factor of interest can be compared within more homogeneous groups. The ANU research similarly demonstrates how blocking structures can accommodate extraneous variation in practical experimental designs. NIST
Blocking should therefore reflect a technically meaningful source of variation.
It is not simply a way to organise the results after the experiment has finished.
Consider whether one treatment affects the next
Sequence can matter for another reason.
The previous treatment may change the conditions under which the next treatment is tested.
A coating formulation may leave residue in processing equipment.
A thermal process may leave equipment at a different temperature.
A software configuration may alter cache contents or system state.
In these cases, simply changing the order of treatments may not be enough.
The experimental method may need an appropriate cleaning step, reset, stabilisation period or other procedure intended to reduce the carryover from one run into the next.
Where carryover cannot be eliminated, the research team should identify whether it can be observed or otherwise accounted for when interpreting the results.
A balanced sequence may sometimes help.
It should not be assumed to remove every carryover effect.
The relevant question is what physical or technical mechanism connects one run to the next.
When the Test Sequence Cannot Be Changed
Some experiments cannot be freely randomised.
A manufacturing process may require substantial reconfiguration between treatments.
A material transformation may be irreversible.
Technical or operational constraints may require the experiments to proceed in a particular order.
That does not automatically make the investigation uninformative.
But it changes the strength of the conclusion that can reasonably be drawn.
The research team should identify which conditions are changing at the same time as the treatment.
It may be possible to take reference measurements at relevant stages, observe the system during stabilisation or conduct additional comparisons under more similar conditions.
In the coating example, the researchers might establish whether equipment temperature has stabilised before comparing the two configurations.
They might also record whether material properties differ between preparation batches.
Where technically feasible, reversing part of the sequence in a later trial may provide further evidence about whether the apparent treatment difference persists.
If treatment and another changing condition cannot be separated, that limitation should remain visible in the conclusion.
For example:
The revised process produced more consistent results during the later test period, but the design did not fully separate the process change from changes in equipment state and material batch.
That conclusion is narrower than saying the revised process caused the improvement.
It is also more faithful to the evidence.
Preserve the Actual Test Sequence
The research record should make clear not only what was tested, but when and in what sequence the trials occurred.
Relevant information may include:
• treatment or configuration used for each run;
• actual run order;
• equipment state;
• material batch;
• relevant environmental conditions;
• reset, cleaning or stabilisation procedures;
• interruptions or unexpected events; and
• changes from the planned experimental sequence.
This is a practical experimental-record framework rather than a prescribed R&DTI template.
Where the planned sequence changes, the reason should also be recorded.
An unexpected result may legitimately cause the research team to alter the experimental programme.
What matters is that the later record does not make the revised sequence appear to have been the original design.
Current Australian Government R&DTI guidance states that records should be created at the time the activity is conducted. Records should show relevant background research and details of what was done, how it was performed, when it occurred and who carried out the work. Experimental plans, trial-run records and test reports showing results and evaluation are examples of records that may demonstrate the systematic progression of work. business.gov.au
For experimental sequence, those records have a practical purpose regardless of any later R&DTI assessment.
They allow another technically competent person to understand whether an apparent treatment effect might instead be associated with the order in which the work was performed.
Experimental Design Does Not Determine R&DTI Eligibility
Randomisation, blocking and planned treatment sequences can strengthen experimental evidence.
They do not by themselves establish that an activity is a core R&D activity under the Australian R&D Tax Incentive.
Under current guidance, a core R&D activity must be conducted for the purpose of generating new knowledge.
Its outcome must be unable to be known or determined in advance by a competent professional on the basis of publicly available or reasonably accessible knowledge, information or experience, and the outcome must be determined through a systematic progression of work based on principles of established science, proceeding through hypothesis, experiment, observation, evaluation and logical conclusions. business.gov.au
An established engineering test can use rigorous randomisation or blocking without satisfying those requirements.
Conversely, where a genuine unknown outcome is being investigated experimentally, the sequence of runs may materially affect whether the resulting observations can support the hypothesis being tested.
The experimental-design method and the R&DTI eligibility test are therefore related only in the sense that a sound experiment may be necessary to investigate the technical question.
The use of a particular statistical or design technique is not itself an eligibility criterion.
Conclusion
An improved result in the second half of an experiment is not automatically evidence that the second treatment is better.
If equipment condition, material batches, environmental conditions or system state also changed with time, treatment and sequence may be confounded.
Randomising run order can reduce some of that risk.
Blocking can help address known sources of variation.
Reset or stabilisation procedures may be necessary where one treatment affects the next.
And when the sequence cannot be changed, the limitation should remain visible in the conclusion.
The useful question is not simply:
Which treatment produced the better number?
It is:
Would the comparison still support the same conclusion if test order were not aligned with the treatment?
Sources & Further Reading
Talk to Ignition Research if your manufacturing, software or engineering experiments involve sequential testing under changing operating conditions and the order of experimental runs could affect how the result is interpreted. As a Registered Research Service Provider, Ignition Research assists businesses with research question definition, experimental design, technical investigations and contemporaneous supporting records within its registered research fields.
RSP registration confirms that the provider has met the relevant registration requirements for its registered research fields. It does not constitute government endorsement of a project or guarantee R&DTI eligibility.
We do not determine R&DTI eligibility or provide tax advice. The R&D Tax Incentive is a self-assessment program, and companies remain responsible for assessing their activities, expenditure and circumstances and obtaining appropriate tax advice.
This article provides general information only and is not tax, legal or financial advice.
Working on a project like this?
Tell us what you're building and the part you're not sure will work. A researcher from our Adelaide team (registered Research Service Provider RSP000047) reads every enquiry and replies within two business days. No documents needed.
