Quick answer: A field trial can still generate useful experimental evidence when random assignment is impractical, but an observed improvement does not necessarily establish that the intervention caused it. A credible comparison, relevant environmental records and explicit assumptions can help researchers distinguish the proposed technical effect from other plausible explanations. Where causal attribution remains uncertain, the conclusion should reflect that limitation rather than overstate what the trial has demonstrated.
A Before-and-After Improvement Is Not Necessarily a Technical Effect
Many engineering experiments take place in environments that cannot be fully controlled.
A manufacturer trials a revised process on an operating production line. An environmental engineering team tests a new filtration medium at an existing drainage site. A building technology company evaluates an alternative cooling method in an occupied facility.
In each case, the researchers may be unable to randomly assign the experimental treatment across otherwise comparable conditions.
The commercial system already exists. Operating requirements constrain what can be changed. Environmental conditions continue to vary throughout the investigation.
Consider a company developing an alternative biofiltration medium for stormwater treatment.
The team installs the material in an existing roadside treatment system and measures pollutant concentrations before and after the modification.
The results show an improvement.
That finding may be encouraging, but several explanations remain possible.
The rainfall pattern may have changed. Upstream construction activity may have declined. Incoming pollutant concentrations may have fallen independently of the treatment. Maintenance or hydraulic conditions may also have changed.
The trial has demonstrated that the measured outcome changed.
It has not necessarily established why.
This distinction concerns causal attribution: whether the observed difference can reasonably be attributed to the intervention being investigated rather than to other changes occurring at the same time.
The Australian Centre for Evaluation explains the same underlying causal problem through the concept of a counterfactual: what would have happened without the intervention. Because that alternative outcome cannot be observed directly for the same unit at the same time, causal analysis needs a credible basis for comparison. Its guidance is written for government program and policy evaluation rather than engineering R&D, but the underlying causal-inference principle is relevant here. Australian Centre for Evaluation
The important question is therefore not simply:
Did the system improve?
It is:
Can the experimental evidence distinguish the proposed technical effect from other plausible explanations?
Build a Credible Comparison When Randomisation Is Impractical
Random assignment can strengthen causal inference by reducing systematic differences between experimental conditions.
In field engineering, however, randomisation may be impractical or impossible.
An existing treatment facility cannot always be divided into independently operated systems. A building owner may not permit experimental changes to be assigned randomly across essential equipment.
That does not make useful experimentation impossible.
It means the limitations of the comparison need to be addressed deliberately.
Use a relevant untreated or existing condition
Where feasible, a comparable system that does not receive the intervention can provide a useful reference.
For the stormwater example, researchers might monitor a system using conventional media alongside the modified system.
If both are exposed to sufficiently comparable external conditions, the comparison may help distinguish changes associated with the new material from changes affecting both systems.
But comparability cannot simply be assumed.
The sites may differ in catchment characteristics, pollutant loads, hydraulic behaviour or maintenance history.
Those differences should remain visible when the results are interpreted.
The purpose of the comparison is not to create the appearance of a perfect control.
It is to make important competing explanations easier to evaluate.
Compare the pattern of change, not only the final result
Where both an experimental site and a suitable reference site have observations before and after an intervention, a before–after comparison across the two sites may provide stronger evidence than looking only at the experimental site's final outcome.
Before–After–Control–Impact (BACI) designs use this general logic in environmental monitoring: changes at potentially impacted locations are considered relative to changes at reference locations over the same period. Australian environmental monitoring guidance also cautions that natural spatial and temporal variation can still complicate attribution, which is why the suitability of reference sites matters. DCCEEW
Related statistical approaches such as difference-in-differences compare changes over time between treated and comparison groups.
However, causal interpretation depends on important assumptions. In particular, difference-in-differences relies on the idea that, without the intervention, the difference between the treated and comparison groups would have remained stable over time—the parallel trends assumption. Historical data may help assess whether that assumption is plausible, but it cannot prove it. Australian Centre for Evaluation
For a technical field trial, such a method should therefore be used only where its assumptions make sense for the physical system and available data.
A statistical method cannot compensate for an unsuitable comparison.
Consider a staged intervention where the system allows it
Sometimes a technical change can be introduced at different times across otherwise comparable units.
A manufacturer might trial a revised process on one production line while another continues using the existing method.
If the second line later receives the same modification, the first period may provide a useful comparison.
That design may help distinguish the intervention from broader facility-wide changes.
But differences between production lines, materials, operators or operating conditions still need to be considered.
The objective remains the same:
Build a comparison that can challenge the proposed explanation rather than simply confirm it.
Record the Factors That Could Explain the Result
A credible field experiment requires more than measurements of the final outcome.
Researchers also need enough information about the conditions under which those outcomes were produced to evaluate plausible alternative explanations.
In an environmental treatment trial, relevant conditions may include rainfall, flow rates, influent characteristics, maintenance events and upstream activity.
In a manufacturing trial, material batches, operating loads, equipment condition and process settings may matter.
For building systems, external temperature, occupancy and control configuration may influence observed performance.
These factors do not all need to be held constant.
In many field environments, that would be impossible.
The research team should instead identify which factors could materially affect the interpretation and determine how they will be measured, recorded or otherwise addressed.
Measurement consistency also matters.
If sampling locations, instruments or analytical methods change during the investigation, an apparent difference may reflect the measurement process rather than the technical intervention.
The research record should therefore preserve enough context to reconstruct the conditions associated with the observations.
Where the experimental design changes during the trial, the reason should also remain visible.
Current R&DTI record-keeping guidance states that records should be created at the time the activity is conducted. Records should show relevant background research and what was done, how the work was performed, when it occurred and who carried it out. Experimental plans, records of trial runs and test reports showing results and evaluation are examples of records that may demonstrate a systematic progression of work. business.gov.au
Those records also have a practical experimental purpose.
They allow another technically competent person to consider whether the evidence supports the proposed explanation, whether an alternative explanation remains credible and what additional evidence may be needed.
More observations do not necessarily solve a causal-attribution problem.
If an important confounding influence remains unrecognised or unmeasured, collecting more data under substantially the same design may simply reproduce the same ambiguity.
Match the Conclusion to the Strength of the Evidence
A field trial does not need to deliver perfect causal certainty to produce useful evidence.
But the strength of the conclusion should match what the experimental design actually supports.
Suppose the modified stormwater system shows improved pollutant removal relative to a sufficiently comparable untreated system across the conditions investigated.
That may strengthen the case that the treatment contributed to the observed improvement and justify a broader investigation.
If the difference appears only under a narrow set of conditions, the next question may concern whether the effect persists elsewhere.
If both the treated and comparison systems improve similarly, the evidence may weaken the explanation that the new material caused the observed change.
These are different outcomes.
An uncertain causal conclusion should not automatically be treated as evidence that the technology failed.
Equally, an encouraging association should not be presented as proof that the intervention caused the improvement.
A defensible technical conclusion might therefore say:
The intervention was associated with improved performance relative to the selected comparison under the conditions tested, while identified differences between the systems limit the strength of causal attribution.
That is often more useful than making a stronger claim the experiment cannot support.
Keep the R&DTI Question Separate From the Causal Method
A carefully designed field comparison does not, by itself, establish eligibility for the Australian R&D Tax Incentive.
Under current guidance, a core R&D activity must be conducted for the purpose of generating new knowledge and involve an outcome that cannot be known or determined in advance by a competent professional on the basis of knowledge, information or experience that is publicly available or reasonably accessible.
That outcome must be determined through a systematic progression of work based on principles of established science, proceeding from hypothesis to experiment, observation and evaluation and leading to logical conclusions. business.gov.au
The use of a comparison site, BACI-type design or quasi-experimental statistical method does not itself satisfy those requirements.
Routine monitoring, ordinary performance comparison or established validation work does not become core R&D merely because the analysis is sophisticated.
Where the underlying technical outcome genuinely cannot be determined in advance and hypothesis-driven experimentation is required, the field design may form part of the systematic investigation.
The relevant question remains the nature of the technical activity, not the statistical label attached to it.
Conclusion
A field trial can produce useful experimental evidence even when random assignment is not practical.
The challenge is that a before-and-after improvement may have several plausible explanations.
A credible reference condition, consistent measurement and records of relevant environmental and operational changes can strengthen the team's ability to distinguish the proposed technical effect from those alternatives.
Where the design cannot fully resolve causal attribution, the appropriate response is not to manufacture certainty.
It is to state clearly what the evidence supports, what assumptions remain and what additional comparison or experiment would make the next conclusion stronger.
Sources & Further Reading
Australian Centre for Evaluation — Impact Evaluation
Australian Centre for Evaluation — Quasi-experimental Methods
Australian Government — Environmental monitoring guidance on BACI and reference sites
Australian Government — Conducting core R&D activities for the R&D Tax Incentive
Australian Government — Record keeping for the R&D Tax Incentive
Talk to Ignition Research if your engineering or environmental technology project requires field testing but site conditions prevent straightforward randomised experimentation and the challenge is determining what comparison will allow the evidence to be interpreted credibly. As a Registered Research Service Provider, Ignition Research assists businesses with research question definition, experimental design, technical investigations and contemporaneous supporting records within its registered research fields.
RSP registration confirms that the provider has met the relevant registration requirements for its registered research fields. It does not constitute government endorsement of a project or guarantee R&DTI eligibility.
We do not determine R&DTI eligibility or provide tax advice. The R&D Tax Incentive is a self-assessment program, and companies remain responsible for assessing their activities, expenditure and circumstances and obtaining appropriate tax advice.
This article provides general information only and is not tax, legal or financial advice.
Working on a project like this?
Tell us what you're building and the part you're not sure will work. A researcher from our Adelaide team (registered Research Service Provider RSP000047) reads every enquiry and replies within two business days. No documents needed.
