Quick answer: An unexpected experimental result should not be excluded simply because it differs from the other observations or weakens the preferred conclusion. Researchers need to determine whether it reflects a measurement error, an experimental deviation or genuine technical behaviour. Where the cause remains uncertain, the result may require further investigation rather than removal from the analysis.
An Unusual Result May Be the Most Important Observation
A manufacturer is investigating a new bonding method for prefabricated building components.
The research team prepares specimens using a modified adhesive formulation and evaluates their mechanical performance.
Most specimens show improved bond strength compared with the existing method.
However, one specimen fails at a substantially lower load.
The engineers suspect that something went wrong during preparation. They consider removing the result because it does not appear representative of the other specimens.
That may seem reasonable. Experimental measurements can be affected by equipment limitations, preparation errors and other sources of variation.
But there is another possibility.
The unusually weak specimen may reveal an interaction between the adhesive, substrate condition and curing process that the original experimental design did not anticipate.
Removing the observation without investigating its cause could conceal a technically important limitation.
The challenge is to distinguish an invalid measurement from a valid observation of unusual system behaviour.
An instrument malfunction may produce a recorded value that does not represent the physical event. A genuine failure under correctly documented experimental conditions may be unusual but technically valid.
Both can appear as outliers in a dataset.
For a research team, the question is therefore not whether an observation is inconvenient or statistically unusual.
It is whether there is a defensible technical reason to exclude it from the particular analysis.
Investigate the Observation Before Changing the Dataset
An outlier is an observation that differs markedly from other observations or from an expected pattern.
Its presence does not establish the cause.
Statistical screening can help identify unusual observations, but it cannot independently determine whether they are measurement errors or genuine outcomes.
NIST's Engineering Statistics Handbook recommends investigating potential outliers before considering their removal. An unusual observation may reflect erroneous data, but it may also reveal an important feature of the process being studied. NIST — Detection of Outliers
The investigation should begin with the experimental records.
Check the Measurement Process
The research team should examine whether the equipment operated correctly when the unusual result was obtained.
Relevant evidence may include calibration information, instrument settings, equipment logs and observations made during the test.
For a mechanical strength experiment, the team might investigate whether the specimen was positioned correctly and whether the loading arrangement followed the established procedure.
Australia's National Measurement Institute maintains primary measurement standards and supports measurement traceability, helping ensure that measurements can be compared with confidence.
However, traceable calibration does not automatically establish that an individual experiment was performed correctly or that the measurement method was appropriate for the investigation. Those questions still need to be assessed against the experimental conditions. Australian Government — Australia's Measurement System
If the evidence establishes that an equipment malfunction made the measurement invalid, exclusion from the relevant performance analysis may be justified.
The reason should be documented, and the original record preserved.
Examine Specimen Preparation and Experimental Conditions
An unusual result may arise from a difference in how the specimen was prepared.
In the bonding example, relevant variables might include surface condition, adhesive application, material compatibility and curing conditions.
A preparation deviation does not necessarily make the observation irrelevant.
Its significance depends on the research question.
If the experiment was designed to investigate the bonding mechanism under tightly controlled conditions, a documented preparation error may mean that the specimen cannot answer that particular question.
However, if the research concerns whether the bonding method remains reliable under realistic manufacturing variability, the same observation may be important.
The team should consider the purpose of the experiment before deciding how to classify the result.
Avoid Excluding Results Solely to Improve the Analysis
Researchers may be tempted to remove extreme observations because they increase variability or weaken the apparent treatment effect.
This creates a risk of selective interpretation.
Where possible, exclusion criteria should be established before the results are reviewed and applied consistently. If an unusual result is investigated after the event, the reason for any subsequent exclusion should be documented rather than justified solely by the effect on the final conclusion.
The Australian Code for the Responsible Conduct of Research 2018, developed by the National Health and Medical Research Council, the Australian Research Council and Universities Australia, establishes broad principles for responsible research conduct, including honesty, rigour, transparency and accountability. Its supporting guidance addresses the management of research data and information. NHMRC — Australian Code for the Responsible Conduct of Research 2018
The Code's formal application to a particular commercial project depends on the relevant institutional, funding and contractual arrangements. Its underlying principles nevertheless provide a useful benchmark for transparent data handling.
A defensible approach preserves the original observations, explains any exclusions and distinguishes corrections supported by evidence from analytical choices made after examining the results.
The objective is not to retain every observation in every calculation. It is to ensure that the treatment of the data is technically justified and transparent.
What If the Outlier Changes the Conclusion?
An unusual observation becomes particularly important when the project's investment decision depends on whether it is included.
Suppose the modified adhesive formulation appears to provide a substantial improvement when the weak specimen is excluded.
When the specimen is included, the overall result becomes less convincing.
The team should not automatically select whichever analysis produces the more attractive conclusion.
Instead, it should consider what the difference reveals about the available evidence.
Compare Reasonable Analytical Treatments
Researchers may evaluate the results with and without a disputed observation, provided both analyses are clearly identified and the reason for each treatment is explained.
This can help determine how sensitive the conclusion is to the handling of that observation.
If the conclusion remains similar under defensible alternatives, the disputed result may have limited influence on the immediate engineering decision.
If the conclusion changes materially, further investigation may be necessary.
This type of sensitivity analysis does not establish whether the observation is valid. It reveals how strongly the conclusion depends on the assumption made about it.
Investigate the Failure Mechanism
An anomalous result can also suggest a new technical question.
For example, further examination may reveal that the weak bond occurred under a particular combination of surface moisture and curing temperature.
The researchers may then need to determine whether that interaction can be explained using existing technical knowledge.
If the mechanism is already understood, established engineering controls or process adjustments may be sufficient.
If the outcome cannot be known or determined in advance by a competent professional on the basis of current knowledge, information or experience, a further systematic experimental investigation may be appropriate.
The next experiment should test a defined explanation rather than merely attempt to reproduce a more favourable result.
Consider What the Business Needs to Decide
Not every anomalous result justifies another research programme.
Its importance depends on the intended application and the consequences of the observed behaviour.
For a non-critical product characteristic, the uncertainty may have limited influence on the investment decision.
For a bonding system where an unexpected failure could undermine essential performance requirements, the same uncertainty may be significant.
Management should understand whether the unusual result challenges the proposed technical mechanism, the manufacturing process or the interpretation of the measurements.
The appropriate next step may involve improved measurement, a revised experimental design, additional independent specimens or a decision not to proceed under the current assumptions.
Preserve the Original Result and the Reasoning Behind the Decision
Experimental records should allow another technically competent person to reconstruct how an anomalous observation was handled.
The record should identify the original measurement, experimental conditions, relevant equipment information and any deviations from the planned procedure.
It should explain why the result was retained, excluded from a particular analysis or investigated separately.
Where the research team changes its interpretation, the reasoning should remain visible.
Current Australian Government R&DTI record-keeping guidance states that records should be created at the time the activity is conducted. Records should show relevant background research and what was done, how the work was performed, when it occurred, who carried it out and the resources used. Experimental plans, trial records and test reports showing results and evaluation are examples of records that may demonstrate the systematic progression of work. Australian Government — Record Keeping for the R&D Tax Incentive
These records help distinguish an invalid measurement from a genuine technical observation and make the reasoning behind the final conclusion traceable.
Keep the R&DTI Question Separate From the Data-Handling Decision
Excluding an outlier, correcting an established measurement error or applying standard statistical procedures does not automatically constitute core R&D.
Under the current R&DTI requirements, a core R&D activity must be conducted for the purpose of generating new knowledge. Its outcome must be unable to be known or determined in advance on the basis of current knowledge, information or experience, and it must be determined through a systematic progression of work based on principles of established science, proceeding from hypothesis to experiment, observation and evaluation and leading to logical conclusions. Australian Government — Conducting Core R&D Activities
Where an unresolved technical outcome requires systematic experimentation to generate new knowledge, the relevant activities may fall within these criteria, subject to the company's self-assessment.
The existence of an outlier does not itself establish R&DTI eligibility. The record should explain what was actually uncertain and how the investigation addressed that uncertainty.
For founders and CFOs, this provides a clearer basis for understanding whether additional expenditure would resolve a meaningful technical question.
For CTOs and engineering leads, it helps prevent an unexpected observation from disappearing between laboratory testing and the next design decision.
Sometimes an outlier is simply an invalid measurement.
Sometimes it reveals an overlooked operating condition or failure mechanism.
The difference cannot be established by deleting the result. It must be investigated.
Conclusion
An unusual result should not be removed simply because it weakens the preferred conclusion.
The first task is to determine whether the observation is invalid, reflects a deviation relevant to the experimental question or reveals genuine technical behaviour.
Where the evidence establishes that a measurement is invalid, exclusion from the relevant analysis may be justified and should be documented.
Where the observation is technically valid but its cause remains uncertain, it may be necessary to investigate further or limit the conclusion that can be drawn.
The objective is not to make the results look consistent.
It is to ensure that the conclusion reflects the evidence the experiment actually produced.
Sources & Further Reading
NIST/SEMATECH — Detection of Outliers
National Health and Medical Research Council — Australian Code for the Responsible Conduct of Research 2018
Australian Government — Australia's Measurement System
Australian Government — Conducting Core R&D Activities
Australian Government — Record Keeping for the R&D Tax Incentive
Talk to Ignition Research if your materials, manufacturing or engineering experiments produce anomalous results that challenge the expected performance of a new technology. Investigating those observations can help determine whether the uncertainty lies in the measurement process or the underlying technical mechanism. As a Registered Research Service Provider, Ignition Research assists businesses with research question definition, experimental design, technical investigations and contemporaneous supporting records within its registered research fields.
RSP registration confirms that the provider has met the relevant registration requirements for its registered research fields. It does not constitute government endorsement of a project or guarantee R&DTI eligibility.
We do not determine R&DTI eligibility or provide tax advice. The R&D Tax Incentive is a self-assessment program, and companies remain responsible for assessing their activities, expenditure and circumstances and obtaining appropriate tax advice.
This article provides general information only and is not tax, legal or financial advice.
Working on a project like this?
Tell us what you're building and the part you're not sure will work. A researcher from our Adelaide team (registered Research Service Provider RSP000047) reads every enquiry and replies within two business days. No documents needed.
