Quick answer: A decision criterion should usually be defined before the result is known, because the experiment needs a clear basis for deciding what the observation means. That criterion does not itself make an activity R&D, and routine verification against an established specification is ordinarily different from genuine technical uncertainty. But where an outcome cannot be known or determined in advance by a competent professional on the basis of current knowledge, information or experience, a pre-defined decision rule can help connect the hypothesis, experiment, evaluation and conclusion.
A hypothesis needs a decision, not just a result
Many experiments begin with a technical question but no clear rule for interpreting the answer.
A team might say:
“We want to see whether the new control method improves stability.”
“We need to determine whether this material performs better.”
“We want to know whether the new process is more reliable.”
These may be reasonable starting questions.
They are not yet complete experimental decisions.
What does “better” mean?
How much improvement would matter?
Which measure represents stability?
Would an average improvement be enough if the worst-case behaviour becomes less predictable?
What result would justify moving to the next stage?
Without those questions being addressed before testing, almost any result can become open to interpretation after the event.
The Australian R&DTI legislation defines core R&D activities around a systematic progression from hypothesis to experiment, observation and evaluation, leading to logical conclusions. Current Australian Government guidance similarly says evaluation should assess what experimental results mean and consider causal relationships between the parameters varied, held constant, tested or measured.
That evaluation becomes easier when the team has already decided what evidence would materially change its view of the hypothesis.
A decision criterion does not have to be a single pass/fail number.
A decision criterion does not have to be a single pass/fail number.
It may be a range.
It may involve several measures.
It may identify a failure condition that matters more than average performance.
It may also identify conditions under which the evidence should be treated as inconclusive and a revised hypothesis or follow-up experiment is needed.
The important point is that the criterion reflects the question the experiment was designed to answer.
Separate technical significance from a convenient target
A common mistake is to choose a threshold because it is easy to measure rather than because it matters technically.
Suppose a team is investigating whether a revised mechanical system reduces vibration.
It may be tempting to define success as:
“Any reduction is an improvement.”
But a very small reduction may sit within measurement variation or have no meaningful effect on the system.
At the opposite extreme, the team might choose an ambitious target copied from a future product requirement even though the experiment is only intended to test whether a particular mechanism is real.
Both approaches can obscure the research question.
A useful criterion asks:
What difference would change the technical conclusion or the next engineering decision?
For example, the first experiment may not need to prove that the final commercial specification can be achieved.
It may only need to determine whether changing one variable produces a consistent directional effect under defined experimental conditions.
If that effect is established, a later experiment can investigate magnitude, operating range or robustness.
This matters because technically uncertain projects often involve several decision stages.
An early experiment may answer:
Does the proposed mechanism exist?
A later one may ask:
Does it remain stable across expected variation?
A pilot may ask:
Does that behaviour persist under operating conditions?
A scale-up experiment may ask:
Does the mechanism survive the changes introduced by production equipment, geometry or throughput?
Using the final commercial acceptance criterion for every one of those stages can make useful intermediate experiments appear to have failed.
Using no criterion at all creates the opposite problem: the team can keep redefining success after seeing the data.
The better approach is to define a criterion appropriate to the specific uncertainty being tested.
Do not quietly move the goalposts after seeing the data
Real research rarely follows the original plan perfectly.
A team may discover that its initial threshold was unrealistic.
The measurement method may have less resolution than expected.
A new failure mode may appear.
The first run may show that a different variable is controlling the outcome.
Changing the criterion is not automatically a problem.
Changing it without recording why is.
Suppose a project originally states that the prototype must remain stable across a defined operating condition.
The result falls short.
After reviewing the data, the team realises that the original limit was based on an incorrect assumption about the expected environment.
There may be a legitimate reason to revise the criterion.
But the research record should preserve:
• the original criterion;
• why it was selected;
• the result against that criterion;
• the evidence that caused the team to reconsider it;
• the revised criterion; and
• whether the next run is testing the same hypothesis or a different one.
That record protects the logic of the research.
It distinguishes a genuine learning-driven change from a threshold being adjusted merely so that the outcome can be described as successful.
Current government guidance emphasises contemporaneous records: records should be created while activities are being conducted, and examples include experimental plans, trial records and test reports showing both results and evaluation.
This is particularly important where the experimental sequence changes.
If the final report shows only the last version of the success criterion, the pathway that produced that decision disappears.
The result against the original criterion may itself be part of the technical evidence.
It may show that one assumption was wrong, that the operating window is narrower than expected or that a different mechanism needs to be investigated.
A decision criterion should inform the next step
A useful experiment should inform a decision.
Not necessarily a commercial decision.
Not necessarily a final design.
But some change in what the team should do next.
Before the experiment begins, it can therefore help to write three possible outcomes.
If the result clearly meets the criterion: what follows?
Perhaps the team proceeds to a broader operating envelope, field trial or scale-up experiment.
If the result clearly fails: what follows?
Perhaps the hypothesis is weakened, the design is changed or the project stops.
If the result is ambiguous: what follows?
Perhaps the measurement method needs improvement, additional replication is required or a confounding variable needs to be separated.
That third category is often overlooked.
Experiments do not always produce clean positive or negative answers.
A well-designed decision rule should allow the team to recognise when the evidence is genuinely inconclusive instead of forcing it into success or failure.
This also helps management decisions.
For founders and CFOs, the research question is rarely interesting in isolation.
What matters is whether another investment step is justified.
For CTOs and engineering leads, the question is whether the evidence supports implementation, redesign or further experimentation.
A pre-defined decision criterion connects those two levels.
It tells the technical team what evidence matters and gives the business a clearer reason for the next investment decision.
The criterion should therefore be written before the result is known whenever practicable, but it should not be treated as immutable.
Research is allowed to learn.
What matters is that when the criterion changes, the evidence and reasoning behind that change remain visible.
Conclusion
A decision criterion gives an experiment a clearer basis for interpreting its result, but it does not need to reduce every technical question to a single pass/fail threshold.
The criterion should reflect the uncertainty being tested and the decision the evidence is intended to inform. Where the evidence is inconclusive or changes the team's understanding, the criterion or hypothesis may legitimately evolve — provided the reason for that change remains visible in the research record.
The objective is not to make every experiment succeed. It is to make clear what the result means and what should happen next.
Sources & Further Reading
Australian Government — Conducting core R&D activities for the R&D Tax Incentive
Australian Government — Record keeping for the R&D Tax Incentive
Federal Register of Legislation — Income Tax Assessment Act 1997, Division 355
Australian Government — Get help from a research service provider
Talk to Ignition Research if your team can describe the experiment it wants to run but has not yet defined what result would justify proceeding, redesigning or stopping. As a Registered Research Service Provider, Ignition Research assists businesses with research question definition, experimental design, technical investigations and contemporaneous supporting records within its registered research fields.
RSP registration confirms that the provider has met the relevant registration requirements for its registered research fields. It does not constitute government endorsement of a project or guarantee R&DTI eligibility.
We do not determine R&DTI eligibility or provide tax advice. The R&D Tax Incentive is a self-assessment program, and companies remain responsible for assessing their activities, expenditure and circumstances and obtaining appropriate tax advice.
R&DTI note: A pre-defined success or decision criterion does not itself establish R&DTI eligibility. Eligibility depends on whether the actual activity satisfies the applicable core or supporting R&D requirements.
Working on a project like this?
Tell us what you're building and the part you're not sure will work. A researcher from our Adelaide team (registered Research Service Provider RSP000047) reads every enquiry and replies within two business days. No documents needed.
