Quick answer: More measurements do not necessarily mean more independent evidence. If 500 readings all come from the same prototype, batch, device or production run, they may describe that experimental unit very precisely without showing that the result will recur independently. Where a technically uncertain outcome is being investigated, the experimental design should distinguish repeated measurements within an experimental unit from independent replication at the level relevant to the hypothesis.
Repeated measurements and independent replication answer different questions
Commercial R&D can produce large datasets very quickly.
A sensor can record every second. A software benchmark can process thousands of requests. A materials test can capture a continuous response curve. A production line can generate hundreds of observations during one run.
That volume can create an impression of strong evidence.
But the number of data points is not necessarily the number of independent experimental repetitions.
Consider a prototype fitted with a temperature sensor.
During one four-hour trial, the system records thousands of temperature values.
Those readings may tell the team a great deal about how that particular prototype behaved during that particular run.
They do not automatically show that a second independently assembled prototype would behave the same way.
Current Australian Government guidance describes an experiment as a scientific procedure undertaken to test a hypothesis. It states that records should explain how the experiment was conducted or planned and may, for example, describe the parameters varied, held constant, observed or measured.
The practical question is therefore:
What is the thing that was independently subjected to the experimental condition?
In experimental design, that is often the experimental unit: the entity to which the treatment or experimental condition is independently applied.
Depending on the hypothesis, that might be:
• a component;
• a separately manufactured specimen;
• a distinct production batch;
• a different site;
• a separately configured system;
• an independently trained model instance where training variability is the uncertainty being tested; or
• an independent run after the system has genuinely been returned to an appropriate starting condition.
Measurements taken within one of those units may improve understanding of that unit.
They do not automatically create additional independent experimental units.
Repeated measurements from the same experimental unit may also be correlated. Treating every measurement as though it were an independent replication can therefore overstate how much independent evidence the experiment actually contains.
Decide what source of variation the experiment needs to test
Whether independent replication matters depends on the conclusion the team wants to draw.
Suppose a manufacturer changes a material formulation.
The first specimen performs much better than the existing formulation.
The team measures that specimen at ten locations.
All ten measurements are consistent.
That is useful evidence about variation within the specimen.
It does not yet answer whether the manufacturing process will produce the same material properties in the next independently produced specimen.
If the technical question concerns manufacturing repeatability, the experiment needs evidence across independently produced specimens, not simply more measurements on the first one.
The same logic applies in software.
A team may run a new algorithm against one fixed dataset thousands of times.
Those runs may reveal computational stability or sensitivity to random initialisation.
But if the research question is whether performance transfers to materially different real-world data populations, repeatedly analysing the same population may not resolve the relevant uncertainty.
For field engineering, ten sensors installed at one site may provide excellent spatial information about that site.
They are not automatically equivalent to ten independent sites if the conclusion concerns behaviour across different installations or environments.
This is why the experimental unit should follow the hypothesis.
Ask:
Where could the variation that threatens our conclusion actually enter the system?
If it enters during manufacturing, independently manufacture specimens.
If it enters during installation, include independent installations.
If it enters through the data population, design the comparison around genuinely distinct data populations where appropriate.
If it enters between production runs, repeated measurements within one uninterrupted run may not be enough.
The objective is not to maximise the sample count.
It is to expose the source of variation that matters to the technical conclusion.
Do not turn one successful unit into false confidence
This distinction becomes commercially important when a project moves towards scale-up.
Suppose one prototype performs exceptionally well and generates a large volume of clean monitoring data.
Management may reasonably feel that the system has now been “tested extensively”.
But extensive observation of one unit and evidence of repeatable production are different things.
The prototype may contain an unusually favourable component.
Its assembly tolerances may be better than normal.
Its material batch may sit near the centre of the specification.
A technician may have assembled it more carefully than the eventual production process will allow.
None of those possibilities makes the original experiment poor.
They simply define what the evidence actually supports.
A disciplined technical conclusion might be:
The mechanism was demonstrated on this configuration under these conditions.
That is different from:
The production method reliably produces systems with this behaviour.
The second conclusion requires evidence at another level of variation.
This distinction can also prevent unnecessary experimentation.
If the immediate research question is whether a proposed mechanism is technically plausible under a defined set of conditions, one carefully designed experimental unit may sometimes provide enough evidence to justify the next research stage.
That does not yet establish repeatability across independently produced units.
Later, when the question changes from:
Can this work under these conditions?
to:
Can we produce this behaviour reliably across relevant variation?
the experimental design should change with it.
Experiment count should therefore not be set mechanically.
It should follow the claim the evidence needs to support.
Australian Government guidance describes evaluation as analysing experimental results and assessing causal relationships between the parameters varied, held constant, tested or measured. Logical conclusions are then drawn from that evaluation.
A large dataset is valuable only if its structure allows the team to support the conclusion it actually needs.
Record what was independently repeated
Good records should make the experimental structure visible.
A spreadsheet containing 500 rows can look impressive six months later while hiding the fact that every row came from the same specimen.
As a practical traceability approach, the research record should make the hierarchy behind the observations visible.
For example:
• which measurements came from the same specimen;
• which specimens came from the same manufacturing batch;
• which runs used the same configuration;
• whether the system was reset between runs;
• which datasets or sites were genuinely separate; and
• which sources of variation were deliberate rather than incidental.
Current R&DTI record-keeping guidance states that records should be created at the time the activity is conducted.
Records should show what was done, how the work was performed, when it occurred, who carried it out and the resources used.
Examples of potentially relevant records include experimental plans, records of trial runs and test reports showing results and evaluation.
At a broader research-data level, the Australian Research Council's Research Data Framework highlights reproducibility, provenance and attribution as important benefits of improved research-data practices.
For commercial experimental work, one practical application of provenance is retaining enough information to identify which observations came from the same experimental unit.
That distinction can materially change what the next test should look like.
Before approving another round of testing, a CTO or engineering lead can ask:
Do we need another measurement, or another independent experimental unit?
If the unresolved uncertainty is manufacturing variability, another hundred measurements from the same system may add precision without adding the independent replication the decision requires.
If the unresolved issue is short-term stability within that same system, those hundred measurements may be exactly the right evidence.
The same number of measurements can therefore have very different evidentiary value depending on where the uncertainty exists.
Conclusion
More measurements are not necessarily more independent evidence.
Repeated observations from one experimental unit can improve precision and reveal variation within that unit, but they do not automatically demonstrate that the result will recur across independently produced specimens, batches, systems, sites or runs.
A strong experimental design identifies where the uncertainty enters the system and places replication at the level needed to test that uncertainty.
The useful question is not simply:
How much data have we collected?
It is:
What has actually been repeated independently?
Sources & Further Reading
Australian Government — Conducting core R&D activities for the R&D Tax Incentive
Australian Government — Record keeping for the R&D Tax Incentive
NIST/SEMATECH — e-Handbook of Statistical Methods: Design of Experiments Terminology
Talk to Ignition Research if your project is generating large volumes of test data but it is unclear whether you have repeated the measurement, the specimen, the production run or the experiment itself. As a Registered Research Service Provider, Ignition Research assists businesses with research question definition, experimental design, technical investigations and contemporaneous supporting records within its registered research fields.
RSP registration confirms that the provider has met the relevant registration requirements for its registered research fields. It does not constitute government endorsement of a project or guarantee R&DTI eligibility.
We do not determine R&DTI eligibility or provide tax advice. The R&D Tax Incentive is a self-assessment program, and companies remain responsible for assessing their activities, expenditure and circumstances and obtaining appropriate tax advice.
This article provides general information only and is not tax, legal or financial advice.
Working on a project like this?
Tell us what you're building and the part you're not sure will work. A researcher from our Adelaide team (registered Research Service Provider RSP000047) reads every enquiry and replies within two business days. No documents needed.
