Quick answer: Not as a fully independent test of generalisation. If results from a designated test set influence model selection, feature choices, parameter settings or repeated refinements, that dataset has effectively become part of the development process. The resulting performance may still be useful, but it should not be presented as an independent estimate of performance on genuinely unseen data. A protected evaluation set or another appropriately designed evaluation procedure may be needed before the business relies on the result.
A Test Set Can Become Part of Development Without Anyone Intending It
A software research team develops a machine-learning model to predict equipment maintenance requirements.
The technical objective is clear: determine whether a new modelling approach can identify emerging equipment problems more reliably than the existing method.
The team assembles historical operating data, trains several candidate models and evaluates their performance.
The first results are promising.
However, the engineers notice that the proposed model performs poorly on certain equipment configurations.
They adjust the features, change the model architecture and run the evaluation again.
Performance improves.
The process continues until the team identifies a configuration that produces a satisfactory result.
From a development perspective, this looks like ordinary technical progress.
From an experimental perspective, an important question has emerged:
Was the test dataset used only to evaluate the selected model, or did its results help determine which model was developed?
Training, validation and final evaluation serve different purposes.
Training data is used to fit the model.
Validation data may be used to compare candidate approaches, tune parameters and guide development decisions.
An independent test set is intended to provide evidence about how the selected model performs on data that did not influence those development decisions.
If engineers repeatedly modify the system in response to test-set performance, information from that evaluation has fed back into development.
The model does not need to receive the raw test records directly for this to occur.
The development team's decisions can carry information from the evaluation results back into feature selection, model structure or parameter choices.
At that point, the designated test set has effectively taken on a validation role.
The result may still tell the team something useful about the development process.
It no longer provides the same degree of independent evidence about performance on unseen data.
Research from Monash University and the University of New South Wales on machine-learning forecast evaluation discusses how flawed data partitioning, leakage and evaluation practices can produce misleading conclusions about predictive performance. The paper focuses particularly on time-series forecasting, but the distinction between development evidence and independent evaluation is relevant more broadly to predictive modelling. Monash Research Management
The problem is not iterative development.
Iterative development is often exactly what the research requires.
The important distinction is:
Evidence used to develop a solution should not automatically be treated as independent evidence of how well that solution generalises.
Data Leakage Can Occur Before the Model Is Trained
Repeatedly consulting a test set is not the only way evaluation evidence can become compromised.
Information can also enter the development pipeline through data preparation, feature construction or inappropriate dataset boundaries.
This is commonly described as data leakage.
In this context, data leakage means that information unavailable at the intended prediction point is nevertheless used, directly or indirectly, during model development.
It is not the same thing as a privacy or cybersecurity data breach.
Consider the maintenance-prediction example.
The company has collected sensor observations from multiple machines over extended operating periods.
The engineers randomly divide individual observations into training and testing datasets.
However, observations from the same machine and the same operating episode appear in both groups.
The model may then learn characteristics specific to those machines or episodes rather than a relationship that transfers reliably to independent equipment.
A strong test score may therefore provide weak evidence about performance on a newly installed machine.
The appropriate dataset boundary depends on the question the model is meant to answer.
If the intended use is to predict future behaviour for equipment already represented during development, a carefully structured time-based evaluation may be appropriate.
If the intended use is to predict behaviour on previously unseen machines, separating development and evaluation data by equipment identity may provide a more relevant test.
The evaluation design should follow the intended claim.
Preprocessing Can Introduce Future Information
Leakage can occur even before model fitting begins.
Suppose a team calculates normalisation parameters using the complete dataset and only then divides the transformed data into development and evaluation sets.
Information from the eventual evaluation period may already have influenced the transformation applied to the development data.
For forecasting problems, the issue can be particularly subtle.
A feature calculated using observations from an entire operating period may include information that would not have been available when the prediction was supposed to be made.
The resulting model can appear more accurate than a model operating under genuine real-time conditions.
The Monash and UNSW forecast-evaluation research gives specific examples of leakage arising when smoothing, decomposition or normalisation is performed across an entire time series before the training and test periods are separated. It also notes that features can inadvertently contain future information that would not have been available at the actual forecast point. Monash Research Management
A useful safeguard is therefore to design the entire evaluation pipeline around the information that would genuinely have been available at the relevant prediction point.
That means considering:
• when observations become available;
• how features are generated;
• when preprocessing parameters are fitted; and
• whether any part of the development pipeline has access to information from the period or population being used for independent evaluation.
The objective is not simply to keep one spreadsheet labelled “test” separate.
It is to keep the information needed for independent evaluation out of the development decisions being assessed.
A Protected Test Set Still Needs to Test the Right Thing
Keeping a final evaluation set untouched during development can strengthen the independence of the performance estimate.
But separation alone is not enough.
The data must also represent the conditions about which the team intends to draw conclusions.
An independent dataset containing only ordinary operating conditions may tell the business little about performance during unusual equipment behaviour.
A test set drawn from the same machines used during development may not answer a question about generalisation to newly installed equipment.
A random observation-level split may also be inappropriate where strong temporal, machine-level or event-level dependencies exist.
Where available data is limited, cross-validation or other structured validation procedures may help use the data efficiently.
For some problems, nested evaluation procedures can help separate model selection from performance estimation.
No single method is appropriate for every project.
The key principle is simpler:
The evaluation design should preserve a meaningful distinction between developing the model and obtaining evidence about the claim the business intends to make about it.
Record When Evaluation Evidence Enters Development
A final performance report may contain accuracy measures, error distributions and comparisons with an existing approach.
Those numbers are difficult to interpret without knowing how the model and the evaluation data were selected.
As a practical traceability approach, the research record should make clear:
• the relevant data sources and dataset versions;
• how development and evaluation data were partitioned;
• which preprocessing steps were fitted on which data;
• which datasets were used to guide model selection or refinement;
• which datasets remained independent;
• when evaluation results were first examined; and
• whether the model was changed after those results were known.
This is a practical experimental-record framework rather than a prescribed R&DTI template.
If a model was modified after test-set results were examined, that does not mean the development work becomes invalid.
It changes what can reasonably be claimed from that evaluation.
For example, a team may discover that the original data preparation included information that would not have been available during real operation.
The team corrects the pipeline and repeats the investigation.
The earlier result remains part of the research history.
It should not, however, be presented as independent evidence of predictive performance under the corrected conditions.
Where genuinely independent test evidence is no longer available, the team may need additional representative data or another defensible evaluation strategy before making a stronger generalisation claim.
Current Australian Government software guidance provides examples of records that may support software-related R&D activities, including investigation tasks, design notes, architecture decision records, test plans and outputs, logs, dashboards and performance graphs. The guidance emphasises that it is the information in the records, rather than a prescribed document format, that matters. business.gov.au
Current general R&DTI guidance also states that records should be created at the time the activity is conducted. business.gov.au
Good Evaluation Practice Does Not by Itself Create an R&DTI Activity
Rigorous dataset management, leakage prevention and independent testing are good experimental practices.
They do not, by themselves, make model development eligible for the Australian R&D Tax Incentive.
Current Australian Government guidance requires a core R&D activity to be conducted for the purpose of generating new knowledge and to involve an outcome that cannot be known or determined in advance by a competent professional on the basis of knowledge, information or experience that is publicly available or reasonably accessible.
The outcome must be determined through a systematic progression of work. business.gov.au
The current Software Development sector guide similarly distinguishes genuine technical hurdles requiring hypothesis-driven experimentation from routine software work that applies established tools, methods or techniques. business.gov.au
The newer AI-specific guidance makes the same boundary particularly clear for AI activities: using AI does not itself establish R&D, and routine data preparation, performance checks, established testing or parameter adjustment where the effects are already understood are generally unlikely to constitute core R&D activities. business.gov.au
For this reason, correcting an ordinary implementation error or applying an established validation method should not be characterised as R&D merely because it improves the credibility of the model evaluation.
Where a genuine technical hurdle remains and the outcome can only be resolved through hypothesis-driven experimentation, the model-development and evaluation activities should be assessed according to the actual work undertaken.
An Independent Evaluation Should Be Capable of Challenging the Model
For management, the important question is not whether the final performance number looks strong.
It is whether the evidence used to produce that number was sufficiently independent of the decisions that produced the model.
A model that performs well on a dataset repeatedly consulted during development may still be technically promising.
But that result is not equivalent to independent confirmation of performance.
Depending on the intended claim, the next evaluation may require data collected after model selection, observations from separate equipment, a genuinely protected holdout population or another design that prevents the development process from adapting to the evaluation conditions.
If performance remains satisfactory, the business has stronger evidence for the next engineering or deployment decision.
If performance deteriorates, that difference is also informative.
It may show that the earlier result reflected adaptation to a particular dataset, machine population or evaluation procedure rather than the broader technical relationship the team intended to establish.
That outcome may justify a revised hypothesis, a different modelling approach or a narrower claim about where the model works.
The purpose of independent evaluation is therefore not to confirm the development team's preferred result.
It is to provide evidence capable of challenging it.
Conclusion
A dataset does not remain an independent test set simply because it is labelled one.
Once its results begin influencing feature choices, model architecture, parameter settings or other development decisions, the evidence has entered the development process.
The same problem can occur indirectly through preprocessing, inappropriate data boundaries or information from the future.
A credible evaluation therefore needs two things:
independence from the decisions that produced the model, and relevance to the conditions about which the business wants to draw conclusions.
A strong score is useful.
A strong score on genuinely appropriate unseen evidence is much more informative.
Sources & Further Reading
Australian Government — Software Development Sector Guide for the R&D Tax Incentive
Australian Government — Artificial Intelligence-related activities and the R&D Tax Incentive
Australian Government — Conducting Core R&D Activities for the R&D Tax Incentive
Australian Government — Record Keeping for the R&D Tax Incentive
Talk to Ignition Research if your machine-learning research involves repeated model refinement, shared datasets or uncertainty about whether the reported performance still represents independent evaluation. As a Registered Research Service Provider, Ignition Research assists businesses with research question definition, experimental design, technical investigations and contemporaneous supporting records within its registered research fields.
RSP registration confirms that the provider has met the relevant registration requirements for its registered research fields. It does not constitute government endorsement of a project or guarantee R&DTI eligibility.
We do not determine R&DTI eligibility or provide tax advice. The R&D Tax Incentive is a self-assessment program, and companies remain responsible for assessing their activities, expenditure and circumstances and obtaining appropriate tax advice.
This article provides general information only and is not tax, legal or financial advice.
Working on a project like this?
Tell us what you're building and the part you're not sure will work. A researcher from our Adelaide team (registered Research Service Provider RSP000047) reads every enquiry and replies within two business days. No documents needed.
