Quick answer: Possibly, but the missing observations need to be understood before the results can be interpreted confidently. If a sensor stops recording during the conditions an experiment is intended to investigate, analysing only the available data may give a misleading picture of system performance. Researchers should establish why the observations are missing, whether the gaps are associated with the technical behaviour under investigation and what the remaining evidence can reasonably support.
A Missing Reading May Be Part of the Result
A company is developing an alternative environmental monitoring method for an industrial water-treatment system.
The research team wants to determine whether the proposed approach can identify short-duration changes in water quality under variable operating conditions.
Sensors are installed, the experimental protocol is established and observations begin.
After several weeks, the collected data suggest that the new method performs consistently under ordinary conditions.
However, closer examination reveals that one sensor stopped transmitting during periods of unusually high flow.
The missing observations introduce a problem that is not immediately visible in the overall performance results.
If the researchers analyse only periods with complete data, they may exclude precisely the conditions that the monitoring method was intended to address.
The remaining observations could therefore make the technology appear more reliable than the evidence actually demonstrates.
This is different from simply having fewer measurements.
A sensor failure may be associated with the operating condition itself.
A communications device may lose connectivity when environmental conditions deteriorate. A sensor may stop recording when a measurement exceeds its effective operating range. A data-processing system may fail to retain observations during periods of unusually high activity.
In such cases, the absence of data may contain information about when the experimental system becomes unreliable.
CSIRO has used high-frequency automatic water-quality sensor data in agricultural applications and has reported research into estimating missing observations and predicting water quality using artificial intelligence. 澳大利亚科学与工业研究组织
The important distinction is therefore between:
a gap that reduces the amount of evidence and a gap that systematically removes evidence from the conditions central to the hypothesis.
The second problem can materially affect what the experiment is capable of establishing.
Investigate the Missing Observations Before Filling the Gap
When observations disappear, the immediate response may be to repair the sensor, interpolate the missing values or remove incomplete records.
Those actions may be appropriate for routine data management.
They should not replace the question:
Why is the data missing, and does that reason affect the experiment?
Identify where the loss occurred
A missing value does not necessarily mean the physical sensor failed.
The interruption may have occurred during measurement, local storage, transmission, data processing or database ingestion.
NSW's OPENAIR guidance treats environmental sensing as an integrated system involving sensing devices, communications, software and platforms, databases and related digital services. It also provides dedicated troubleshooting guidance for sensing devices and network operations. 空气质量中心
For an experiment, the location of the failure matters.
If the sensor made a valid measurement but transmission failed, the original observation may still exist locally.
If the instrument itself stopped responding, the physical condition may never have been measured.
If the observation reached the platform but was rejected during processing, the problem lies elsewhere again.
Diagnostic logs, timestamps, equipment configurations and maintenance records may help distinguish these possibilities.
The objective is not simply to repair the monitoring system.
It is to understand what evidence was actually lost.
Ask whether the missing periods differ from the observed periods
Researchers should examine when the gaps occurred and compare those periods with other available information about the operating environment.
Were observations missing predominantly during high flows?
During elevated temperatures?
During equipment transitions?
Did another instrument record unusual conditions at the same time?
Did the same type of gap occur in the reference system?
These questions help determine whether the available observations are representative of the conditions the experiment was intended to investigate.
The Bureau of Meteorology provides a useful illustration of the broader issue. Its historical temperature guidance notes that gaps can arise from damaged instruments, station interruptions and automatic weather-station failures, and that changes in instrumentation or observing practices can also affect the comparability of long-term records. The Bureau of Meteorology
The same principle applies to experimental monitoring.
An incomplete dataset should not automatically be interpreted as though the observed periods represent the conditions that were not recorded.
Treat reconstructed values as estimates
Interpolation, model-based reconstruction and other gap-filling methods can sometimes provide useful estimates.
But an estimated observation is not the same as a measurement that was actually recorded.
Consider a sensor that stops transmitting during a rapid change in operating conditions.
A reconstruction based on measurements immediately before and after the gap might produce a smooth transition.
If the real system experienced a short-duration peak during the missing interval, that reconstructed series could conceal precisely the event the experiment was intended to detect.
The dataset might look complete while remaining unable to answer the original research question.
Measured and reconstructed observations should therefore remain distinguishable.
Researchers should also consider whether reasonable alternative assumptions about the missing interval materially change the conclusion.
If the answer depends heavily on how the gap is reconstructed, the evidence may support only a narrower finding.
Design the Next Experiment Around the Evidence That Is Missing
A trial containing missing data does not necessarily need to be discarded.
The appropriate next step depends on what the gap has left unresolved.
If the missing observations resulted from an identified equipment or communications fault that can be addressed using established methods, the team may correct the system and repeat the relevant part of the test.
That is different from discovering that the proposed sensing method itself cannot reliably operate under the conditions being investigated.
Suppose the technical question is whether a new sensing approach can maintain useful measurements during rapidly changing flow conditions.
If the instrument repeatedly fails during those conditions, the failure may itself be relevant to the technical question.
The team should not simply remove those periods and report performance from the easier operating range.
A subsequent experiment might use an appropriate independent measurement method to establish what occurred during the disputed periods.
The reference method should provide useful evidence about the condition being investigated rather than reproduce the same failure pathway.
The protocol can also establish in advance how gaps will be identified and how they will affect interpretation of the experiment.
For example, it may distinguish between an invalid reading, an absent reading and an observation that was successfully measured but not transmitted.
If repeated loss of data occurs under the condition central to the hypothesis, that may be an experimental finding rather than merely a data-cleaning problem.
Keep the R&DTI Question Focused on the Technical Activity
Missing observations do not themselves create an eligible R&D activity.
Routine monitoring, correction of a known communications problem or application of an established gap-filling method may simply be ordinary technical or data-management work.
Under the current Australian R&D Tax Incentive requirements, a core R&D activity must be conducted for the purpose of generating new knowledge.
Its outcome must be unable to be known or determined in advance by a competent professional on the basis of knowledge, information or experience that is publicly available or reasonably accessible anywhere in the world when the activity begins.
The outcome must then be determined through a systematic progression of work based on principles of established science, proceeding through hypothesis, experiment, observation, evaluation and logical conclusions. business.gov.au
Where an unresolved technical question genuinely remains—for example, whether a sensing approach can function reliably under a particular dynamic operating condition—the experiments investigating that question should be assessed against those requirements.
The existence of missing data, sophisticated reconstruction techniques or specialised monitoring equipment does not establish eligibility by itself.
The question remains what technical outcome was unknown and what experimental work was required to determine it.
Preserve the Original Evidence and the Gap
Current R&DTI record-keeping guidance states that records should be created at the time the activity is conducted.
Records should show relevant background research and details of what was done, how the work was performed, when it occurred and who carried it out. Experimental plans, records of trial runs and test reports showing results and evaluation are examples of records that may demonstrate the systematic progression of work. business.gov.au
For an experiment affected by missing observations, a practical record may identify:
• which observations are original measurements and which are reconstructed;
• when and under what conditions gaps occurred;
• where in the measurement or data pathway the failure occurred;
• what assumptions were used to estimate any missing values;
• what independent evidence was available for the missing periods; and
• whether alternative reasonable assumptions would change the conclusion.
This is a practical experimental-record framework rather than a prescribed R&DTI template.
Preserving the original data alongside corrected or reconstructed versions can also make it easier for another technically competent person to understand how the final conclusion was reached.
The objective is not to hide an incomplete experiment by producing a cleaner dataset.
It is to preserve the relationship between the evidence actually observed and the conclusion ultimately drawn.
What Can the Remaining Evidence Actually Support?
For engineering and investment decisions, the central question is whether the available observations cover the conditions in which the technology needs to perform.
A monitoring method may work consistently during ordinary conditions but remain untested during the high-flow events it was designed to detect.
That distinction should remain visible.
A defensible conclusion might be:
The monitoring method demonstrated consistent performance within the observed operating range, but reliability during high-flow events remains unresolved because measurements were unavailable during those periods.
That is more accurate than treating the observed portion of the dataset as evidence of performance across the complete operating range.
In some cases, independent records may provide enough information to evaluate the missing period without repeating the entire experiment.
In others, another experiment may be necessary.
Simply collecting data for longer is not necessarily the answer if the same measurement system continues to omit the conditions that matter most.
The next investigation may need a different sensor arrangement, an independent reference measurement or a revised acquisition method.
A dataset with fewer observations is not automatically weak evidence.
But a dataset that systematically loses observations under the conditions central to the hypothesis may support a much narrower conclusion than its overall performance figures suggest.
Conclusion
Missing data is not always just an inconvenience to be repaired.
Sometimes the pattern of missing observations tells the research team something important about the system being tested.
If measurements disappear randomly, the main effect may be a reduction in available evidence.
If they disappear specifically during the operating conditions the experiment was designed to investigate, the missingness can affect the validity of the conclusion itself.
The useful question is therefore not simply:
How should we fill the gap?
It is:
Why did the gap occur, and what does its location mean for the hypothesis we were trying to test?
Only after that question is understood should the team decide whether reconstruction, additional evidence or another experiment is appropriate.
Sources & Further Reading
CSIRO — Water-quality monitoring and missing-data estimation
CSIRO describes high-frequency automatic water-quality sensor data and research into estimating missing observations and predicting future water quality. CSIRO water-quality sensor research
NSW Government — OPENAIR Best Practice Guide
Australian Bureau of Meteorology — About Air Temperature Data
Australian Government — Conducting Core R&D Activities
Australian Government — Record Keeping for the R&D Tax Incentive
Talk to Ignition Research if your environmental monitoring, engineering or data-systems research relies on sensor observations that become incomplete under critical operating conditions and you need to determine what those gaps mean for the experiment. As a Registered Research Service Provider, Ignition Research assists businesses with research question definition, experimental design, technical investigations and contemporaneous supporting records within its registered research fields.
RSP registration confirms that the provider has met the relevant registration requirements for its registered research fields. It does not constitute government endorsement of a project or guarantee R&DTI eligibility.
We do not determine R&DTI eligibility or provide tax advice. The R&D Tax Incentive is a self-assessment program, and companies remain responsible for assessing their activities, expenditure and circumstances and obtaining appropriate tax advice.
This article provides general information only and is not tax, legal or financial advice.
Working on a project like this?
Tell us what you're building and the part you're not sure will work. A researcher from our Adelaide team (registered Research Service Provider RSP000047) reads every enquiry and replies within two business days. No documents needed.
