Quick answer: Matching observed data does not necessarily establish that a technical model represents the underlying physical mechanism correctly. Different combinations of parameters, or different model structures, can sometimes produce similar results while making substantially different predictions under new conditions. When that distinction matters to an engineering decision, researchers may need an experiment specifically designed to distinguish between the competing explanations.
A Good Fit Does Not Always Identify the Mechanism
A company is developing a thermal management system for an industrial energy storage product.
The engineering team builds a computational model to predict how the system responds to changing operating loads.
The model is calibrated against measurements collected during controlled testing. Its predicted temperatures closely match the observations.
The result appears to provide a sound basis for the next development decision.
However, another model, based on different assumptions about internal heat transfer, produces a similarly close match.
Both models reproduce the existing measurements.
Yet when the engineers simulate a more demanding operating cycle, their predictions diverge.
One suggests that the proposed design can maintain acceptable temperatures.
The other indicates that additional cooling capacity may be required.
The difficulty is not necessarily that either model has been implemented incorrectly.
The available observations may be insufficient to distinguish between the mechanisms represented by the models.
In scientific modelling, this problem is associated with identifiability.
Parameter identifiability concerns whether the available evidence can distinguish a model parameter from alternative values within the specified model structure. More broadly, researchers may struggle to distinguish between competing model structures that produce similar observable behaviour.
These are related but different problems.
A model's parameters may be difficult to estimate from the available observations. Separately, the mathematical representation itself may not adequately capture the physical processes governing the system.
A 2021 study co-authored by a CSIRO researcher examined parameter identifiability in a soil nitrogen model. The researchers found that correlated parameters could produce similarly good fits to measured data, making it difficult to determine their individual values from those observations alone. The study highlighted the need for additional measurements that could independently inform at least one of the parameters. Vilas et al., Considering Unknown Uncertainty in Imperfect Models
For an engineering business, this matters because a model that reproduces known behaviour may still provide unreliable guidance about conditions it has not encountered.
Calibration establishes agreement with selected observations.
It does not automatically establish that the model's parameters are uniquely identifiable or that its internal explanation is correct.
Why More Data May Not Resolve the Problem
When competing models produce similar results, the intuitive response is often to collect more measurements.
That can be useful, but only if the additional observations contain information capable of distinguishing the alternatives.
Consider the thermal management example.
Suppose the available measurements were collected while the energy storage system operated under relatively stable conditions.
One model attributes the observed temperature behaviour primarily to thermal mass.
Another attributes similar behaviour to heat-transfer characteristics.
Both models fit the available data.
Collecting more temperature observations under nearly identical operating conditions could improve the precision of the comparison without resolving the underlying ambiguity.
The additional data may confirm that both models reproduce the same behaviour.
It may not establish which explanation is better supported.
Distinguish Measurement Uncertainty From Model Uncertainty
Measurement uncertainty concerns the reliability and precision of the observations.
Model uncertainty concerns the assumptions, structure and parameters used to represent the physical system.
These sources of uncertainty should not be treated as interchangeable.
If the measuring instruments are insufficiently precise, improving the measurement process may be necessary.
If different parameter combinations remain consistent with the available observations, more precise measurements of the same behaviour may still leave the problem unresolved.
Identify Parameters That Compensate for One Another
Some models contain parameters that can produce similar effects on the measured output.
An increase in one parameter may be offset by a change in another.
For example, different assumptions about thermal conductivity and contact resistance may produce similar temperature predictions under certain conditions.
This does not necessarily mean that either parameter has been estimated reliably.
The research team should investigate whether plausible combinations of parameter values produce similarly acceptable fits.
Sensitivity analysis, parameter estimation and uncertainty analysis may help identify these relationships.
However, analytical methods cannot create information that the experiment did not collect.
If the observations contain insufficient information to distinguish the parameters, the experimental design may need to change.
Design an Experiment That Separates the Explanations
A useful next experiment should be selected because the competing models predict meaningfully different outcomes under its conditions.
This is different from conducting another demonstration that both models can reproduce.
In the thermal management example, the researchers might identify operating conditions under which thermal mass and heat-transfer behaviour influence the temperature response differently.
They could then design a controlled experiment to examine the system's transient response under those conditions.
The purpose would be to determine whether the observed behaviour is more consistent with one proposed mechanism than another.
Choose Conditions That Expose Meaningful Differences
The experimental conditions should follow the technical question.
For example, a carefully controlled change in operating load may reveal behaviour that remained hidden during steady operation.
Alternatively, measurements at additional physical locations may provide information about internal temperature distribution.
The research team should determine which observations are most capable of distinguishing the competing explanations.
This requires consideration of experimental feasibility, measurement reliability and relevant operating constraints.
A more demanding test is not automatically a more informative test.
The selected conditions must address the source of uncertainty.
Evaluate the Models Against Evidence That Can Challenge Their Predictions
Researchers should avoid assessing competing explanations exclusively against observations already used to construct or calibrate them.
Where practical, the conditions and evaluation measures for a new experiment should be defined before the observations are collected.
The resulting evidence can then be used to assess how well the competing models predict behaviour they were not calibrated against.
This does not guarantee that the better-performing model represents the underlying physics correctly. It provides a stronger basis for comparing their predictive performance under the conditions examined.
CSIRO's research on subsurface modelling includes comparisons between model predictions and physical experimental observations, illustrating the role of independent evidence in evaluating model performance. CSIRO — Modelling Comparisons
Allow for an Inconclusive Result
A discriminating experiment may still fail to distinguish the proposed mechanisms.
The observed difference could be too small, measurement uncertainty could remain substantial, or the models could require further development.
That is not necessarily a failed research process.
It identifies a limitation in the available evidence.
The researchers should record which explanations remain plausible and whether another experiment could materially improve the understanding needed for the engineering decision.
From Model Uncertainty to an Engineering Decision
Resolving every parameter-identifiability problem is not always necessary before proceeding.
The importance of the uncertainty depends on what the model is being used to decide.
If different plausible models produce sufficiently similar predictions within the intended operating range, the remaining uncertainty may have limited influence on the immediate engineering decision.
However, if the models lead to different conclusions about equipment capacity, durability or reliability, the distinction may be commercially significant.
Management should understand whether the available evidence supports proceeding with the current design, conducting further experiments or limiting the model's application to a narrower range of conditions.
A model should not be described as validated for conditions that the available evidence has not adequately addressed.
For projects involving the Australian R&D Tax Incentive (R&DTI), model uncertainty or parameter identifiability does not itself establish eligibility.
Under section 355-25 of the Income Tax Assessment Act 1997, a core R&D activity must involve an outcome that cannot be known or determined in advance on the basis of current knowledge, information or experience, but can only be determined through a systematic progression of work based on principles of established science, proceeding from hypothesis to experiment, observation and evaluation and leading to logical conclusions. The activity must also be conducted for the purpose of generating new knowledge.
Routine calibration, established simulation methods and ordinary engineering analysis do not automatically constitute core R&D activities. The actual activities must be assessed against all applicable requirements.
Current Australian Government record-keeping guidance states that records should be created at the time the activity is conducted. For model-based investigations, a practical research record may identify the competing hypotheses, model structures, parameter assumptions, experimental conditions, observations and reasoning behind the conclusions. These are practical documentation recommendations, not a prescribed R&DTI template.
For engineering leaders, the objective is to establish whether the evidence is sufficient to rely on the model for the intended technical decision.
When several explanations remain consistent with the observations, the most valuable next experiment may be the one designed to make their predictions diverge.
Conclusion
A good fit to existing data does not necessarily identify the physical mechanism that governs a system.
When different parameter combinations or model structures produce similar results, collecting more observations under the same conditions may not resolve the ambiguity.
The more useful approach is to identify where the competing explanations differ and design an experiment that can test that difference.
The objective is not to prove that one model is universally correct.
It is to determine which explanation is better supported by the available evidence and whether that evidence is sufficient for the engineering decision at hand.
Sources & Further Reading
Vilas et al. — Considering Unknown Uncertainty in Imperfect Models: Nitrogen Mineralization as a Case Study
CSIRO — Modelling Comparisons, Underground Storage Characterisation
Australian Government — Conducting Core R&D Activities for the R&D Tax Incentive
Australian Government — Record Keeping for the R&D Tax Incentive
Talk to Ignition Research if your engineering or computational research produces competing models that fit the same observations but predict different outcomes under relevant operating conditions. As a Registered Research Service Provider, Ignition Research assists businesses with research question definition, experimental design, technical investigations and contemporaneous supporting records within its registered research fields.
RSP registration confirms that the provider has met the relevant registration requirements for its registered research fields. It does not constitute government endorsement of a project or guarantee R&DTI eligibility.
We do not determine R&DTI eligibility or provide tax advice. The R&D Tax Incentive is a self-assessment program, and companies remain responsible for assessing their activities, expenditure and circumstances and obtaining appropriate tax advice.
This article provides general information only and is not tax, legal or financial advice.
Working on a project like this?
Tell us what you're building and the part you're not sure will work. A researcher from our Adelaide team (registered Research Service Provider RSP000047) reads every enquiry and replies within two business days. No documents needed.
