Digital twins have a high adoption rate, with some great benefits reported. They are generally managed as an engineering project with a scope and timeframe for implementation. Unfortunately, little goes into the twin’s management tools after implementation, when the engineering team disband. Good results are often lost as processes drift or change, people are fed up with false alarms, and the twin needs help.
Peter Drucker, management consultant and author, wrote, “You can’t handle what you can’t measure.“This also applies to digital twins. Consider embracing the confusion matrix to monitor and manage the performance of your digital twins.
Measuring the performance of digital twins
The current state of digital twins lacks a key performance indicator (KPI) to manage their effectiveness. A review of the industry literature on digital twins found no KPIs or other measures of the effectiveness of the twin itself. Each digital twin affects the KPIs of what it models, but these are secondary. This report discusses KPIs for the performance of the digital twin.
Confusion matrix applied to digital twins
A confusion matrix provides a specific table layout for visualizing the performance of a digital twin algorithm. This includes twins using first-principle math, machine learning, or both. It measures alerts generated by the twin so that all stakeholders can easily interpret the veracity of the twin and respond appropriately.
The four-cell array contains true positives and negatives, and false positives and negatives. This matrix provides a dashboard for measuring false positives and false negatives (errors) by the digital twin. Performance is managed by monitoring false reads and continuously improving the twin by reducing them.
Performance digital dual confusion matrix
The most common twin is a digital performance twin for predictive maintenance (PdM) in the O&M portion of an asset’s lifecycle. PdM is used for the examples in this Insight. The concepts can be applied to other types of twins as they involve simulation for predictions that can be compared to actual conditions.
- True Positive (TP) occurs when an alert generated by the model is confirmed by a maintenance planner or technician as valid.
- True Negative (TN) does not have alerts because the PdM twin does not generate alerts when there is no indication of a problem.
- False positives (FP) apply to alerts for which no issues were found.
- False negatives (FN) record failures without a corresponding alert.
Integration with EAM system for data collection and reporting
For end users with internal maintenance personnel, these metrics can be obtained transparently. Automate data collection by integrating the digital twin with the enterprise asset management (EAM) system where technician work orders are processed and managed. First, the alerts generated by the PdM twin are transferred to the EAM system. The workflow includes the automatic creation of a maintenance work order for the maintenance planner to review, sort, approve, and schedule. Modern EAM systems have an application programming interface (API) for this function.
A dashboard can be created to display the confusion matrix for each digital twin – preferably associated with the EAM system where asset management KPIs are monitored. One approach would be to add a checkbox field in the work order to get the necessary data from the planner or technician. For those who only have a few digital twins, this checkbox is probably best done by the planner.
As the quantity and maturity of digital twins increases, this role can be transferred to technicians. With mobile devices, technicians can process work orders – including a checkbox for a false positive – while doing their jobs.
Modern EAM systems have a way of adding fields in work orders to collect necessary data from technicians and / or planner. These new systems also allow the creation of custom reports or dashboards to track the “confusion matrix” for a digital twin.
ARC Advisory Group clients can view the full report on the ARC Client Portal
If you would like to purchase this report or get information on how to become a customer, please contact us
Keywords: Digital Twin Performance, KPI, Confusion Matrix, Predictive Maintenance, ARC Advisory Group.
 The negative true cell may need to be left blank. Or consider a modification of the confusion matrix by substituting the failure rate (FR) before the twin, that is, the run time divided by the previous mean time to failure (MTBF). Then comparing TP to FR gives an indication of the deterioration of the asset i.e. a TP greater than FR is a bad trend. Please contact me if you have any comments or recommendation ([email protected]).