Developing a super landmark, multivariate prediction model to predict early warning score alerts in hospital inpatients: a proof-of-concept study (Preprint)

Source abstract

BACKGROUND Wearable vital signs sensors (WVSS) could improve outcomes for hospitalised patients through improved detection of physiological deterioration but the best way to utilise data recorded by WVSS remains unclear. Many systems propose novel alerts which seek to replace or complement established systems such as the National Early Warning Score 2 (NEWS2). OBJECTIVE Develop a clinical prediction tool which predicts NEWS2 5+ alerts recorded by nurses using WVSS data from adult inpatients in the United Kingdom. METHODS Data was collected from 57 patients participating in two clinical studies of WVSS: 44 with Covid-19 and 13 with haematological malignancy receiving chimeric antigen receptor T-cell therapy (CAR-T). Patients wore three WVSS measuring heart rate/respiratory rate (HR/RR), temperature and oxygen saturations (SpO2) every minute. Contemporaneous vitals were recorded by nurses. A mixed-effects, Cox proportional hazards (CPH) model was developed using a stacked, super landmark dataset derived from vitals recorded during the first 49 hours of WVSS application. This included a 1-hour, run-in period to establish a baseline. The dataset was divided into 1-hourly landmark times followed by 1-hour-long prediction horizons. The outcome of interest was a National Early Warning Score 2 (NEWS2) 5+ alert from nurse-recorded vitals during each horizon. The CPH model was trained using a “ground truth” dataset. This comprised all landmark times when an outcome event was known and when WVSS-derived predictors were available in the preceding hour. The model was internally validated using patient-level bootstrap sampling (200 samples). Calibration was inspected visually and discrimination assessed using Harrell’s C-index. Rates of NEWS2 5+ alerts from nurse-recorded vitals in each horizon were compared with alerts predicted by the model. They were also compared with alerts expected if WVSS vitals were used to determine NEWS2 directly. This analysis was completed for the ground truth data and for the entire super landmark dataset after WVSS data gaps were filled by multiple imputation. RESULTS The super landmark dataset contained 2577 hourly landmarks. In 425/2577, the outcome (NEWS2 alert during horizon) and the predictors (WVSS data from the preceding hour) were known. This ground truth subset contained 129 NEWS2 5+ alerts and the model predicted 111/129 (sensitivity: 86.0%) with a specificity of 240/296 (81.1%). The optimism corrected C-index was 0.773 (95% CI: 0.705 to 0.833). In the ground truth subset, the nurse-recorded alert rate was 129/425 (30.4%), the model alert rate was 167/425 (39.3%), and the wearable rate was 373/396 (94.2%). In the imputed, super landmark dataset, the nurse-recorded alert rate was 191/693 (27.6%), the model alert rate was 1015/2577 (39.4 [95% CI: 37.4-41.4]%), and wearable rate was 2478/2577 (96.2 [95% CI: 95.3-96.9]%). CONCLUSIONS This proof-of-concept study suggests that prediction models may help reduce alert rates from WVSS. It illustrates a route by which such models could help to standardise alerting, enabling nurse-recorded and wearable vitals to contribute to the same alerting system. CLINICALTRIAL The COSMIC-19 study was registered with clinicaltrials.gov (registration: NCT04581031) and was approved by Yorkshire and the Humber – Bradford Leeds Research Ethics Committee, UK (reference 20/YH/0156). The EMBRaCE-GM study was registered with clinicaltrials.gov (registration: NCT05099237) and was approved by the Wales 6 Research Ethics Committee, UK (reference 21/WA/0251).