KPIN replaces Microsoft’s Language Understanding Intelligence System (LUIS) technology and the VSN Query Report system for the online component of SCPMG’s omnichannel solution for care navigation. See Fig. 1 for an overview of the omnichannel strategy; it illustrates how the organization manages multiple patient-facing channels, all of which are guided by common databases to ensure operational consistency. KPIN’s roll-out to 4.9 M SCPMG patients began October 1, 2024, supporting the patient portal’s appointment booking workflow. KPIN matches the patient’s reason for visit with appropriate care offerings, while further streamlining the user-friendly interface. The IVR channel, another mechanism for self-booking, continues to be supported by the VSN Query Report system; and the Agent channel remains open, where a robust customer relationship management tool supports the organization’s live agents.
KPIN supports the organization’s online channel, which is complemented by the Agent-assisted channel; while online self-service functionality encompasses an expansive set of options, live Agents are available telephonically (24 h a day, 7 days a week) to assist with care navigation. Integration with common databases ensures all channels adhere to consistent guidelines.
The patient portal explicitly advises patients not to seek emergency care through the online booking platform. It instructs them to call 911 or visit the nearest hospital for medical or mental health emergencies. These advisory statements appear on the interface before patients interact with KPIN. We recognize that patients may not adhere to such instructions, a point which underscores the importance of KPIN’s CAS.
Patient demographics
Table 1 summarizes the demographics of the unique 1,045,904 patients that interacted with KPIN from October 1, 2024, to March 1, 2025. 35.47% of patients fell in the 30–49 age bracket, which was the largest by volume. The 18–29 and 50–64 age brackets captured 17.08% and 16.73% of patients, respectively; and 12.52% were 65 or older, which suggests a narrowing of a digital divide impacting the elderly12. 59.93% of patients were female, 40.01% were male, and 0.06% identified as non-binary or preferred not to specify their gender. For ethnic groups, the Hispanic segment was the largest with 36.93%, followed by the Caucasian population at 30.04%, and other ethnic groups made up the remaining 33.03%.
Model Performance
The performance of KPIN’s models, including those supporting CAS and CNS, are robust across all pathways. Table 2 summarizes the performance metrics for the CAS models. For detecting high-acuity symptoms, such as chest pain, the accuracy is 96% (95% CI: 93.7–98.0%), with precision and recall of 97.5% (95% CI: 95.8–99.0%) and 96.0% (95% CI: 93.8–97.9%), respectively. The F1-Score is 96.7% (95% CI: 94.5–98.5%). Table 3 presents the performance metrics for the CNS models, which achieves an accuracy of 81.9% (95% CI: 80.0–83.6%), a precision of 85.6% (95% CI: 84.0–87.2%), a recall of 81.9% (95% CI: 80.1–83.7%), and an F1-Score of 82.8% (95% CI: 81.1–84.5%). Figure 2 illustrates the AUC values for the CAS models and the CNS models at 0.977 and 0.889, respectively.
The ROC curves illustrate the performance of CAS and CNS in distinguishing between relevant clinical outcomes. The area under the curve (AUC), a measure of overall model performance, is 0.977 (red solid line) for CAS and 0.889 (blue solid line) for CNS. The black dashed line represents the performance of a random classifier. Red Solid Line – CAS (AUC: 0.977). Blue Solid Line – CNS (AUC: 0.889). Black Dashed Line – Random Classifier.
The performance of the CAS models highlights their effectiveness in identifying potential critical cases and prompting patients to respond to questions before completing their booking. This high accuracy stems from rigorously defined critical case criteria, developed in collaboration with physician subject matter experts. This targeted approach aligns with literature emphasizing the importance of early identification of specific critical cases in such systems13.
In contrast, the CNS models are designed for broader generalization, trained to handle more diverse patient inputs and clinical pathways within the booking experience. This adaptability is crucial for maintaining real-world performance in care navigation on the digital channel14,15.
Utilization
From October 1, 2024, to March 1, 2025, KPIN facilitated 2,969,945 encounters. An encounter is defined as an interaction with KPIN, which begins when a patient enters and submits a ‘Reason for Visit,’’ as depicted in Supplementary Fig. 1. On average, KPIN processed 19,154 encounters per day, with the highest daily volume reaching 39,364 encounters. After submitting a ‘Reason for Visit,’’ a user may be presented with additional questions, to refine KPIN’s identification of the most appropriate care path, as illustrated in Supplementary Fig. 2 and Supplementary Fig. 3. An encounter is complete when KPIN produces its output, which is a set of recommended offerings.
The observed abandonment rate, which quantifies the extent of incomplete encounters, was 2.94% (IQR: 2.77–3.11%); this indicates that 97.06% of encounters resulted in recommendations for care. Under the LUIS system, each set of clarifying questions typically included 4–6 items, each displayed on a separate page. KPIN enhanced this workflow by limiting each page to three items or fewer and introducing a second page only when necessary to guide patients toward the appropriate clinical navigation pathway. This streamlined user interface was designed to reduce user abandonment rates.
KPIN integrates data (e.g., user age and gender) from multiple databases with the “Reason for Visit” input, to generate tailored clinical care offerings, which are safe and appropriate to book. A booking can be scheduling an appointment, or another clinically appropriate option (e.g., redirecting to the pharmacy page for medication refills, sending a secure message to a provider, going to a walk-in nurse clinic for a vaccination). Unlike most fee-for-service entities, our integrated value-based organization enables many reasons for visit to be addressed via options beyond scheduling an appointment; and this conserves access, yields efficiencies, and satisfies patients’ preference for convenience. We thus count the presentation of non-appointment offerings toward the successful booking rate, which is a novel metric arising from KPIN’s deployment. A successful booking is defined as a patient completing the KPIN process and either (1) being offered at least one appointment option and the patient selects one, or (2) being presented with an alternative, appropriate care option—such as sending a message—through which the patient’s reason for visit can be addressed. A non-successful booking is one where the patient is offered at least one appointment and does not book it.
The KPIN interface allows users to modify or re-enter their “Reason for Visit” at any stage, a behavior reflected in the data. Consequently, a single patient can engage in multiple duplicate encounters but ultimately settle on only one booking. We thus had to account for this type of behavior, which is typical of users in navigating web and app interfaces, in our assessment of the KPIN successful booking rate16. The adjusted successful booking rate, defined as the ratio of bookings within 10 min (numerator) to initial encounters (denominator), was 53.68%. The 10 min timeframe we applied to exclude duplicate encounters, which is the adjustment modifier, was determined through the “knee point” identification technique17, as detailed in the Methods section. See Fig. 3 for examples of how this method applies in various scenarios. Based on limited data from LUIS, the pre-KPIN successful booking rate was 34%.
Patient A engaged in three encounters, none of which is a duplicate, because all of the encounters occurred outside of the 10 min cut-off. Patient B engaged in three encounters, one of which is a duplicate. She booked two appointments, neither of which is a duplicate, as they were distinct appointment time slots. The third encounter is considered a duplicate because it occurred within 10 min of the second and was not associated with a booked appointment. Duplicates are excluded from the calculation of the adjusted successful booking rate.
While the adjusted successful booking rate of 53.68% may appear unremarkable, it underscores the value in maintaining the robust integrated booking ecosystem in which the organization has invested. In instances where digital booking does not address the needs of the patient, he or she can be directed to the agent-assisted channel where an agent can provide the additional needed support (24 h a day, 7 days a week). Supported by consistent guidelines applied across all channels, the channels can thus operate complementarily.
Patient experience
KPIN users with a registered email address or phone number were eligible to participate in a satisfaction survey, accessible within one day of their encounter. Between July 2024 and February 2025, spanning three full months before and after KPIN’s go-live, 114,396 surveys were distributed, with response rates of 2.61% and 2.77% for the pre- and post-go-live periods, respectively. Users were asked to indicate their level of agreement with specific statements on a 5-point scale, as illustrated in Supplementary Table 1, with responses subsequently converted to percentage scores.
Two-sample z-tests showed statistically significant improvements in post-KPIN mean scores for overall satisfaction, level of effort, and ease of navigation, which were higher by 2.56% (two-tailed p = 0.0021), 1.94% (two-tailed p = 0.0162), and 3.21% (two-tailed p = 0.000196), respectively. Additionally, the survey included a free-text feedback prompt: “Please tell us in your own words what we did well and if there are any areas where we could improve.” A total of 1531 (50.746%) surveys included patient comments. A sentiment analysis of these comments was conducted, classifying feedback as positive, negative, or neutral. Pre-KPIN comments were classified as 58.798% positive, 39.771% negative, and 1.431% neutral, while post-KPIN comments were classified as 67.428% positive, 30.409% negative, and 2.163% neutral. Thus, we observed an 8.63 percentage point increase in positive sentiment after KPIN deployment (two-proportion z-test; two-tailed p = 0.0005). The independence assumption for the z-test is met, as no participants were surveyed in both the pre- and post-KPIN groups. The 8.63 percentage point increase for positive sentiment highlights KPIN’s beneficial impact on users’ experience for the online channel.
link