Forecasting Emergency Department Crowding: A Prospective, Real-time Evaluation

Introduction

The Emergency Medical Treatment and Active Labor Act (EMTALA),1 passed in 1986 by the United States Congress as part of the Consolidated Omnibus Budget Reconciliation Act, mandates that all patients presenting to an emergency department (ED) must be screened and stabilized, regardless of medical condition or ability to pay. Emergency medical services are the only form of health care in the United States legally guaranteed to be accessible, providing an important safety net.2 3 A seemingly clairvoyant article published in 1958 recommended that the number of EDs should increase in preparation for future demand.4 Unfortunately, between 1990 and 2005 the annual number of ED visits increased from 87 million to 115 million, while the number of EDs decreased from 5,172 to 4,611.5 Due to the growing problem of crowding, the Institute of Medicine found in 2006 that American EDs are “at the breaking point.”6 Prior research has shown that the consequences of ED crowding, which include delayed treatment,7 8 patient elopement,9 10 prolonged transport,11 12 increased mortality,13 14 and financial losses,15 16 are numerous and affect the entire health care system.

Background

The ability to forecast near-future ED crowding should enable new strategies of coping with the problem. For instance, a tool that accurately describes operating conditions several hours into the future might allow for just-in-time dynamic resource mobilization or coordination of primary care, hospital, and ED processes. Several measures of ED crowding have been proposed, including the Emergency Department Work Index (EDWIN),17 the Real-time Emergency Analysis of Demand Indicators (READI),18 the National Emergency Department Overcrowding Scale (NEDOCS),19 the Emergency Department Crowding Scale (EDCS),20 and the Work Score.21 These measures can accurately reflect present ED crowding, although they do not attempt to forecast future ED crowding.22 23

Reports in the literature have described efforts to forecast ED crowding using techniques such as time series regression,24 deterministic modeling by differential equations,25 and discrete event simulation.26 In the latter, we described the development and retrospective validation of ForecastED, a simulation-based tool to forecast near-future ED crowding.26 We based the decision to use discrete event simulation on the following rationale: Crowding is a complex phenomenon that can be summarized by numerous different measures,27 28 such as the number of waiting patients, boarding patients, or occupied beds. Most forecasting techniques require the investigator to select a dependent variable before model development. By contrast, a discrete event simulation can output a detailed list of patients projected to be in the ED in the future, and from this information the forecasts of many different outcome measures can be derived.

Research Questions

To our knowledge, no previous report has demonstrated the application of an instrument to forecast ED crowding in real time. This research gap must be addressed before a forecasting system can achieve practical value. We sought to answer two questions: First, can the ForecastED tool provide real-time crowding forecasts using data from existing ED information systems? Second, how well can it predict near-future values of several crowding measures in a live, operational setting?

Methods

Design, Setting, and Participants

We prospectively validated the ForecastED tool during a three-month period (5/1/07–8/1/07). The study did not involve any direct patient contact, and the local Institutional Review Board approved the study by expedited review.

We conducted the study in the adult ED of a tertiary-care, urban, academic Level 1 trauma center. The adult ED cares for more than 50,000 patients annually. It contains 41 licensed, monitored beds, four of which are designated for trauma patients. In addition, four fast-track beds are available for patients with minor complaints from 11 am to 11 pm, and eight dedicated rooms are available for patients with psychiatric complaints. Patients are triaged by the Emergency Severity Index (ESI), an ordinal score ranging from one, for the most urgent patients, to five, for the least urgent patients.29 Institutional policy allows for ambulance diversion if any of the following criteria are true, and are not expected to improve within 1 hour: “(1) all critical care beds in the ED are occupied, patients are occupying hallway spaces, and at least 10 patients are waiting; (2) an acuity level exists that places additional patients at risk; or (3) all monitored beds within the ED are full.” In practice, the administrator on duty initiates ambulance diversion when all licensed beds are occupied and at least 10 patients are in the waiting room. The ED staff were blinded to the study to avoid one potential source of bias.

We included data from all patients who received care in the adult ED during the validation period, with the following exceptions: We excluded visits involving purely psychiatric complaints because they are treated in a separate unit, and little crossover occurs between general-purpose and psychiatric beds. We excluded visits by patients who were dead on arrival to the ED, as well as visits by patients who were admitted directly to a hospital unit without receiving care in the ED, because these patients generally consume little ED resources at our institution. Because the study was conducted in real time, information that identified a patient for exclusion was not always available immediately. Such patients remained in the study until the time that information became available to exclude them.

System Architecture and Data Collection

The technical details and assumptions of the core ForecastED tool have been described in previous work.26 Briefly, the tool is a discrete event simulation that consists of several random number distributions that describe processes including patient arrivals, evaluation and treatment, and need for hospital admission.30 These distributions drive system behaviors including queuing in the waiting room and awaiting inpatient beds. Discrete event simulations of ED patient flow have been described previously in the literature, generally for evaluating proposed operational changes;31 32 33 34 the ForecastED tool differs substantially because it was designed for the specific purpose of real-time forecasting. The tool may be considered a computerized model of a virtual ED—patients have varying degrees of sickness, form queues in the waiting room, receive care in licensed beds—with the key difference being that time flows much faster in the virtual ED than in the actual ED. This property allows us to initialize the virtual ED based on the known state of the actual ED, rapidly move through several hours of simulated time, and obtain crowding measurements at a desired point in the future. The raw simulation output consists of a list of patient encounters projected to be ongoing in the future, together with information describing the course of each individual patient through the ED. This output may be translated into forecasts of future operational data, using the same technique that a list of current patient encounters can be summarized into present operational data. For example, to calculate the present waiting room count, one may total the number of patients in the physical waiting room of the ED; similarly, to forecast the future waiting room count, one may total the number of patients in the virtual waiting room of the simulation output.

The ED at the study institution is outfitted with a robust electronic whiteboard system, which has been previously described in the literature.35 36 The electronic whiteboard provides a central point of access to many different pieces of data that describe the minute by-minute operating status of the ED. Its roles include tracking patients, monitoring safety, and alerting personnel to important issues. Numerous aspects of ED patient data are made available in real time by Oracle databases (version 10g, http://www.oracle.com) underlying the electronic whiteboard system.

The core ForecastED tool was written in American National Standards Institute (ANSI) compliant C with no dependencies on external libraries; moreover, it is agnostic to the local ED information technology environment. To enable real-time forecasting of crowding, we developed a wrapper program using the Python programming language (version 2.3.5, http://www.python.org) that interfaces with the existing databases and automates the process of obtaining forecasts. The functions of this program were to obtain deidentified patient data from local ED information systems, to provide these data to ForecastED as input, to record the subsequent forecasts, and to display the output using the graphical web interface shown in Figure 1. The ED personnel did not have access to the interface, which was password-protected; this was intended to prevent the forecasting tool from influencing decision-making about ambulance diversion, as this would confound the results of the validation. Six patient-level variables were obtained from ED information systems when needed26: (1) time of initial registration at the ED, (2) time placed into an ED treatment bed, (3) time of hospital bed request if applicable, (4) time of discharge from the ED facility, (5) triage category assigned to the patient, and (6) whether the patient left without being seen.

The wrapper program performed two tasks at regular intervals: (1) Every day at midnight, the program used the most recent four weeks of historical patient data to estimate the parameters of each random number distribution needed for the simulation. This was intended to keep the simulated processes up to-date despite long-term fluctuations that might occur in patient demand or local workflow. (2) At every 10-minute interval, the program identified the set of patients presently in the ED and initialized the ForecastED tool accordingly. It used the mean of 1,000 simulation replications to obtain forecasts of several crowding measures at each hourly interval up to 8 hours into the future.37

Outcome Measures

We obtained forecasts of seven distinct crowding measures at 10-minute intervals: (1) waiting count, defined as the number of patients in the waiting room; (2) waiting time, defined as the average time since presentation among patients in the waiting room; (3) occupancy level, defined as the total number of patients in ED beds divided by the number of licensed treatment beds (this value may exceed 100% when patients are treated in nonlicensed areas such as hallway beds or chairs); (4) length of stay, defined as the average time since presentation among all patients in ED beds; (5) boarding count, defined as the number of patients awaiting hospital admission; (6) boarding time, defined as the average time since hospital bed request among patients awaiting hospital admission; and (7) probability of ambulance diversion, defined as a close approximation of the local diversion policy using the probability of having 10 or more patients in the waiting room and an occupancy level of at least 100%. These outcome measures were identical to those used in the preliminary, retrospective validation of ForecastED.26

We used the actual outcome measure, from the corresponding point in the future, as the reference standard for validating each forecast. At the time when each forecast was recorded, the value of the reference standard was not yet known. After the validation period ended, we obtained actual values of the six continuous outcome measures from information systems. The local aeromedical service, which maintains official records of diversion status independently of the ED, provided ambulance diversion log files.

Statistical Analysis

We validated the simulation forecasts of each outcome measure 2, 4, 6, and 8 hours into the future. We used the coefficient of determination (R²) to measure the reliability of the simulation forecasts for each continuous outcome measure with respect to the reference standard. This statistic describes the percentage of variation in the future outcome measures explained by the simulation forecasts. We calculated the R² with 95% confidence intervals (CI) using 250 iterations of the ordinary bootstrap method.38

The values of R² would not be affected by the calibration of the simulation forecasts;39 furthermore, because ForecastED was not fitted to predict any specific dependent variable in the least-square sense, it would have been erroneous to assume the residuals were centered around zero. We investigated the possible bias by calculating the mean and standard deviation of the residual forecasting error for each continuous outcome measure. A residual mean differing from zero, with respect to the standard deviation, would reveal the presence of a systematic bias in the forecasts.

We calculated the area under the receiver operating characteristic curve (AUC) to assess the discriminatory power in forecasting ambulance diversion status. This statistic summarizes overall discriminatory power for a binary outcome, where a value of 1.0 denotes perfect discrimination and a value of 0.5 denotes no discrimination.40 We calculated the AUC with 95% CI using 250 iterations of the ordinary bootstrap method.38 All statistical analyses were conducted using R (version 2.3.1, http://www.r-project.org).

Results

During the study period, a total of 13,239 10-minute intervals were observed out of a possible 13,248 (99.9%). Brief network downtimes accounted for the missed observations. A total of 14,448 visits by 11,539 unique patients occurred in the adult ED during the study period, of which 1,348 visits were excluded (9.3% total, 1.0% psychiatric, 0.0% dead on arrival, 8.3% immediately admitted to the hospital). Females represented 54.9% of the total patients, and the median age was 39 years. A total of 73.8% of the patients arrived by car, 17.1% by ambulance, 2.2% by helicopter, and 6.9% by other or unknown means. Hospital admissions resulted from 22.7% of the ED visits. A total of 77 ambulance diversion episodes, each lasting an average of 5.4 hours, occurred during the study period (18.8% of the total time).

The reliability of the simulation forecast for each continuous outcome measure is presented in Table 1. The forecasts explained more than 50% of the variation in the occupancy level, length of stay, boarding count, and boarding time up to 8 hours into the future. The percentage of future variation explained decreased as the length of the forecasting window increased. For example, the simulation forecasts of the occupancy level had R² values of 0.76, 0.67, 0.61, and 0.57, respectively, in predicting the actual occupancy level 2, 4, 6, and 8 hours into the future.

The calibration of the simulation forecast for each continuous outcome measure is presented in Table 2. The residual mean had small magnitude, relative to the standard deviation, for every outcome measure except the boarding time, suggesting that the forecasts were unbiased for most measures of crowding studied. The model consistently underestimated the boarding time 2, 4, 6, and 8 hours into the future, respectively, by −6.6 ± 2.7, −7.3 ± 3.1, −7.7 ± 3.3, and −7.8 ± 3.5, demonstrating a systematic bias for this outcome measure.

The receiver operating characteristic curves for discriminating future ambulance diversion status are presented in Figure 2. The AUC at 2, 4, 6, and 8 hours into the future, respectively, was 0.93 (95% CI: 0.93, 0.94), 0.90 (95% CI: 0.90, 0.91), 0.88 (95% CI: 0.87, 0.88), and 0.85 (95% CI: 0.84, 0.86), suggesting good discrimination. To illustrate the response of the ForecastED tool to periods of ED crowding, Figure 3 shows a time series plot of the 6-hour forecast probability of ambulance diversion, superimposed on episodes of ambulance diversion.

Discussion

We have successfully deployed the ForecastED tool in an adult ED. Three technical barriers existed with respect to the real-time implementation. First, the tool required the timely electronic availability of ED operational data. The study was conducted in a setting where an electronic whiteboard system was already accepted by faculty and staff,35 36 so this issue was considered resolved at the inception of the research. Second, a reliable, automated technique was needed to estimate parameters and obtain forecasts at scheduled intervals. We developed a Python wrapper program to address this need, and the results showed appropriate uptime statistics. Third, some mechanism was needed to instantly disseminate the forecasts to interested parties. To accomplish this, we created a graphical web interface as shown in Figure 1, which is updated with every observation.

The validation results demonstrate that the ForecastED tool can provide real-time, accurate predictions for a variety of ED crowding measures. Using information from past and present patients, the tool accurately predicted five out of seven outcome measures tested, up to 8 hours into the future. It fared less well in forecasting aspects related to the waiting room, which may suggest the crowding status in the waiting room is more volatile than the crowding status in other parts of the ED workflow. The forecasts of the waiting count and waiting time may be most useful when considered 4 hours or less into the future.

The results suggested that the forecasts for each continuous outcome measure were well calibrated for all outcome measures except for the boarding time, which showed evidence of a systematic bias. This result was consistent with our previous observations, and a likely mechanism exists within the simulation assumptions that govern allocation of inpatient hospital beds.26 This issue may be resolved by linearly calibrating the forecasts of boarding time according to the known bias in the residuals.

The results also indicated that the forecasts accurately predicted the probability of future ambulance diversion status up to 8 hours into the future. The time series plot shown in Figure 3 illustrates that the predictions closely fit the pattern of ambulance diversion in the study setting, with a peak in the signal associated with most diversion episodes. A sharp rise in the signal tends to precede the start of each diversion episode, while a sharp fall in the signal tends to precede the end of each diversion episode.

The screenshot in Figure 1 illustrates a unique capability of ForecastED: because a discrete event simulation model fits the pattern of individual patient flow, rather than a specific dependent variable, it can describe crowding from multiple input, throughput, and output perspectives.27 Thus, it not only warns that an ED will be crowded; it may also pinpoint why the ED will become crowded, which may be different at different times. For example, on a given day crowding may be attributed to a rapid inflow of patients that overload the waiting room, as shown in Figure 1. On another day crowding may be caused by large numbers of boarding patients, who remain in the ED for long periods while awaiting hospital admission.

In addition to the graphical web interface described above, other methods of distributing forecasting results are possible. The tool could interface with a pager system to alert on-call providers of severe crowding, for example, or it could interface with e-mail systems to distribute alerts. The proper method of disseminating warnings of crowding may vary between institutions depending on the local culture and preferences. Any operational change must be tuned to the organizational structure of an institution. Furthermore, some precedent exists for neighboring EDs to share operational data through regional networks,41 42 and this precedent could be applied to achieve local sharing of forecasting results. This would potentially enhance cooperation in areas with multiple busy tertiary care centers, particularly level 1 trauma centers.

One consideration during the prospective implementation and deployment of the ForecastED tool was how to clean patient data in real time. It is standard practice to identify and remove outlier data, when appropriate justification is provided.43 We justified the previously stated patient exclusion criteria on the grounds that not all patients recorded in the ED information system participate in the normal ED patient flow. The criteria would be straightforward to apply retrospectively, but not prospectively. For example, a patient with a purely psychiatric complaint would be identified and excluded at the beginning of a retrospective data analysis. However, a patient who is in the waiting room during the prospective study might not be identified, and hence excluded, until after the patient is placed into a dedicated psychiatric bed. Thus, the patient would affect the prospective validation until adequate information existed to mark him or her for exclusion. Although we were unable to eliminate this challenge, the results suggest this does not compromise the utility of the forecasting tool.

No changes should need to be made to the core ForecastED tool when transporting it to other institutions, because it was written in the standard C programming language with no dependencies on external software. The chief alteration required to deploy the tool at other institutions would involve changing the Python wrapper program that connects to ED information systems, since different institutions have different database storage schemes. Because the software automatically recalibrates the model parameters every day at midnight, no additional changes should be needed at other sites. Based on local preferences, the tool could also be adapted to forecast outcome measures aside from those used in our study. For example, other institutions are likely to have ambulance diversion criteria that differ from our study setting. No changes would need to be made to the simulation itself, and minor changes would need to be made to the code that processes the simulation output, to forecast ambulance diversion according to different criteria. The process of real-time system deployment should be repeatable in any ED that has six required patient-level variables available electronically.

Our study was limited in part because it took place over a four-month period in the adult ED of a single academic institution. Within our site seasonal patterns may exist that were not represented in the study period, which could lead the forecasting accuracy to deteriorate. Also, other hospitals may have differences in organizational structure or patient populations, leading to variable performance among sites. The present study does not allow for comment on these issues, so further validation at our institution and external sites will be needed to establish how these findings generalize with respect to time and place.

Our study was also limited in part because we made no intervention based on the tool. This allowed us to validate the forecasting accuracy in a live, operational setting; however, the goal of ForecastED is not merely to provide information, but to spur action based on this information to alleviate ED crowding. The question of how interventions triggered by the forecasts would directly impact patient care remains a valuable topic for further research. Few reports have discussed the prospect of allocating personnel and beds on demand,44 perhaps because the technology to determine when to mobilize such resources has generally been unavailable. We conclude the discussion below with two scenarios whereby the early warning provided by the ForecastED tool could help alleviate the negative effects of ED crowding.

Our first conjectural scenario considers the possibility of flexible staffing, which might be facilitated by operational forecasts. Nurses are often understaffed and overworked, with ED nurses caring for more than 5 patients simultaneously in some settings.45 Administrators plan staffing schedules based on long-term daily and weekly trends in patient volume, with the intent of maintaining a patient-to-nurse ratio near 4:1. Short-term fluctuations in ED operations render this difficult, however. This problem might be addressed by scheduling more nurses to staff the ED than are needed during average operating conditions; subsequently, when the ForecastED tool anticipates periods of lessened need, nurses could be given the option of ending a shift early. This strategy avoids the potential morale impact of mobilizing off-duty staff to achieve flexible staffing, while mitigating the additional costs of keeping the ED overstaffed at all times. Using this mechanism, the staffing level could be tailored dynamically as needed.

Our second conjectural scenario targets the widespread problem of ED boarding, which is the practice of retaining patients admitted to the hospital in ED beds for extended periods until an inpatient bed becomes available.46 This temporarily reduces the effective capacity of an ED, because the space where new patients could be treated remains occupied. When anticipating the daily demand for inpatient beds, hospital administrators must consider factors like transfers from other hospitals and operating room schedules. They must also consider hospital admissions originating via the ED, and forecasts generated by our tool could assist the accuracy of this process. In this way, the ForecastED tool might improve daily discharge planning, facilitating coordination between hospital services.

Conclusions

In summary, we have deployed and prospectively validated the ForecastED tool, which provides potentially useful forecasts of various ED crowding measures up to 8 hours into the future. In keeping with the principle “you can't manage what you can't measure”, this may enable new, proactive strategies for coping with the ED crowding problem. This work may provide a means of protecting and strengthening the fragile safety net of the health care system.