Hospital Pharmacist
Vol 11 No 7 p283-285
July/August 2004

Automated dispensing — how to evaluate its impact

By Cate Whittlesea, MRPharmS, PhD, Ceri Phillips, PhD, David Roberts, MRPharmS, Robin Burfield, Jeremy Savage, FRPharmS and Cheryl Way, MRPharmS

There is much anecdotal evidence to suggest that introducing automated dispensing helps pharmacy staff deliver better services to patients and other trust staff and can result in cost savings being made. For example, dispensing errors are said to be reduced and stock distribution is said to be improved. However, few scientific studies have been carried out to evaluate these issues and so a sound evidence base confirming (or indeed countering) these perceived advantages has yet to be built up.

This article sets out an overview of an “evaluation tool kit”, developed by pharmacists at three trusts and two universities in Wales in order to evaluate the impact of automated dispensing at their hospitals. It is hoped that it might be useful to others who want to carry out a similar exercise. The issues evaluated by the tool kit are set out in Panel 1 (below), with general advice and further information on particular aspects of the evaluation being detailed below.

General advice

Making the most of data that is already available is one of the best pieces of advice we can offer. Not only does this reduce the time spent developing new methods, but it helps the production of reliable results, because staff are familiar with the systems used. Panel 2 (right) shows the areas of the evaluation where existing data can be used (either “as is” or with minimal adaptation), areas where the tool kit uses existing data collection methods (even though new data itself will need to be collected) and areas where new methods have been developed for inclusion in the tool kit.

Much of the available data (particularly for looking at issues related to stock control efficiency and changes in workload) is in the form of drug issue statistics, obtained from pharmacy computer systems. This includes information about the stock issued in connection with outpatient prescriptions, inpatient prescriptions, discharge prescriptions, ward requisitions and internal orders. It also includes details about items returned to pharmacy, stock adjustments and inter location transfers made.

Planning is also key. To obtain a full picture of the effects of automation it is vital that data is collected before any pharmacy processes are changed in anticipation of the installation of automated dispensing. Experience gained in this project suggests that starting to collect pre-implementation data six months before the system is installed and post-implementation data six months after the system has been established, is ideal. For post-implementation evaluation, it is particularly important to avoid the immediate post-installation period.

Prioritisation may also be an issue. When the study was undertaken in Wales, university researchers were able to assist with, for example, the design of the study and the analysis of the data, and information technology support was given. Where such resources are unavailable, consideration should be given to collecting a more limited range of activities (say dispensing incidents, turn around time for discharge prescriptions, dispensing rate and dispensary skills mix).

Written protocols and information sheets should be provided throughout. It is also important to engage staff fully with the data collection process, especially where they are being expected to take onboard unfamiliar methods.

Dispensing errors

For evaluating the number of dispensing errors pre- and post-implementation, the tool kit essentially employs the methods used in the UK Dispensing Error Analysis Scheme (DEAS). This involves collecting information about, for example, the type of error made, the type of staff who made and discovered the error, the name, form and strength of the drug involved, the risk to the patient, the patient outcome and any contributory factors.To be used in the tool kit, the DEAS system needs to be adapted to include information about the location of the stock, so that any errors that occur involving stock not actually stored in the robot after it is installed can be correctly attributed.

It is important to note that only a small number of errors are likely to be reported, and so statistically meaningful results can be hard to achieve. Analysis therefore needs to take place over quite a lengthy period of time (ideally six months before and six months after implementation).

Dispensing incidents

Because error rates are so low, collecting incident (as well as error) data is also a good idea. The method used for this in the tool kit is to record all errors detected by people doing the final check on a “suspected dispensing incident record sheet”, similar in design to that required for the DEAS dispensing error report. Guidance for assessing the “risk to patient” and “recurrence likelihood fields” is also included. Data should again ideally be collected for six months before and six months after implementation. It is also a good idea to collect data from around the period of implementation to include in the analysis.

Ideally quality assurance mechanisms to check on dispensed prescription items should be performed independently by another staff member at random sample periods. Reports, submitted by this second independent assessor, could then be compared with those generated by the staff undertaking the final checking of dispensed items to allow comparison of the number of reports submitted and to assess the validity of the data.

Distribution incidents

A distribution incident is defined in the tool kit as a mistake relating to a ward or department box, detected before it leaves the pharmacy department. Many departments undertake such analysis routinely but if this is not the case, then incidents should be recorded for two separate one week periods both pre- and post-automation. During these periods, all distribution orders should be checked by senior staff trained to undertake this function, with any incidents and the associated stock locations recorded. The tool kit contains a report form defining 17 incident categories, which was developed from the analysis undertaken routinely.

Turn around time

Ideally, data on the pharmacy time required to dispense prescriptions for outpatients, inpatients and patients who are being discharged should be collected. Inpatient data, however, can be difficult to obtain, because it is hard to collect accurate timings relating to each item or patient from pharmacy requisition sheets.

Where computer terminals exist at all the appropriate locations (ie, including where the prescription is received and the final check takes place), staff will need to add the relevant time into the pharmacy computer system. Where there are no computer terminals in the appropriate locations, times will need to be recorded separately on a manual data collection form Non-urgent prescriptions (ie, those where staff know they have a lot of time to dispense the prescription before it will be collected) should be removed from the analysis.

Out of hours supply

Automated dispensing brings with it the potential to save pharmacist time by enabling on-call pharmacists to dispense items from home using a lap-top computer into an area that can only be accessed by nursing and medical staff.

The tool kit includes an analysis of the number of occasions that on call pharmacists try, and also succeed, in using the remote dispensing function. For successful attempts, the distance the pharmacist would otherwise have to travel to the hospital and the time saving if these staff had received “time off in lieu” is also covered.

Stock control

For departments that currently have an annual stock take, an assessment of the time and money spent on this activity pre- and post-implementation is also included in the tool kit. This involves collecting data about the number of staff employed, their grade and the number of hours they spend on the stock take (which would either be paid to them as overtime or accrued as time off in lieu). Depending on the number of lines stored post-implementation in the robot, it is likely that annual stock takes can be phased out in favour of an approved system of rolling stock checks and this can be factored in to the evaluation.

Clearly, any cost savings made from better stock control (or indeed other aspects of automated dispensing) need to be set off against the costs of installing and running the system.

Dispensing rate

Use is made in the tool kit of a capacity planning method that has already been used and validated in NHS hospitals in Wales to bench mark the dispensing rate (at an average rate of 10.0 items per person per hour) and to evaluate the skills mix of pharmacists, technicians and ATOs engaged in dispensing.¹ Data should be collected for a period of three consecutive days before installation and then for two separate three-day periods post-installation. This is recommended because it is not known how long staff will take to adapt to the changes. Average dispensing rates for each period and indicators of skill mix can then be calculated.

Distribution work load

Time sheets are used in the tool kit to collect information about the current work patterns of support staff involved in distribution-related activities (such as stock ordering, ward top up activities, dealing with returned items and replenishing consumables in the dispensary). Staff are asked to assign a particular category of activity to each 15 minutes of time as they work (rather than retrospectively).

In case support staff find themselves waiting while stock is picked, a “waiting” category is included. An “other” category is also included, for staff to document any activity not listed. Training should be given, or a pilot study run, to ensure that staff know how to fill in the forms appropriately. It is also a good idea for the study protocol to direct staff who are unsure how to code their work to an identified member of staff for assistance.

Time sheets should be filled in for a one-week period pre-installation and another one-week period post-installation. Staff who carry out dispensing and ward top-up activities are only required to collect data on the particular days during the study period that they are engaged in ward top up activities.

It is important that staff who are asked to keep time sheets are aware that the collection of this data is about the processes involved in distribution activities and not about staff productivity. In particular, staff should be made aware that “waiting” is a legitimate activity under the study. It is essential to consult trade union and trust representatives about the design of the study before data is collected.

Ideally, quality assurance mechanisms including observation and/or double checking should be incorporated to ensure data reliability. Issue statistics (see “general advice” above) should also obtained from the pharmacy computer system to determine if the work load is comparable.

Support staff attitudes

Support staff’s views on automated dispensing are also covered by the tool kit. Questions asked include those about:

· Whether automated dispensing has improved distribution and ordering
· Whether the design and operation of automated system is appropriate
· Whether the reporting of errors and incidents has changed
· What their attitude is towards the proposed key roles for pharmacy support services identified by the “Spoonful of sugar” report²
· Whether there are barriers to support staff undertaking these roles and if so, what the barriers are

It is a good idea to include a covering letter from the chief pharmacist or research team, explaining that the questions have no right or wrong answers and that individual staff opinion forms an important part of the study. Colour coding can be used for a multicentre study. The option to canvas opinion from support staff at hospitals where automation is not in place could also be considered. When choosing such comparator hospitals, it is important to make sure that there are no rumours that automation is to be installed at the particular hospital involved, because that might affect the results obtained.

Ward staff satisfaction

Another questionnaire is used in the tool kit to assess the satisfaction of ward staff with the services they receive from pharmacy staff. It is designed to reduce bias — staff are asked to provide information on clinical ward pharmacy services (which should change more slowly post automation) as well as their view on dispensing and distribution services. In order to maximise response, the questionnaire is quite short and employs a Likert scale. Again, colour coding can be used to identify replies in a trust where off-site hospitals or limited pharmacy services are provided. Ideally, it should be sent out to the managers of wards and departments (including those off-site) to be filled in six months before and six months after implementation. To encourage ward staff to express their views freely a covering letter explaining the study from the hospital’s chief pharmacist (or from the research team if appropriate) can be sent out with each questionnaire. If a research team is being used, then a freepost envelope to their address should also be included.

Outpatient satisfaction

The tool kit uses a questionnaire designed to obtain outpatients’ levels of satisfaction on issues such as prescription turn around times and the provision of patient centred advice and education for compliance. The idea is that the questionnaire be sent to all outpatients attending the pharmacy department during a two-week period before, and a two-week period after, automation. These periods should correspond with those used to collect data on actual turnaround times.

It is important to put up a notice informing patients that the study is taking place in the pharmacy waiting area and to send a covering letter with each questionnaire that includes the telephone number of a member of staff who is available to answer any questions about the study. In common with the other questionnaires included in the tool kit, surveys are to be filled out anonymously, and so it is not possible to send out reminder letters or additional questionnaires.

Measuring outpatient satisfaction in this way will not be appropriate where the majority of outpatients attending the pharmacy receive FP10HPs.

Conclusions

It is hoped that the tool kit set out in this article, or at least elements of it, can be used by other hospital trusts to evaluate the effects of any automated dispensing systems they introduce. That way, an evidence base regarding the benefits (or otherwise) of automation can be built up. It might also be possible to compare the different systems available.
Other issues, such as comparing the pre-and post-implementation attitudes of pharmacists and support staff might also be appropriate.³

References

1. Hiom S, Roberts D, Hawksbee M, Burfield R, Francis M, Walker K, Lord S, Warner N. Benchmarking the current dispensing rate of Welsh hospital pharmacies. International Journal of Pharmacy Practice 2003;11(suppl):R85 (PDF 40K)
2. Audit Commission. A spoonful of sugar: medicines management in NHS Hospitals. London: The Commission; 2001.
3. Coleman B. Hospital pharmacy staff attitudes towards automated dispensing before and after implementation. Hospital Pharmacist 2004;11:248–51 (PDF 80K)