Laboratory setting and slide volume

In 2007 we started with the implementation of the first iteration of our digital pathology system. Our department of pathology at the University Medical Centre Utrecht (The Netherlands) is a medium-sized academic pathology laboratory with about 20 000 surgical pathology cases in 2007, 7000 cytology cases, and 330 autopsy cases with approximately 70 000 blocks produced each year. That amounted to ~1 37 000 histological stains and 30 000 immunohistochemical (IHC) stains. That meant that as a laboratory, we produced about 3211 surgical pathology slides per week.

Slide scanners

The main drivers for choosing scanners back in 2007 were scanning performance (speed and reliability) and openness of the system. That meant support for an application programming interface (API) and software development kit (SDK) by the vendor for accessing the system and the image format. Both the API and SDK were used locally to integrate with our pathology reporting system U-DPS (Stichting Palga, Utrecht, The Netherlands) and our laboratory information system (LIS) (Finalist, Groningen, The Netherlands).

Using the money from the Smarter & Better grant, matched by an additional grant from our division management, we decided to buy two Aperio ScanScope XT scanners, supplemented by a third one courtesy of Aperio to create the desired long-term scanning capacity. These scanners were equipped with an Olympus×20/0.75 Plan Apo objective, resulting in a resolution of 0.50 µm/pixel, whereas a magnification doubler could increase the resolution to 0.25 µm/pixel (a technician had to stop the scanner and push a lever to activate the doubler). The scanners used a line scanner as an image sensor and they used points spread through the tissue to construct a focus plane. Images were stored as a proprietary .svs file to form a pyramid multiresolution file format where the compression factor and method could be configured as scan parameter (LZW, JPEG or JPEG 2000). The .svs file format later formed the basis for the pathology DICOM (Digital Imaging and Communications in Medicine) format.7 The .svs file format remained very popular throughout labs for research purposes because of its relative openness to access the pixel data with external programmes and libraries. With the release of the OpenSlide library, however, that point became less relevant since all formats generated by vendors were more of less supported by the OpenSlide library.8 The scanners could be loaded with eight trays of which each could hold 15 slides. That made it possible to scan 120 slides per scanner in one batch.

Slides were scanned after the diagnostic process with the microscope was finished in the traditional way. Basically, that meant that scanning slides was just an additional step before archiving slides. Since slide archiving was taken care of by the morgue and autopsy assistants, they were the ones that were trained to perform slide scanning and they very much appreciated this challenging extension of their work. They developed an efficient scanning schedule where they would start a batch in the early morning and would return to the scanning in between other work to put new batches into the scanners, putting in a last batch before going home. Also in the weekends when they had to come in for morgue or autopsy service they would run batches. No quality control of the wholeslides (WSI) was done at the time, although scanning failures that were encountered by chance were manually corrected, usually by putting in more focus points.

The scanners provided some problems in the beginning especially with regard to glass fragments from the slides that tended to clog the scanners’ fine mechanics, but later ran pretty much like clockwork with little regular maintenance that included vacuum cleaning the scanners of glass fragments.

Some adjustments in the lab processes for producing slides turned out to be necessary. The scanners would sometimes get stuck because of lopsided coverslips, and sometimes slides could not be completely scanned because of too large sections or poorly mounted sections or ribbons that would reach beyond the coverslip. The lab technicians were therefore carefully instructed to not cut too large sections and to mount them centrally, and to make sure coverslips fitted the sections perfectly. Convincing the pathologists and residents to stop scribbling and putting marks on the slides (that would often result in focusing problems) failed miserably, but fortunately, the scanner operators did not mind cleansing the slides of greasy fingerprints and marks with xylene before scanning. To prevent double scanning, all scanned slides were marked with a dot on the label.

There was at the time no alarm system in place to signal scanner failures, meaning that occasionally the operators would encounter failed batches in the morning, necessitating a rescan after fixing the problem.

Storage

The first iteration of our storage system took already into consideration that it had to be fast and large enough to store our slides that were to be scanned every day for the coming years. From the start, it was our vision that all images would be kept indefinitely. We therefore had to set up a large scalable storage facility in cooperation with the central information technology (IT) department of our hospital, and we implemented a hierarchical storage management solution (Sun Microsystems, Santa Clara, California, USA), where initially all images were stored on fibre channel hard disk drives and also copied to tape in a buffered way. The total hard disc capacity for rapid access was initially 6 TB. This size was chosen to accommodate the production of 1 month at the time, since cases are usually discussed during multidisciplinary meetings within 1 month after diagnosis. The tape library consisted of two tape streamers accessed by a robot with an initial capacity of about 120 TB, consisting of tapes with a capacity of 750 GB each. The tape system worked as a second-tiered storage system, so when a user requested an image that was no longer online, it would copy it from the tape back to disk to make it available again, which could take a few minutes, allowing pulling out of old cases even during the multidisciplinary meetings if necessary. This system remained in place up to the point when we migrated to an all disk system. However, the first iteration of the all disk access system was again a two-tiered object-based storage system where access to the data went through a gateway that would retrieve from the second tier to the online storage. The reason was that our digital pathology system was incompatible at that moment with object-based storage systems and required traditional protocols to access the data, which were provided in the form of a gateway system which would handle all queries. For the tape storage system, we had built a software layer (which was possible for that system) that would notify the user if the data were still archived and the approximate time needed to retrieve the data. The disk system had no such possibility, which resulted in users very frequently getting timeouts while trying to retrieve archival cases. The user experience during that time plummeted, which resulted in pathologists not using the system as actively as before.

Migrating to the object-based storage system also changed the business model in terms of storage that we operated up to that point. We went from a system that we invested upfront to a pay-as-you-go model. The continued support for the iteration of that system at that point required a level of faith and trust that was incompatible with the business aspect of reality. We were storing massive amounts of data (at the point of migration, we had almost 350 TB of archive data) and a system that was underperforming due to the storage issues.

After a series of meetings and escalations to the central IT storage group, it was decided that hospital-wide, we would migrate to a new bulk storage system that would natively support use cases such as ours, storing large files for archive purposes without the use of middleware layers for access. The second iteration of the disk-based storage system that we migrated to had superb performance in comparison and made pathologists quite excited to again use the system. At that point, however, the rest of the system was getting quite old, so it was slowly falling behind in terms of business necessities.

Integration and applications

To automate the laboratory workflow process, the scanner system was integrated to our patient reporting system via the barcode on the slide. The slide barcode at that time contained a slideID that was used to identify the slide through our LIS. The scanners would scan the slide and query the LIS for information which would then be used to update the image database. That enabled assignment of slides to cases and add information about block, specimen and staining. Using the API of the image management system, we built a software layer on top that would link a report in UDPS to the digital slides. Clicking on a link in the pathology report would present the pathologists with a collection of thumbnails of all scanned images, macroscopy photos, as well as scanned order forms. Clicking on a thumbnail would prompt opening of that image in the Aperio viewer. On the bottom of the pop-up window, there was a link to add slides to a worklist for a specific multidisciplinary meeting.

That system worked quite well since the integration relied on simple API calls and existing functionality that existed on both the slide scanner system as well at the LIS side. There were, however, several issues that became apparent later on. The barcode on the labels at the time was a type code 128 1D barcode, which included a checksum to verify the contents, significantly lowering the chance of misreading barcodes. However, the information density was low, which did not permit addition of more information than needed. At the time, we included only a numeric ID which by itself was quite error prone. If another barcode were to be scanned with an ID that might happen to correspond to a slideD, then that slide would be added to the wrong pathology report number and have the wrong metadata added to it. That posed several issues that came to light much later when we realised that several consultation cases, where the label was situated in way that both our barcode and the requesting labs barcode where visible, where misassigned. However, the largest batches of slides that were misclassified were the IHC slides due to the double barcode they used. We use labels for our IHC slides that contain two types of barcodes, one used by the LIS system for identification and the other for our staining system to link the glass slide to a staining protocol. As soon as we had used the IHC system long enough, the number range used started to overlap with the slideIDs. That meant that almost 16 months of IHC slides were misassigned. This only became apparent when we were reviewing the scanned slide archive to migrate to another system. All of those slides had to be manually corrected before migrating, which cost us 3 person months, as well as a 3 months of delay to start the migration. We have since switched to 2D datamatrix barcodes and added more information to make the slideID truly unique.

Research, teaching and outside consultation

Soon after implementation of the scanners, many researchers started to find their way to our department to have their tissue slides scanned. They found it much easier to carry an external hard drive with their WSI that could be viewed on any computer after downloading the free Aperio viewer than to drag slides around that needed to be viewed in a (usually remote) room with a microscope. Moreover, WSI allowed researchers to easily capture snapshots from digital slides for, for example, publication and presentation purposes without the need of having an expensive photomicroscope. Regularly, this led to scanning capacity problems but also brought in some business, as we would charge slide scanning to people from outside the department.

The process of switching to practical sessions with WSI instead of slides with microscopes was started soon after we had the scanners installed. Within a year, we had all practical sessions digitised within the Digital SlideBox environment (Slidepath, later acquired by Leica). The students appreciated the digital practical sessions much better than the microscopic sessions.

Anticipating a rapid growth in digital consultations from outside labs, we purchased a licence to the mScope application (Aurora, Montreal, Canada), where, based on our wishes also, an expert panel module was built in. Getting serious numbers of digital consultations miserably failed, although the expert panel module was successfully used for a number of years by the Dutch Breast Panel and SKION, the Dutch Paediatric Haematology Panel. We came up with the innovative but untimely idea to buy a small mobile Aperio whole slide scanner to instal locally in labs requiring instant diagnostic support in case of understaffing, scanning locally and importing the images into mScope for remote digital diagnostics. This never worked and the scanner has pretty much been collecting dust for years.

Interlude

The first generation of our digital pathology system, which began on November 2007, started showing signs of its age by the end of 2014. The scanners were running on Windows XP SP3 and were end of life by 2014. In order to upgrade them we would have to seriously invest in new scanners. Also the current home-made software system was not fast enough anymore, with the database of the image management system growing larger at the same time that more users were logging on to the system and the rate of scanned slides per day increasing every year. By that time we were producing almost 50% more slides compared with when we first invested in the system so capacity started being an issue. At the same time, the scanners needed several minutes per slide 20× and scanning at 40× was prohibitive in terms of scan time.

The needs of our department were changing, and the system at the time no longer satisfied the requirements of daily diagnostics. There was an increase in the number of multidisciplinary meetings per week. Further, slide scanning after the diagnostic process with the microscope had been finished was no longer perceived as satisfactory. Although preparing and doing multidisciplinary meetings with digital images was considered a great asset saving much time, not all slides were digitally available for all meetings since they were perhaps too recent and not yet scanned, and switching between digital and microscope during meetings was less than easy. Further, slides were untraceable if they were in the scanner.

Therefore, we realised that slides needed to be scanned before they were delivered to the pathologists to make sure that all slides would be digitally available to all residents and pathologists at all times, at the same time providing them with the option to do diagnostic work digitally. These factors were the main drivers for deciding it was time to take the next step and move to the second-generation digital pathology system, a primary digital diagnostics workflow.9 To prepare for this, we conducted a series of studies to validate digital diagnostics.10–17

Project establishment

In 2015, we decided that we needed to take the next step and not only renew our fleet of scanners but also change the way we do diagnostics. After having discussions with key industry people and scanner manufacturers, we realised that the available hardware on the market was mature enough to handle a fully digital workflow and that there were several manufacturers that could compete for an implementation contract. The ambition was born, a professional highly efficient workflow system for primary digital diagnostics without negative effects on throughput time, ready for implementation of image analysis/AI while maintaining our principle of a complete digital archive.9

The new system would comprise several high throughput scanners, a workflow system to manage it, an image management system as well as the IT hardware to host it. The amount of capital investment needed and the fact that our institution is publicly funded warranted a European tender process to select the vendor.

We opted not to follow the traditional tender process, that is, compiling pages of technical characteristics that the solution had to adhere to and ending up selecting the lowest twice daily, but instead followed a ‘best-value procurement’, where we would request a very limited set of requirements that were knock-off criteria, which were prerequisites to two times per day.

Among the requirements was that the system delivered would include scanners with enough capacity to scan the daily production and cover slides produced during peak hours: at least 800 slides/day and 120 slides/hour. In this way, the throughput of the system would not act as a bottleneck for the diagnostic workflow. Apart from throughput, we did not set any requirements concerning image and scan quality.

Eventually, a consortium composed of Sectra AB (Linkoping, Sweden) and Visiopharm (Hoersholm, Denmark) (as the reseller of Hamamatsu Photonics K.K. in The Netherlands) came out as the winner, proposing a system based on Hamamatsu (Hamamatsu City, Japan) scanners and Sectra PACS as the image management and workflow system.

We began immediately implementing the solution in one of the most ambitious projects of our department since we quickly realised that changing the workflow to a primary digital diagnostic workflow left no department unit unaffected, which meant that we had to set up workgroups that were composed of representatives of every unit within our department.18 This exercise in extreme change management helped involve all of our staff in the new project, allowing everybody to partake in the implementation and have them take ownership of the project. That might have been one of the most important aspects in successfully implementing this system.

Slide scanners

The proposed solution offered by the vendors was three Hamamatsu XR scanners and one Hamamatsu RS scanner with a fluorescence unit. The Hamamatsu XR scanner is a high-throughput scanner which works as our main workhorse, whereas the RS was selected to scan 2×3 slides and fluorescence so focused on all specialty cases. It quickly became evident that even though the scanners could achieve the scan times advertised, in practice the scan times were lower due to several factors. First, average tissue size is larger than then popular advertised 15×15 mm tissue size. Tissue placement on the slide as well as the use of control cores for IHC stainings make the average slide size about 18.5×18.5 mm. Second, IHC stained slides when negative are very lightly stained, which in return are hard to be recognised by the scanner tissue detection algorithm. That required rescanning of the slides with extra focus points adding to the total scan time. Third, more H&E slides than anticipated needed to be rescanned due to focus issues.

Even though we had enough capacity to scan our daily production, we seem to have very few margins for error, so we had to make some changes in order to scan more efficiently. By having the scanner to query the LIMS using the unique slide identifier contained in the barcode, it retrieved the slide metadata. We then used the metadata to instruct the scanner to use a different profile per slide in order to scan more efficiently. The goal was to minimise the total number of rescans. We constructed around 10 different profiles which ensured that the slide would always be scanned without fail. That allowed us to create a margin of error and also manage to create some room to anticipate great peaks of slide production.

One of the requirements of the project was to use DICOM in order to store our scans. Unfortunately, when we started the project, there were very few vendors that even had a DICOM implementation let alone a full integration. We started a pilot programme to use DICOM for our scans but quickly scaled it back due to several integration issues. The DICOM integration was developed to convert the scanner’s proprietary format files to DICOM. That created a bottleneck in the scanning process, which was prohibitive for our production scanners, since we would have to transfer the files to an intermediate conversion server and then send them to our Picture and Archiving Communication System (PACS). At the end, we opted to run the DICOM conversion only on one scanner as a beta process in order to learn from it. Other issues that we encountered trying to use DICOM were

1. DICOM populates the headers with patient information and slide metadata, which it retrieves using the unique slide identifier. However, when the scanner fails to read the barcode, the conversion fails and the process of rectifying the failed data retrieval costs a significant amount of time per slide. We have not yet found a solution around this problem.

2. The DICOM conversion needed an extra middleware infrastructure to handle the conversion. Since it requires high read and write operations, we needed to invest in high bandwidth storage system to handle the Input/Output (I/O) load. That would have increased our IT support and budget.

3. Our PACS system supports DICOM but does not support DICOM operations for pathology images such as C-STORE and C-MOVE (as of writing this article, they have implemented some of the operations with some restrictions), which meant that after generating the DICOM files, we sent them to PACS via the import nodes reserved for the proprietary scan formats such as ndpi from Hamamatsu and not via a C-STORE operation. That means that there is a non-zero chance that a DICOM slide would be imported in PACS missing part of the data (usually the 40 x layer).

Our policy moving forward is when choosing a scanner, to opt for a native DICOM generation (scanner produces DICOM from the source and not via a conversion step).

Storage

Since the start of our digital pathology journey, we have opted to keep all of our scanned slides in order to create a digital archive to reduce reliance on our glass archive. That has resulted in having a 12 year digital archive of almost 2 PB of data. That has created strain in the digital storage system that we are using as well as to our budget. Thankfully, because of the demands that we have as an archive, it has pushed our central IT to invest in storage systems and expertise, which subsequently has reduced costs for all hospital users.

The way our storage system was set up was to ensure maximum redundancy of all events even, making sure that the complete archive was duplicated in two data centres, so in the event of a catastrophic failure in one of the data centres, the other location would be available. Keeping an archive mirrored in two locations doubles the cost of storage, so we weighted the costs against the benefits and at the end it boiled down to one question: if we were to lose our entire archive to a catastrophic event, how would that affect our daily diagnostic practice? Since our archive is separate than our short-term storage, we would still be able to use the system and we would just lose access to our digital archive. So, we decided to revert to a single location archive, reducing our costs to almost 50%. That has created a situation where we expect a reduction in price per TB every 3 years approximately, reducing the overall cost of our complete archive even though it keeps increasing in size every year.

The specifications for the storage needed to support a digital pathology operation are actually quite low end, which translates into using archive storage (high-density nodes with low throughput) for our daily operations without any noticeable drop in performance.

Workflow

To be able to scan slides within 15 min after being coverslipped, quick slide drying was necessary to prevent slides from getting stuck in the trays due to mounting medium remnants. We therefore switched back from a xylene replacement to xylene which provided much quicker drying, and reinstalled an airflow slide drying table in the lab that had been in storage for some years. The scanner operators, who were now the histotechnicians, implemented a procedure for slide cleansing, including removing mounting medium remnants before putting slides in the scanner tray.

Initially, we had in mind to instal the scanners in the lab according to lean principles, but they turned out to be noisy and producing too much heat and ozone, necessitating putting them into a separate room with proper climate control. A large format display was installed, which receives the output of all scanners and combines them in one view to monitor all operations from one place (figure 1). A quality control procedure of scanned images is in place as described in detail before.18

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Large screen in the scanner room to monitor progress of the four scanners.

We encountered image quality issues shortly after the system had gone live, related to the limited numerical aperture of the scanner lenses, leading to fairly thick optical slices blurring nuclear detail.18 Helicobacter pylori and intraepithelial lymphocytes were difficult to see; dysplasia could be overdiagnosed; and counting of mitoses was more difficult. We therefore created a workaround by making the gamma adjustable in the PACS system and switching from sections 4–3 μm thick.