Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Erratum to: Chapter 2 in: Y. Sucaet and W. Waelput, Digital Pathology, SpringerBriefs in Computer Science, DOI 10.1007/978-3-319-08780-1_2

The authors have retracted this chapter published in Digital Pathology (Pages 15–29, DOI 10.1007/978-3-319-08780-1_2), as it contains multiple portions of text that have been duplicated from the OpenSlide website, http://openslide.org/. In particular, Sections 2.4.1.3, 2.4.1.4, 2.4.1.6 and 2.4.2.1 contain text that has been copied from the pages http://openslide.org/formats/leica/, http://openslide.org/formats/aperio/, http://openslide.org/formats/hamamatsu/ and http://openslide.org/formats/mirax/, without sufficient permission or attribution.

The whole chapter ‘Hardware and Software’ by Y. Sucaet and W. Waelput in Digital Pathology (2014), (pp. 15–29) is retracted and replaced by a new chapter based on author’s request.

Chapter 2

Hardware and Software

Abstract

Lest those interested in exploring the field not understand the nuts and bolts of the system, no book on digital technology is complete without some background on the available hardware and software. The field is changing rapidly and specific examples may be already obsolete at the time this book goes to press. At the same time, we have found that certain principles have remained constant for a relatively long time now, and we believe that providing readers with some general technical background will help on the path to implementing successful digital pathology solutions.

Keywords

Digital pathology • Slide scanner • Tile scanner • Line scanner • File format  • Medical imaging • WSI • DP • MRXS • NDPI • SVS • BIF

This chapter talks about the technology that is used to arrive at digital pathology. As the pathologist is dependent upon his microscope, so is the digital pathologist dependent upon a digital camera or slide scanner for the creation of a single, high-magnification digital image of an entire microscopic slide or whole slide image (WSI).

This chapter is split into two parts. In Sect. 2.1, we elaborate on the various hardware components necessary to acquire virtual slides. In Sect. 2.2, we survey the various approaches to data storage and file format organization that different vendors have developed.

How Are Digital Pathology Images “Captured”?

Basically, WSI hardware consists of a robotic/automated microscope with specialized acquisition software. Some instruments are more specialized and purpose-specific in their design and construction than others. The simplest setups consist of add-on cameras on top of conventional microscopes. This is a great start if all you want to do is capture regions of interest (ROIs) and share them with colleagues or embed them in your publications. However, they are not necessarily suited for whole slide imaging. In order to do WSI, as well as systematically digitize your entire workflow, you need at least a robotic staging table as well. The robot then cooperates with the software component to move the slide, capturing individual ROIs that they are stitched back together to generate a WSI. Special viewing software is usually provided so that it appears that a seamless image was obtained by the entire slide. As an alternative, there also are devices like microscopes with mounted cameras but automated stages. The advantage is that viewed ROIs can be stored even when you switch between different slides (Fig. 2.1).

Fig. 2.1
figure 1

a A Zeiss AxioVision setup with a mounted camera and robotic stage (courtesy of HistoGeneX). b A 3DHistech high-volume slide scanner (courtesy of HistoGeneX)

Full size image

Technology has by now become sufficiently specialized so that some companies only sell complete integrated systems (scanners). However, others sell individual components as well. Examples of the latter are Hamamatsu, which sells its own Nanozoomer slide scanner as an integrated system, but also sells its components to TissueGnostics for their automated solutions.

How Do Slide Scanners Work?

Slide scanners are the highest level of abstraction for digital microscopy. They have both hardware and software components. We distinguish five levels, from lowest to highest: slide handling, slide scanning, optics, detection and, finally, acquisition software. These five levels are depicted in Fig. 2.2.

Fig. 2.2
figure 2

Different layers of processing in digital pathology

Full size image

The first slide scanner was designed by James Bacus in 1994, during a period of rapid Internet expansion worldwide. The corresponding BLISS system, which is now recognized as having been the first virtual microscopy system ever developed, was designed to generate virtual microscope slides. Meanwhile, a WebSlide Server, Browser, and ActiveX Viewer were developed to allow for viewing virtual microscope slides over the Internet. Over the next several years, the Bacus Group developed and patented the methods and apparat to perform automated assays of biological specimens, immunoploidy analysis, measurements of tissue thickness, and tests of neoplasm progression, as well as devices to allow for the remote control of microscopes, the creation of virtual microscopy slides, the magnification of specimen images, and the Internet, intranet, and local viewing of such slides [1–19]. Moreover, Bacus Laboratories not only created the first virtual slide system, and they also created the first market for it. They did this by framing their system as an educational tool. Their plan was to ultimately replace standard microscopes with virtual microscopy in medical education. They achieved this by combining their virtual microscopy system with a collection of educational “slides,” for which institutions could lease access licenses annually. Because of its successful business model, Bacus Laboratories was purchased by Olympus America Inc. in 2006.

In terms of features, the capacities of slide scanners today vary widely. For example, some can do bright-field images only, some can do fluorescence images only, and some can do both. The price of a model generally correlates with its slide-loading capacity, which can range from one to 400 slides per batch. Slides can be handled as a single slide/stage, as stand-alone autoloaders (“hotels”), as slide trays, or as slide magazines. The type of slide can vary as well: While most scanners today still handle basic 1” × 3” slides only, others—such as Aperio and Huron Technologies—also support 2” × 3” and even larger (whole mount) slides. The more variability that is allowed for physical slide media, the harder it is to batch-process large numbers of slides.

Two approaches exist to scanning a slide: tile scanning (Bacus patents) [20] and line scanning (Aperio patents) [21]. In both cases, the resulting images (tiles or strips) are fitted together into a single large image (i.e., the WSI). With a tile scanner, the slide is scanned as a series of rectangular tiles. For each tile, the highest physical magnification desired is used (e.g., 40× or 20×). The tiles are then stacked into a WSI, like bricks forming a wall. This is done either concurrently with or after the scanning process, via the acquisition software. Conversely, with line scanning, after magnification, strips are combined side by side into a single image. Proponents of the latter approach claim that it generates fewer seams and, hence, fewer optical aberrations (Fig. 2.3).

Fig. 2.3
figure 3

a/b Tile versus line scanning: note the huge decrease in the number of seams with line scanning

Full size image

One particular problem related to scanning is focusing. A pathologist looking through a microscope automatically adjusts the focus depending upon the area of the slide he or she is looking at, the thickness of the specimen, the type of glass slide used, etc. With a scanner, this process must be automated. With both tile and line scanners, it is possible to auto-focus each field after moving the stage, but this can be very time-consuming, especially with tile scanners. A better approach is to focus on every nth field being scanned. This is both faster and simpler; but the placement of focus points lacks context, and it is still possible to waste time on larger areas that, by chance, do not require refocusing. A focus map is another solution. With this approach, focus points are distributed over the tissue forming a surface. Focus is only recalculated for intervening tissue. The number of selected focus points can be controlled via the scanner software. A trade-off is usually made between more focus points (less speed) and greater accuracy. The settings can be tissue-dependent, and a technician can maintain a preset list of “profiles” that can be referred to, depending upon the type of specimen that needs to be scanned.

Z-stacking is becoming increasingly commonplace, but this poses its own unique challenges to file format organization (see later in this chapter). The new frontier is now spectral imaging.

Virtual Slide Formats

How Are WSI Data Organized?

After the acquisition software in the scanner obtains a digital image representation of a slide, it needs to store this information somewhere. This again can be seen as a two-step process, whereby first data compression takes place and subsequently data are stored, usually in a vendor-specific file format.

Digital slide image formats typically consist of one or more files that contain high-resolution scanned areas as well as image information in the form of metadata. The resolution of such images varies, but usually ranges from ten to hundreds of thousands of pixels per dimension (width and height). Various techniques are currently employed to make it easier and quicker to process such images using computer software.

The Pyramidal Format

Scaled versions of the original image (called “zoom levels”) are often created and stored in a single “container” format. This is usually called a “pyramidal format,” since every scaled-down image is smaller than its previous level, just like a pyramid gets smaller and smaller the higher up you go. By storing pre-computed scaled-down versions of the high-resolution image, a computer program can quickly render a smaller version of the image by reading pixel data from the zoom level closest to the scale currently being displayed.

The pyramidal format increases display performance at the cost of storage efficiency. For this reason, many vendors try to minimize the actual scan area that is being stored. This is done by spotting the significant areas while scanning the slide and only storing these in high resolution. This leads to a digital slide image with many sparse high-resolution areas, which may follow the pyramidal format independently. For different tissue types, the tissue detection parameters (called “profiles” by some vendors) often must be fine-tuned (Fig. 2.4).

Fig. 2.4
figure 4

A pyramidal stack represented symbolically on a sample image

Full size image

Tiles

To further optimize random access and minimize disk read operations (input/output or I/O), digital slides split the image into smaller rectangular areas (tiles). Every zoom level is therefore a grid of tiles of the same size. When a computer program needs to display a small part of a high-resolution image, it is able to reduce the data being read by selecting only those tiles that intersect with the current viewport.

Slide scanning is performed in steps. The scanner’s camera moves along the slide and takes pictures which are then stitched together by the scanner’s software. Some vendors decide to store overlapping images of the slide and let the viewing software do the stitching. This is done because selecting stitching offsets that depend on the visible parts of the image every time may reduce stitching artifacts. This, in turn, would have otherwise been introduced if stitching had happened during scanning. In this case, stitching hints are stored as metadata along with the image.

Regardless of when the stitching process takes place (during scanning or while the image is viewed), images acquired by the scanner require adjustments. Overlapping regions might have differences in brightness and contrast, known as shading, due to the different positions of the camera, each time a photograph is taken. Various techniques are employed to address this issue, such as blending and histogram equalization.

Color Spaces

The most common illumination techniques found in digital slides are bright-field and fluorescence. Bright-field microscopy images typically store pixels in the RGB (red, green, blue) model (color space) or YCBCR (another family of color spaces) for JPEG images. Grayscale (points of equal RGB values; essentially a subspace of the RGB model) is especially used in the case of fluorescence microscopy slides to store the intensity of the reflected emission. This is then multiplied by a constant factor in order to be colorized for display purposes.

Compression Schemes

Digital pathology images employ various compression and image representation schemes, which may or may not lead to color information loss. Some of the compression schemes that are used are Raw, JPEG, JPEG2000, PNG, LZW, and DEFLATE.

Vendor-Specific File Format Implementations

The images that are generated by digital slide scanners are very different from the JPEG images typically obtained using your cell phone or digital camera. Top start with, they are vastly larger and more complex. This is aptly demonstrated in Fig. 2.3, in which a typical high-resolution digital camera image is compared to a typical digital WSI. Note that the WSI has almost 1500 times as many pixels (200,000 × 100,000 vs. 4,600 × 3,000) as the camera shot (Fig. 2.5).

Fig. 2.5
figure 5

Comparing a traditional high-resolution digital camera photograph (of coauthor YS and his bouncing daughter) against a typical digital whole slide image

Full size image

There also are a large number of different WSI file formats, which are necessary because of the multiple applications for which these images are used, beyond simple viewing. For example, Zeiss format (czi and zvi) images (see further in this chapter) can encompass as many as 6 or 7 dimensions, versus the 2-dimensional JPEG you generate with your home camera, to accommodate their use for microbiology, time-lapse, fluorescence, and other applications. Below we briefly describe just some of these various formats, highlighting their basic characteristics, as well as their advantages, disadvantages, and differences.

TIFF-based Formats

TIFF

TIFF images are used by scanner vendors to store digital slides. The TIFF format natively supports storing images in grids of tiles and is generally well suited for random access. It allows for multiple images (directories) to be stored within a single file and for various compression schemes to be used. Since a slide’s size may overcome the maximum 4-GB threshold, the BigTIFF format is also common. It essentially uses 64-bit pointers to store offsets within the file.

Typically, one tiled TIFF directory is the high-resolution image, while several others may follow that are down-scaled versions of the original. One downside of the plain TIFF format is that there is no definitive way to specify which directory is for the high-resolution image and which are for the down-scaled images, because the specifications do not anticipate relationships between the directories. The display software attempts to overcome this by making assumptions; for example, the largest directory (in width or height) may be considered the original image and every other directory a smaller zoom level.

Open Microscopy Environment (OME) TIFF (Extensions .tif, .tiff)

The OME-TIFF format was designed to incorporate both the rich metadata that is included within the OME-XML format, and the pixels that exist within the multi-page TIFF format. In this way, it is compatible with a much broader range of applications. There are several other main features of OME-TIFF datasets that make them distinct from other formats [22].

First, each and every dataset contains image planes that are stored either within a single multi-page TIFF file, or spanning multiple TIFF files. With either of these options, virtually any image organization scheme is possible.

Second, embedded within each TIFF file’s header there is a complete OME-XML metadata block that describes the dataset. In this way, the metadata remain intact even if some of the dataset’s TIFF files become displaced. This OME-XML metadata block can contain anything that is permitted within a standard OME-XML file.

Third, the standard TIFF mechanism is used to store one or more image planes in each of the constituent files, rather than encoding pixels as Base64 chunks within the XML. Since TIFF is an image format, when there is at least one image plane, it makes sense to use OME-TIFF instead of OME-XML.

A more complete description of the OME-TIFF format, including companies that support it, public image repositories that permit image downloads as OME-TIFF, more detailed technical information, an example code, and sample data, is available online at https://www.openmicroscopy.org/site/support/ome-model/ome-tiff/.

Ventana BIF (Extension .bif)

Ventana slides are stored in single-file BigTIFF format. The first directory contains a label image, usually in tiled format. The label image is a thumbnail that includes the actual physical label for the glass slide. The directory specifies the XMP tag (700) and stores valid XML metadata about the slide. Next comes a thumbnail, and the high-resolution image follows.

BIF images contain overlapping tiles, for which an appropriate algorithm is required for them to be correctly rendered. The directory containing the high-resolution images also specifies the XMP tag that contains tile stitching hints between neighboring tiles. The rendering algorithm calculates global coordinates for every tile, based upon these hints. This may result in stitching artifacts in parts of the image. Subsequent image directories do not have such information, and tile positions are calculated via reduction to the base level.

Other TIFF Formats

Several other vendors use derivatives of the TIFF format. These include Leica SCN, Aperio SVS, Trestle, and Hamamatsu NDPI. Sometimes, it suffices to rename the proprietary file extension into .tif to visualize files in common software packages such as Adobe Photoshop. Several file formats are then documented in more detail at the OpenSlide project: http://www.openslide.org/formats.

Other Format Types

Not all vendors follow the TIFF format. 3DHISTECH Mirax (.mrxs extension) slides are stored in a multi-file JPEG format with proprietary metadata and indexes. One slide corresponds to many files in a single folder. Each file contains an aspect of the format, such as an overview image, a particular zoom level, or annotation data. The index files information on where to find particular pieces of data into the individual .dat files.

The Olympus file format (.vsi extension) is derived from TIFF. Like Mirax, it consists of a collection of different files in a single folder, with the .vsi file serving as an index file. Olympus files can contain multiple regions of the same physical slide, scanned at variable levels of resolution. High-resolution pixel data are stored in extensible tile server (.ets) files that are maintained in subdirectories (defined in the “main” .vsi file). ETS is a proprietary file format that is used to store multi-dimensional data organized in tiles. In most instances, a single region of a slide is stored in tiled pyramidal fashion within an ETS file.

Finally, there is Zeiss. Like Hamamatsu and Leica, Zeiss has multiple file formats defined for whole slide imaging. The CZI format was designed to mimic open microscopy environment (OME) specifications (http://www.openmicroscopy.org). It is intended to be maximally compatible with OME-TIFF and OME-XML data formats, while maintaining the specifications that are essential to optimize the use of Carl Zeiss ZEN software.

ZVI is older than the CZI format, but still widely because of the widespread use of the platforms that use it. AxioVision is one of the programs that support the format, and a plug-in for ImageJ is also available (and comes standard with the Fiji toolbox). Within a ZVI file, a multi-dimensional space is defined to facilitate time-lapse, multiple (fluorescent) channels, and mosaic-style recordings.

Additional information on each format can be found here (sometimes after signing a license agreement):

The Role of DICOM

As the proverbial 800-pound gorilla in the room, DICOM deserves its own paragraph. DICOM stands for Digital Imaging and Communication in Medicine and is a network maintained by the National Electronic Manufacturer’s Association (NEMA) and supported by large-image management systems called picture archive and communication systems (PACS). Various PACS systems are used in hospitals and laboratories to manage images used for clinical and research purposes in medicine; this includes, among other functions, their storage, and retrieval.

Since 2009, a new supplement has been added to the DICOM standard. This supplement is known as “Supp 145, Whole Slide Imaging in Pathology.” The supplement was developed by Workgroup 26 and describes how an extension has been made to the DICOM standard to allow for the storage of very large images. The DICOM standard defines a family set for the image, called “instances” as per the DICOM vocabulary. All these instances follow an information object definition which is defined in PS 3.3; currently, version 2011 is the latest available. In all those IODs, DICOM instances have columns and rows defined as unsigned short values. What this means is that, in theory, all images must be 64K columns and rows. WSI frequently have images much larger than these pixel dimensions.

Rather than follow what occurs during the TIFF to BigTIFF (64-bit extension) transition, the DICOM committee chose a different, very conservative approach, whereby the unsigned short value for the column and row does not change. Instead, new attributes are added to store the actual pixel dimension. In this scenario, a single WSI cannot be stored within a single “instance.” Instead, a single WSI is inserted in fragments at a series level.

The proposed approach guarantees that all legacy software remains able to process any incoming WSI series, since the attributes in columns and rows are still defined as unsigned short.

One should notice, though, that this supplement pushes the DICOM standard to the edge, since uncompressed pixel data stored within a single DICOM instance are limited to a 232 − 1 byte (4 GB minus a byte, 0xFFFFFFFF being a reserved value). Therefore, the lower level of this pyramid is unlikely to be saved in uncompressed form, since its total size will likely exceed that limit considerably. In such a case, it is assumed that another transfer syntax will be used for those larger pyramid levels (e.g., JPEG). When using an encapsulated transfer syntax (e.g., JPEG type), there is no such limit, and all individual tiles can be stored within a single DICOM instance.

Do-It-Yourself Programming

For people who have experience with programming (be it in a full-blown framework like Java or a scripting language like Python), options exist to get right to work interacting with the data from the various hardware vendors.

OpenSlide is written in C, having its origins at Carnegie Mellon University (93;94). It has binary distributions for various flavors of the Linux operating system, as well as for Windows. Individuals have also reported on how to deploy the library on Apple hardware. Instructions on how to deploy the software on each of the respective platforms are provided on their Web site http://www.openslide.org.

An alternative library is the BioFormats project. This was initially developed at the University of Wisconsin–Madison [23]. BioFormats is written in Java and has a wider selection of supported file formats, but some currently used microscopy formats are missing or only partially implemented. Extensive documentation on the library is provided through the OpenMicroscopy portal at http://www.openmicroscopy.org.

The goals of OpenSlide and BioFormats are slightly different. While both can be used to read proprietary vendor formats, OpenSlide was started as a project to visualize large images. Meanwhile, BioFormats is seen as a way to convert a proprietary format into an intermediate format (OME-TIFF, cf. supra). Perhaps this also explains why OpenSlide seems more performant than BioFormats; it is unlikely that one would do this conversion in real time.

If workflow permits it, BioFormats can simplify matters, because a single file format is the outcome (although OpenSlide abstracts things as well). The file format furthermore not only has image data, but capabilities for additional annotation, something that OpenSlide deliberately avoids.

If you decide that you want to process your own WSI data, you should consider the file format that you work with. OpenSlide is the only one that supports 3DHistech’s MRXS format, whereas BioFormats is the only one that (at least partially) supports the Olympus and Zeiss formats.

Bits, Bytes, and Wires

This book is not intended as guidelines on how you can build your own scanner or write your own WSI viewer software. Nevertheless, no review of digital pathology can be complete without also addressing the hardware and software involved. We have tried to introduce you to some of the intricacies of engineering that were required to develop slide scanners in the first place. Then, we moved on to the software side of things: How are WSI data stored? This is something that we all get exposed to, if only by transferring slides to a colleague via a USB memory key.

Slide scanners have not been around all that long. Two basic modes of operation exist for scanners, and virtually all scanners on the market today can be traced back to one or two sets of patents. Data captured by the slide scanner must be stored on the hard disk of the user’s computer and organized so that it can be visualized optimally. File formats have been devised by different vendors to accomplish this. Because of the pixel size of the raw data (roughly 1,500 times more than the digital camera that you use on vacation) and the different features of the scanners (bright-field, fluorescence, confocal, etc.), various solutions have been thought of. However, these differences make it hard to move from one digital pathology platform to another, and one risks vendor lock-in because of this. Some initiatives for standardization have already been undertaken and are expected to become more center stage in the future.