Craniofacial Superimposition Techniques

Abstract

The technological support for craniofacial superimposition (CFS) techniques used in the initial identifications found in the literature involved a large number of very diverse approaches. This could also be the reason for the current diversity of CFS methods and their terminology, as mentioned before. Rather than following a uniform methodology, every expert tends to apply his/her own approach to the problem, based on the available technology and his/her own knowledge of human craniofacial anatomy, soft tissues, and their relationships. Therefore, CFS approaches have evolved as new technology has become available, although their foundations were previously laid. Some of these approaches were classified in a review by Aulsebrook et al. (1995) according to the technology used to acquire the data and to support the skull-face overlay (SFO) and identification processes, that is, static photographic transparency, video technology, and computer graphics.

4.1 Introduction

The technological support for craniofacial superimposition (CFS) techniques used in the initial identifications found in the literature involved a large number of very diverse approaches. This could also be the reason for the current diversity of CFS methods and their terminology, as mentioned before. Rather than following a uniform methodology, every expert tends to apply his/her own approach to the problem, based on the available technology and his/her own knowledge of human craniofacial anatomy, soft tissues, and their relationships. Therefore, CFS approaches have evolved as new technology has become available, although their foundations were previously laid. Some of these approaches were classified in a review by Aulsebrook et al. (1995) according to the technology used to acquire the data and to support the skull-face overlay (SFO) and identification processes, that is, static photographic transparency, video technology, and computer graphics.

Similar classification schemes have been reported by other authors, which describe how CFS has passed through three phases: photographic superimposition (developed in the mid-1930s), video superimposition (widely used since the second half of the 1970s), and computer-aided superimposition (introduced in the second half of the 1980s) (Nickerson et al. 1991; Aulsebrook et al. 1995; Yoshino and Seta 2000). Moreover, Yoshino et al. (1997) classified some of the computer-aided CFS methods into two categories from the viewpoint of the identification strategy. The first strategy is to digitize the skull and face photographs and then morphologically compare the two images by image processing. The second is to evaluate the fit between the skull and facial images by morphometric examination. Notice that the latter contributions are prior to the image-processing boom of the last decade. Indeed, important issues like 3D modeling and machine learning are neglected. In the case it was used, the computer was usually considered just as a secondary support for the technique even when authors claim they followed a “computer-aided” approach (Ubelaker et al. 1992; Ricci et al. 2006).

Recently, Damas et al. (2011) surveyed existing methods considering a new computing-based classification criterion. This criterion is more related to the use of computers during the different stages of the CFS process itself. They defined the different stages involved in the craniofacial process to properly characterize any CFS system (and specifically computer-aided ones). These stages: face enhancement and skull modeling, SFO, and decision making:

• The first stage involves achieving a digital model of the skull and the enhancement of the image of the face. This stage is not present in all the systems. Indeed, the oldest systems and most of the recent ones still acquire a photograph and/or a series of video shots of the skull, instead of building a 3D model of it.¹ Concerning the image of the face, most recent systems use a 2D digital image. This stage also involves the application of image-processing techniques (Gonzalez and Woods 2008) to enhance the quality of the image of the face that was typically provided when the person disappeared.

• The second stage is the SFO. It consists of searching for the best overlay of either, a 2D image of the skull and face or the 3D model of the skull and 2D image of the face acquired during the first stage.

• Finally, the third stage of the CFS process corresponds to the decision making. Based on the SFO achieved, the identification decision is made by investigating the outline and soft-tissue thickness at various anthropometric landmarks and positional relationships of the skull to face parts based on anatomical data.

This categorization of the superimposition process will be taken into account throughout the current document when reviewing all work carried out in the field.

4.2 Photographic Craniofacial Superimposition

Identification by photographic superimposition is a technique that began to be developed in the middle of the 1930s. The first superimposition was probably made in 1885 by Welcker and His, who used an outline drawing of the skull in relation to drawings of the deceased’s death mask. From the beginning, their studies of the relationship between soft and bony tissues were taken as a basis for developing new studies and different methodologies in this forensic area. Following this, many authors performed different measurements of the soft and hard tissues in order to make reconstructions of their subject’s face for sculptures and paintings. In some cases, these studies were focused on making a comparison between the reconstruction and the actual face represented in paintings. The appearance of the photographic CFS technique allowed for the first time the comparison of an unknown skull with facial images of the presumed deceased. Its evolution has resulted in a variety of techniques and tools depending on several factors, ranging from the skull conservation, to the quality of the images and the knowledge of the expert who proposed it.

We can consider the acquisition of a painting, contour, or photograph of both the skull and face, the replication of the skull size and its orientation with respect to the facial photograph of the presumed victim, and the assessment of the anatomical consistency between the face and the skull as common tasks in photographic CFS. A clear relation with the three stages identified before can be seen with the most common tasks to be carried out in each of them as follows:

• Face enhancement and skull modeling: In this first stage, one or more photographs of the skull have to be taken. The quality of the employed photographs will be related to the quality of the photographic equipment used in the acquisition. In particular, the type of lens, the focal length and the distance, and position of the subject together with other factors are responsible for producing deformations. Many authors have drawn tracings of the face and/or skull to facilitate the superimposition process.

• Skull-face overlay: Almost every author has created their own methodology to project both images, using craneoforos, images with transparencies, or other mechanisms that allow placing the skull and facial photographs in the same position. In addition, some authors made use of anatomical landmarks to make the overlay process easier.

• Decision making: In most of the cases, the decision was based on subjective features due to the lack of soft tissue studies and the difficulty to perform measurements on photographs.

The scientific literature on the implementation of photographic CFS is presented in Table 4.1. Only the methodology followed within the stages of face enhancement and skull modeling and skull face overlay is summarized, as no significant contributions were made on the decision-making stage of the process.

Table 4.1 Review of the literature on photographic superimposition methods

In the following subsections, the different photographic CFS proposals and their contribution to each of the CFS stages are reviewed.

4.2.1 Face Enhancement and Skull Modeling

In this first stage, the initial steps are intended to select and/or obtain clear and measurable images. The most important factors affecting the quality of the portrait to be superimposed that have been taken into account are the selection of the ante-mortem image, the knowledge of all the technical details of the photographic equipment, the focal length, the distance to the camera, etc. (see Table 4.2). In the case of the skull, this could sometimes include working with fragmented remains. In particular, Cocks (1971) developed a photographic CFS methodology, which was also applied to fragmented skulls. Moreover, in order to obtain measurable images, a few workers have followed the example of Glaister and Brash (1937) in using objects of known size, external to the person, to establish a magnification factor for photographic enlargement, presumably because such objects are not always present or clear in ante-mortem photographs (see Fig. 4.1). When present, features of articles of clothing that lie in roughly the same focal plane as the face have been used. Enlargements of snapshots or photographic portraits based on the dimensions of the linear pattern of a tie (Gordon and Drennan 1948), the pattern on the border of a sari (Sekharan 1971), and the diameter of a button on a sweater (Janssens et al. 1978) have yielded successful superimpositions and positive identifications. Measurements of a wooden chair present in a photographic portrait also yielded a magnification factor from which an accurate enlargement was made (Sekharan 1971). In the absence of objects of known size in ante-mortem photographs, several workers have combined the use of anatomical landmarks and anthropometric measurements of the facial skeleton with so-called established values for thicknesses of soft tissues to estimate a magnification factor and, thus, obtain a superimposition by best fit. Among anthropometric measurements, the most common references were the interpupillary distance (Simpson 1943; Reddy 1973; Sekharan 1971) and the features of the anterior dentition (McKenna et al. 1984; Sognnaes 1980).

Table 4.2 Review of the literature on video superimposition methods

Fig. 4.1

Case examples. On the left, the Ruxton case photograph showing the tiara used as a measurable reference object (Grüner 1993). In the center, a snapshot of deceased Michael Wolkersdorfer in which the tie was used as a scale (the image was taken from Gordon and Drennan (1948). On the right, a superimposition of transparencies and tracing of skull Z6 with photograph of victim 3 giving a possible elimination (the image was taken from McKenna et al. (1984)

Figure 4.1(right) shows an example of the latter reference feature. The use of landmarks became more important at the end of the century.

Once the photographs have been selected/obtained and any measurable references have been marked, there are still some tasks to be developed in order to facilitate the SFO stage, for example, trace drawings of the head outline have been quite helpful in many of the solved cases. Pearson and Morant (1934) took a photograph of an Egyptian criminal before his execution to be compared with the skull subsequently prepared after his death. The authors made tracings of the face and skull photographs so that they could superimpose them. This work is considered as the first superimposition using a photograph of the skull and tracings of the face. Glaister and Brash (1937) used a full-size transparent portrait outline on the viewing screen of the camera. They transferred the head outline of both the face and skull photographs to tracing paper. Gordon and Drennan (1948), Cocks (1971), and Reddy (1973) also made use of tracings before performing SFO. In this way, Scully and Nambiar (2002) identified a number of factors, which, in their opinion, directly influenced a positive superimposition. He said that the selection of a good quality ante-mortem photograph is of primary importance to get the most reliable result. Poor quality images can result in poor definition, because the ante-mortem photograph usually has to be enlarged, making the determination of certain features difficult. The expert must choose photographs in which the face under study appears as near as possible to the center of the picture. Use of the outer extremities of the frame should be avoided due to barrel or pin cushion distortion, as the proportions of the face will be considerably distorted. Therefore, the author advised that the face should be in good focus, well lit, and not in partial shadow. Scully and Nambiar (2002) advised that the exact photographic conditions under which the ante-mortem photograph was taken should be discovered: the distance at which it was taken, the focal length of the lens used to record the image, even the settings used on the darkroom enlarger to produce the final print may be of value. The surface to which the ante-mortem photograph is attached should also be exactly parallel to the image plane of the camera. The author affirmed that if these factors are unknown, the final result is negatively affected.

4.2.2 Skull-Face Overlay

There is no common procedure used to carry out SFO and each author tries to do this depending on the equipment they have and their knowledge of the technique. Identification through photographic superimposition usually shares the following two steps: (1) determination of the correct life-size; (2) replication of the same orientation of the face in the photograph when photographing the skull. To perform these tasks, a diverse set of elements (X-ray, negative and positive photographs, outlines, and transparencies), apparatus (light stand, optical bench), and methodologies (measurable objects, distance between landmarks or anthropometric measures, triangulation based on the landmarks and transparencies, asymmetrical features of the facial skeleton) are employed (a summary is presented in Table 4.2).

Glaister and Brash (1937), while addressing the Ruxton Case, adjusted the skull orientation to a similar head position as the one the face showed in a portrait. They used tracings of the head outline in both the face and skull photographs and then superimposed them. The tracing paper with the superimposed outlines was positioned onto the positive print photograph and the paper skull negative. The transparency was then superimposed using different landmarks (prosthion and nasion)and the headoutlines.The positive portrait and the negative skull print were photographed again on x-ray. A positive skull portrait and a negative face portrait, each one with the registration marks photographed, were obtained as a result. Finally, two transparencies were superimposed using the central marks and were photographed again on the x-ray film. This methodology was the basis for later modification by numerous authors.

Simpson (1943) executed a superimposition with an x-ray portrait. The width was standardized between the eye sockets and the full portrait was then superimposed onto the x-ray skull photograph.

The first step in the Wolkersdorfer case, Gordon and Drennan (1948), was to obtain the skull and face images in life-size. They used the natural tie size as a scale. The head outline was drawn and a machine was used to superimpose the skull drawing over the life-size head drawing. This superimposition showed a good anatomical correspondence.

In Webster (1955), the author obtained a 12″ × 10″ photograph of 90 people, within which the victim was present. The equipment was composed of a Ross half-plate camera equipped with a 5-in. Xpres f.4 wide angle lens. They created the deceased’s face negative from the photograph, which was enlarged approximately eight times the size of the original picture. The skull was then oriented in the same position as the face. The negative was placed on the ground glass² and three points (center of nasal bone, extreme prominence of the left malar bone, and the inner end of the left supra-orbital ridge) were marked to obtain the correct alignment of the two images (see Fig. 4.2).

Fig. 4.2

From left to right: (a) enlargement of the deceased’s face and head (actual size); (b) the skull positioned to correspond with the points marked on the camera; (c) final result of the superimposition of the first two images (the image was taken from Webster 1955)

Gruner and Reinhard (1959) designed a light stand and an optical bench for positioning the skull and adjusting the photographic angle, with the aim of making an accurate superimposition, taking the soft tissue thickness into consideration. They stipulated that the correct distance between skull and camera was 1.75 m. The author took a negative of the skull, which was then enlarged. Finally, he printed it as a single image to be used in the overlay.

Cooks (1971)’s methodology takes a photograph of the skull and compares it with a passport picture of the suspected victim. To obtain a good correspondence between the skull and the portrait, the author developed a system of triangulation based on landmarks (nasion, anterior nasal spine, gnathion, orbit points, and mandible) marked in both images. The triangle patterns were drawn onto the transparency looking for correspondence between the superimposed images.

In Reddy’s method (Reddy 1973), the outlines of the salient features of the face are drawn onto the ground glass and the skull is then positioned equally to the face in the portrait thanks to a skull-rest. Those outlines are then matched with the skull, making allowance for the soft tissue covering the bone.

Furue’s method (Klonaris and Furue 1980) represented a breakthrough in craniofacial photographic superimposition. In this study, the authors give importance to photographic perspective. They developed a system taking into account the distance between camera and subject, and they try to replicate conditions similar to those of the ante-mortem photograph. In the first step, the skull was positioned on an adjustable stand in front of a backdrop sheet of contrasting colored cardboard at one end of a bench and a 30-cm grid of 9 mm was positioned in front of skull. A single lens reflex camera was set at the opposite end of the bench and, on the other side, a life-size enlargement of the ante-mortem photograph was positioned behind a grid of 10 mm². Two mirrors were placed such that the full mirror reflected the image of the photograph upon the half-silvered mirror along the central axis of the camera. Two lamps were positioned between the skull and the camera. The images reflected in both mirrors were transmitted to the cameras, and the images of both the skull and the portrait could be seen through the camera viewfinder. The skull is aligned taking into account the perspective incorporated in the ante-mortem image, and the skull image was recorded on the camera film following removal of the half mirror. Finally, they obtained a victim image negative print and skull positive print, which were superimposed.

Meanwhile, Dorion (1983) used a Graflex camera 4 × 5 with a 135 mm lens atf-4.5, and a pivoting head that permitted placement of the skull in the same orientation as the photographic transparency. Both are positioned in front of the camera at a distance of 95.25 cm. A beam splitter was used for varying the intensity of the mounted quartz lights in front of the skull to get a superimposed image viewing through the camera. He made a correct image magnification considering factors such as the distance of the subject from the camera, which was 95 cm with the lens. Then, he enlarged the photograph to a 1:1 ratio using reference landmarks on the skull.

McKenna et al. (1984) selected a photograph in which the maxillary anterior teeth are shown. He used a “phantom-head” device that permits skull movements in different planes. The negative of the photograph was enlarged until the dentition coincided with the dentition of the skull image. Skull features and facial contours traced in the negative portrait were superimposed on a radiographic viewing box. A similar methodology was used by Brocklebank and Holmgren (1989) who developed new equipment with minor advantages.

Maat (1989) explained that an optimal result is possible if the distance between camera and the skull is the same as the distance from which the original photograph was taken; however, this is difficult to achieve. Some authors have established that portraits are taken at a safe distance, if they are taken at a distance of 1.5 m or more (Gruner and Reinhard 1959). On the basis of the above, Maat enlarged a skull image taking into account that all visible landmarks of the superimposed image must cover, in the best possible way, their equivalent points on the original portrait. In particular, he used a geometrical method to calculate projections of anthropometric distances, angles of rotation, and inclination to define the posture of the head or skull with respect to the anatomical frontal plane. He proposed to use a set of anthropometrical landmarks, along with relative reference lines, to calculate the three components of head rotation (“bending forward,” “turning sideways,” and “rolling sideways”) to position the skull. The principle of central projection and a minimum photographic distance of 1.5 m are important preconditions.

Lan (1990) used the distance between ectoconchions to determine the natural head size in the photograph. He had considered differences in the deflection angle with a mechanism that examined eight lines capable of determining the image proportions. In contrast, in some other cases, asymmetrical features of the facial skeleton (Prinsloo 1953; Sivaram and Wadhera 1975) or the presence of a supernumerary anterior tooth (mesiodens) (Asananer 1972) has facilitated identification by superimposition.

Scully and Nambiar (2002) determined the validity of Furue’s methodology. They photographed the skull following similar conditions to those indicated by Furue. Despite Scully et al. using digital images, the result was similar to the conventional direct photography print used in Furue’s methodology.

4.2.3 Decision Making

This phase consists of the systematic evaluation of the correspondence of morphological features between the face and skull. However, most authors have described anatomical landmarks and anthropometric measurements, which, in their opinion, were more significant than others in the determination of a positive identification.

Before the use of photographic superimposition in the identification of victims, several attempts were made to fit tracings or photographs of skulls to painted or photographic portraits of famous or infamous people from history (Glaister and Brash 1937; Krogman and İşcan 1986). Lander compared the skull of an individual of known age, race, and sex with a photograph of the individual, but concluded that “It seems improbable that any one examining the skull would postulate a type of face similar to that seen in the photograph”(Prinsloo 1953).

Some years later, the superimposition of a photograph of the skull and tracings of the face, achieved by Pearson and Morant (1934), were considered to be satisfactory. However, they recognized that such a positive result would not be possible when comparing a skull with an artist’s portrait.

In the Wolkersdorfer Case (Gordon and Drennan 1948), the identity of the victim was established by overlapping a drawing of the available parts of the face with the outline of the head obtaining a positive anatomical correspondence. In the Plumbago Pit Case (Webster 1955), the correspondence between the skull and portrait was positive, and the following points were establishing as being significant in determining the final result from the superimposition: (1) the length of the skull as measured from the nose to the chin; (2) the length of the skull from the top of the skull to the chin or nose; (3) the margin of the eye-sockets; (4) the width of the temple (bi-temporal width); (5) the width of the face as measured from one cheek bone to the other; (6) the position of the mastoid processes; (7) the position of the midpoint of the upper jaw between the sockets for the central incisor teeth, and its relation to the nose; (8) the position of the teeth in the upper jaw on the left side; and (9) the position of the angle of the jaws.

Reddy (1973) overlapped the negative of a photograph and skull by aligning the following characteristic points: the eyes within the two pairs of orbital plates, the nasion, the prosthion, the nasal spine in the center, which is a little above the tip of the nose, the lower border of the nose, the lower border of the upper jaw, and the zygomas below the eyes. The expert observed from the superimposed image that all these points properly matched, making it possible to establish the identity of the victim. He considered that superimposition may provide significant evidence in Court, affirming that it could solve medico-legal problems, such as the identification of an individual from the skull.

Dorion (1983) obtained a positive identification in two cases, comparing soft tissue outlines to bone and dental characteristics, although he considered that photographic superimposition does not serve as the only basis for a positive identification and it should be corroborated by other identification methods or circumstantial evidence.

McKenna et al. (1984) drew clearly defined features of the skull on matte acetate, from the transparency of the skull. The facial contours were traced from the transparency of the enlarged snapshot. The transparencies and tracings were superimposed, and the correspondence between dental landmarks of the skull and face were verified. The photographic superimpositions and composite tracings were presented in court and were accepted as a positive identification. Finally, the author indicated that photographic superimposition could be established not only as a complementary method of identification, but as a means of a positive identification in its own right.

Maat (1989) determined that only a limited number of cephalometric landmarks of the head lie close to the craniometric landmarks of the skull, such as nasion-nasion, subnasale-nasospinale, gnathion-gnathion, gonion-gonion, a point on the lower border of the eyebrow-orbitale superius, and tragion-porion. He suggested that, taking individual variability into consideration, it is very unlikely that someone else other than the victim reflects all those features.

Scully and Nambiar (2002) consider that photographic superimposition can be used as the only means of confirming the identity of an unknown skull, using the craniometric and somatometric landmarks, racial features, soft-tissue thickness, scars, or injuries as a basis. They affirmed that knowledge of craniofacial anatomy is a crucial factor for establishing the identity of the victim.

Krogman and İşcan (1986) recommended the use of photographic CFS whenever possible, arguing that it can provide important corroborative and possibly conclusive evidence for identification.

4.3 Video Craniofacial Superimposition

The common components of almost all video superimposition systems include two video cameras, an electronic mixing device, and a TV monitor. Instead of making photographs, tracings, or drawings in order to properly superimpose the skull and the face, the video cameras provide focused “live images” of the objects (skull and photograph). These systems present a great advantage over the former photographic superimposition procedures by minimizing several problems associated with the photographic systems. For example, the number of manual manipulations and the length of the time required in superimposition are dramatically decreased since an expert can control the matching of two different images on the TV monitor. Nevertheless, the processes of skull orientation and sizing of the ante-mortem photograph and the skull in video superimposition remain troublesome. The face enhancement and skull modeling stage in this modality is very similar to the one in the photographic superimposition technique. For example, Lan and Cai (1985) advocated the use of the same orientation procedure employed in photographic superimposition.

In this section, we will review the existing contributions of video CFS systems. They will be classified according to the stage of the process, which is addressed using a video system. Information about the method used for the remaining stages will be given shortly, along with a brief discussion. Unfortunately, in some of the existing video superimposition methods, these stages are not as clearly distinguished as we might expect. This fact can cause some confusion, as sometimes the authors themselves define their own method as photo, video, or computer-aided CFS when they are actually referring to one or part of one of the three general stages of CFS. In order to be consistent throughout this document, we decided to limit this section to the methods in which a video system is the core apparatus of the superimposition method and there is no use of computers. There are a set of methods (Lan and Cai 1985, 1993; Yoshino et al. 1997) that use video cameras to acquire images of the skull and the face photograph but also make use of computers to perform important tasks within the whole process. Thus, these “hybrid” methods will be reviewed in Sect. 4.4.

As in photographic superimposition, Table 4.2 presents a review of the papers describing video superimposition according to the different stages.

In the following subsections, the different video CFS approaches and their contribution to each of the CFS stages are reviewed.

4.3.1 Face Enhancement and Skull Modeling

As in photographic superimposition, in this stage, the first steps are intended to select and/or obtain clear and measurable images. Therefore, a photograph, which is overexposed or underexposed, blurry, or noisy, will result in a low-quality superimposition (Nickerson et al. 1991). The objectives of image enhancement for video superimposition can be enumerated as follows: obtain measurable images; know the object-subject distance; select the correct ante-mortem photographs; choose the photograph nearest to the moment of disappearance; avoid photographs that show chiaroscuro; know all technical data of the equipment; and filter the image to produce as much useful image information as possible. In order to obtain measurable images, most of the authors used anatomical landmarks and anthropometrical measurements of the skull and the face. The procedure to mark them varies among authors. Some authors, such as Dorion (1983), placed the landmarks on the transparency using movable white cardboard, which blocks out the skull’s characteristic for the same region. Others, for example Bastiaan et al. (1986), located anatomical landmarks on the skull by placing a white pointer in or around an area so that each point can be accurately viewed. Similarly, Fenton et al. (2008) placed tissue depth markers on the skull.

The knowledge of all the technical data of the equipment used is shown to be helpful for the superimposition process. All components can affect the final quality of the results achieved. For example, Brow (1983) replaced the video cameras by semiprofessional Ikegami cameras, considerably increasing the quality of the image.

Iten (1987) said that the first step in superimposition is to determine the correct height and location of the camera. This must coincide with the photographs that the skull is going to be compared with and is defined by a number of variable parameters such as the distance between the camera and object, camera height, and orientation of the skull.

Austin (1999) stated that the quality of the photographs is essential. He received mug shots, driver’s licenses, and family photos to make an identification. Driver`s license photographs are usually full face and benefit from the enlargement capabilities of the equipment. However, mug shots and family photographs often represent varying angles to the camera film plane. The author suggests using at least two photographs with an attempted difference of 90° between them. He recommends avoiding images with obstructing factors (facial hair, long hair, hats, etc.). The facial photographs must show useful morphological characteristics of the person to be identified. Finally, he examined all photographs with a stereomicroscope to ascertain the amount of grain, clarity of features, and shadow details.

4.3.2 Skull-Face Overlay

The next step consists of the comparison of both images. Skull orientation can be performed in the same manner as in photographic superimposition; however, the correct size of the skull is easier to achieve by adjusting the size of the skull using the zoom mechanism of the video camera (Yoshino 2012).The method of skull orientation has varied with the incorporation of new tools, apparatus, and mechanisms (see Table 4.2): Helmer and Grüner (1977a) were probably the first researchers to introduce the video superimposition technique. They substituted a photographic camera with a video camera in the CFS procedure. The equipment consisted of two video cameras, an electronic mixing unit, and two TV monitors. This allows for recording and displaying the skull and the face portrait at the same time. In addition, it also offers the possibility of mixing both images, thanks to the mixing unit. Consequently, this approach significantly reduced the time needed to enlarge the ante-mortem photograph and the skull to the same size. It also presented an outstanding mechanism to help in the proper orientation of the skull. Both were the most time-consuming tasks in the previous photographic-based approaches.

Koelmeyer (1982) used three video cameras to reproduce the facial images, the skull x-ray, and the partially reassembled skull to perform the superimposition between the images. In addition, he made use of the sweep fade-in/out mechanism to help guide the superimposition of the skull portrait and face photograph on both vertical and horizontal axes.

Dorion (1983) replaced a Graflex camera with a video camera. He placed it at a distance of 92.25 cm from the zygomatic process³ of the skull and a face transparency was put in front of the skull. Similarly, Chee and Cheng (1989) used a transparency of the face photograph positioned in front of the skull and recorded both images on the monitor. The face photograph was enlarged to natural size and photocopied onto the transparency.

Bastiaan et al.’s (1986) system incorporated an adjustable support to mount the skull. This allowed movement in three planes. Two video cameras (Hitachi GP-5 colour) were aligned with the ante-mortem portrait and the skull, respectively. Both cameras must be compatible with “Gen-Lock” (which consists of the ability of two video cameras to synchronize together in order to use the facility of video or vision mixing) to obtain synchronized vision mixing. The skull is oriented in the same angulation as the head of the subject in the ante-mortem portrait. They proposed to enlarge the ante-mortem photograph until the teeth in the superimposed video picture overlap. If the teeth are not visible in the face, an estimation of the enlargement factor must be employed. Finally, both photographs were superimposed taking into account different anatomical landmarks (the external auditory meatus, orbits, anterior nasal spine, chin point, angles of the mandible, and zygomatic processes). These points were highlighted on the skull by placing a white pointer in or around the relevant area so that each point could be accurately viewed. A video mixing unit allowed a fade-in/out of both images in the vertical or horizontal plane.

The equipment employed by Iten (1987) consisted of two video tubes: an electronic mixer unit and three monitors. The skull is recorded with video tube1 and reproduced on monitor 1. Then, the photograph is recorded with video tube 2 and reproduced on monitor 2. Both images are mixed on monitor 3, to be appraised and recorded. Then, the zoom mechanism of the video camera is used to adjust the skull image to facial image size on the monitor, using soft-tissue thickness as reference points. Finally, the exact orientation of the skull image is accomplished by adjusting the outlines and anatomical landmarks between face and skull images. For this purpose, Iten used the ratio of the distance between the eyes and the distance between the eye and auditory canal as axes.

McKenna et al. (1984) used two video cameras and two monitors to record images of the skull and the ante-mortem photograph, respectively. They presented a method that aligned the skull through a distinctive anatomical point (e.g., the line between the central incisor teeth at their incisal tips) and clamped it into position. A video camera is focused through a transparent lens cap sight on this reference point and is secured in position. Then, it projects the image of the skull onto a television monitoring screen. A second video camera, focused on the ante-mortem photograph, also projects the corresponding image to the monitor screen. Finally, they made the superimposition using a special effects generator, which allowed the mixing of both images.

Brocklebank and Holmgren (1989) developed novel equipment consisting of a skull-holding jig and a “pan-and-tilt” device to position the skull by virtue of specifically calibrated scales and markings provided on the apparatus. A camera was positioned on an adjustable mount running on twin parallel rails, allowing the camera distance to be varied. The equipment allowed the correct replication of the skull position with the face in the photograph. Finally, the face and skull photographs could be evaluated to determine whether it was a positive or negative superimposition.

Sekharan (1993) determined antero-posterior tilt with a mechanical device, but she added two parallel wires which show the distance and the different landmark planes. She suggested using the vertical distance “d” between the ectocanthions and tragion as a measure for calculating the extent of flexion or extension of the head. The extent of the rotation of the face was calculated from the L/R ratio, where L and R denote the distances between the left and right ectocanthion from the midline of the face. Using these factors, the skull under examination was positioned on a tripod stand with the help of a remote control positioning device (Kumari and Chandra Sekharan 1992).

Shahrom et al. (1996) presented a method to superimpose a skull and a facial photograph, which was very similar to previous approaches. In order to determine the position of the skull, the facial photograph was rotated in the coronal plane to match the skull angulation. Manipulation of the skull position or angulation using the skull holder was only conducted in two planes (sagittal and horizontal). Then, two different video cameras captured the facial photograph and the skull. The video images were mixed into one on the television monitor, and then, the expert determined the relationship between landmarks on the skull and the face.

Iten (1987) developed three different methods of video superimposition for forensic case work: skull/photograph superimposition, radiographic comparison, and photograph/photograph superimposition. These procedures usually involve a still photograph of the skull, which is overlaid with the facial photograph, two video cameras, and a mixing device to superimpose the two images. The video superimposition set-up proved useful for ante-mortem and post-mortem radiograph comparisons. He used the video superimposition equipment to compare scene photographs with known photographs of a suspect. In Austin-Smith and Maples (1994), the superimposition process was presented as follows: A photograph was placed under one of the cameras and the face was focused so that it filled the monitor screen as fully as possible. Tissue thickness markers were positioned on the skull at the appropriate anatomical landmarks. The tissue thicknesses used were determined by ultrasound on living subjects and ranged from thin to obese individuals. The position of the head in the photograph was scrutinized, and the skull was placed on a cork ring to estimate that position. The distance between a horizontal projection from the lateral angle of the eye and another from the external auditory meatus was approximated. The skull was put under the second video camera, the size was adjusted so that the tissue thickness markers fell within the outline of the face, and the proportions of the anatomical features were maintained. Exact positioning was determined by trial and error manipulation of the skull.

Solla and İşcan (2001) modified the previous system. In this case, the facial image was reproduced and digitized in the video mixing unit. A plastic sheet with different landmarks traced was taped to the monitor. The orientation of the skull was adjusted based on these landmarks, taking soft tissue into account, and manipulated manually to get the same position as the face in the portrait. Then, they used a video camera zoom to adjust the size of both images. Both images were then reproduced on a TV monitor for detail comparison. The authors describe the video mixing unit as a device that allows any desired combination of photo-skull comparison, including removing the soft tissue to view the underlying skeletal structures such as the auditory canal, eye sockets, cheekbones, jaw bones, root of nose, teeth, chin, skull contours, and so forth.

Fenton et al. (2008) employed two video cameras, a video cassette recorder, a video mixer, TV monitor, and software similar to that recommended by Austin-Smith and Maples (1994). Their video superimposition process began by placing appropriate tissue depth markers on the skulls. Once this was completed, the “dynamic orientation process” was used to arrive at the best fit possible in the alignment of the skull with the ante-mortem photo.

The dynamic orientation process involves positioning of the ante-mortem photograph under one of the two video cameras so that the image fills most of the TV monitor. Next, the skull is placed under the other camera. Using the mixer and monitor, the skull image is sized so that it can be superimposed onto the image of the face. Once the skull and ante-mortem photo are satisfactorily adjusted for size and basic orientation, the craniofacial proportions of the skull and the face can be evaluated and compared. This is accomplished by manually adjusting the skull so that the key skeletal landmarks align with corresponding landmarks on the face. The set of landmarks employed for this purpose are: porion, right and left Whitnall’s tubercles (to be aligned with right and left ectocanthion points of the face), subnasal points, and gnathion.

In conclusion, the general procedure of video superimposition presents important similarities among the different approaches; however, the techniques or methodologies have varied with the development and incorporation of new tools, apparatus, and mechanisms, which have reduced the problems of the orientation and size of the skull with respect to the facial photograph.

4.3.3 Decision Making

Once SFO is fulfilled, the decision-making process is carried out. The main tools involved in the decision-making stage include fade-in/out, video mixing, or special effect generators (see Table 4.2). Nevertheless, the experience of the forensic anthropologist continues to be of paramount importance in the determination of identity.

Koelmeyer (1982) used three video cameras to reproduce the skull, the facial photographs, and a radiograph. Then, the use of a switcher joystick to fade-out, fade-in, and sweep allows a progressive superimposition of the radiograph, and the skull image on the facial photograph, on both vertical and horizontal axes. As a result, the system creates a more pleasing image and makes a better comparison of bony landmarks possible.

Bastiaan et al. (1986) considered different anatomical landmarks for the decision stage: the external auditory meatus, orbits, anterior nasal spine, chin point, angles of the mandible, and zygomatic processes.

Helmer and Grüner (1977a)’s system allows the division of the superimposed image into various sections that show either the skull or the face. It also provides control over the transparency of both the skull and face images on the TV monitor. Both have resulted in being very helpful for the overall assessment of the anatomical correspondence between the skull and the face. Many authors employ this video equipment with similar configurations and procedures. In particular, Sekharan has carried out positive identifications in 140 cases since 1985 using this methodology (Sekharan 1988). Half of the skull image and half of the photograph can be displayed simultaneously using a special effects generator. Sekharan assured that observation, comparison time, and judgment error could be reduced by a significant rate.

Iten (1987) indicated that one can begin comparing the facial image with that of the skull, once the enlargement and orientation procedures are complete. The comparison is carried out on a monitor that allows the user to mix and produce picture sections in order to compare all anatomical landmarks and morphological features such as the auditory canal, eyes socket, cheekbones, jawbone, root of the nose, teeth, chin, and skull contour. The same procedure described above was performed by Shahrom et al. (1996).

McKenna (1988) used a special effects generator that enables the size of the images of the skull and photographs to be matched by superimposition of the respective dentitions. This allows the points to be easily placed. McKenna adds that the video camera can be replaced by a photographic camera at the same radial distance and the same angular axes. Therefore, overlay transparencies can be prepared from the photographic camera negatives for presentation in court, considering that it allows a static display of the final result of the overlap without the distortion created by the monitor screen.

Brocklebank and Holmgren (1989) marked the outlines of all relevant features of both transparencies on tracing film for an assessment of the degree of anatomical consistency. Suitable hard and soft tissue features included the following: all dental features; outlines of bone margins and corresponding soft tissue features; interpupillary line and midorbital line; and the midline of the face and skull. Then, superimposition of both transparencies took place in order to assess whether there was a positive match or not, and the level of confidence in the resulting identification.

Austin (1999) claimed that the experience of the analyst and the integrity of the equipment are very important. He affirmed that it is necessary to know the relationship of soft tissue with face bones, achieved through the study of known skulls with their own photographs.

Fenton et al. (2008) used the same 12 morphological features, in frontal view, as Austin-Smith and Maples (1994), for establishing a consistent fit between the face and skull. By evaluating facial proportionality and by comparing morphological features of the face and skulls, one skull was excluded as a possible match and one skull was not excluded as a match to the ante-mortem photo.

The most advanced video superimposition systems also make use of computer software that allows the user to quantitatively evaluate the adjustment between a skull and a facial photograph in two and three dimensions.

4.4 Computer-Aided Craniofacial Superimposition

The differentiation between methods that use computer technology and those that do not has already been proposed (Aulsebrook et al. 1995). In the literature, photographic and video superimpositions have been considered to belong to the former category. Meanwhile, methods defined as digital or computer-aided CFS have been considered to belong to the latter. Thus, the distinction between computer-aided and non-computer-aided methods has been clearly guided by the use of computer-based technology within the CFS process up to now. Nevertheless, the role of the computer in this process is presently very important and it has not been considered in previous reviews. Moreover, the analysis of previous contributions is especially difficult when some authors claim they propose a “computer-aided” or “computer-assisted” system (Ricci et al. 2006) yet the computer mainly plays the role of a simple visualization tool.

Whenever a computer is employed as part of the CFS system, the method should be considered a computer-aided technique. The following classification better reflects the state of the art regarding the use of computers in one or all stages of the process and the interaction between various technological approaches: (a) computer-aided craniofacial photo superimposition, (b) computer-aided craniofacial video superimposition, and (c) computer-aided craniofacial 3D-2D superimposition.

Additionally, the classification should differentiate between nonautomatic and automatic methods (Damas et al. 2011). Computer-aided nonautomatic methods use some kind of digital infrastructure to support the CFS process, that is, computers are used for storing and/or visualizing the data. However, they are characterized by the fact that their computational capacity to automate human tasks is not considered. On the other hand, computer-aided automatic methods use computer programs to accomplish an identification subtask itself.

Tables 4.3, 4.4, and 4.5 give an overview of the papers describing computer-aided systems, classified into the three categories already detailed. Studies are listed in chronological order. Additional information about the degree of automation and the input data required is provided where available.

Table 4.3 Overview of computer-aided photo superimposition systems

Table 4.4 Overview of computer-aided video superimposition systems

Table 4.5 Overview of computer-aided 3D-2D approaches

There are some remarks that should be pointed out concerning the three stages of the process:

• Regarding the first stage, automatic methods may deal with either the 2D image of the face or the skull. On the one hand, when dealing with the 2D image of the face, automatic systems accomplish the restoration of the photograph by means of digital image-processing techniques. On the other hand, the aim of automatic methods concerning the skull is the achievement of an accurate 3D model.

• Concerning the second stage, Damas et al. (2011) pointed out a clear division between computer-aided nonautomatic and automatic SFO methods. The former use computers to support the overlay procedure and/or to visualize the skull, the face, and the obtained superimposition. Nevertheless, the size and orientation of the skull are changed manually to correctly match that of the head in the photograph. This is achieved by either physically moving the skull, while computers are simply used to visualize it on the monitor, or (with the help of some commercial software) by moving its digital image on the screen until a good match is found. The automatic SFO methods find the optimal superimposition between the 3D model of the skull and the 2D image of the face using computer programs.

• Finally, regarding the decision-making stage, automatic systems assist the forensic expert by applying decision support systems (Keen and Morton 1978). Moreover, these computer programs use objective and numerical data for evaluating the obtained matching between the skull and the face. Based on that evaluation, the system suggests an identification decision to the forensic expert. Thus, the decision support system is intended to help decision makers compile useful information from the analysis of the SFO outcomes. Of course, the final decision will be always made by the anthropologist according to both the support of the automatic system and his expertise. On the other hand, if the identification decision only relies on the human expert who visually evaluates the superimposition obtained in the previous stage, then the method will be considered as a nonautomatic system, although it might use digital data as a supporting means.

4.4.1 Discussion of Existing Works

In this section, we will review the existing contributions of computer-aided CFS systems. They have been classified according to the stage of the CFS process which is addressed using a computer-aided method. Information about the methods used for the remaining stages will be given together with a brief discussion.

Unfortunately, in some of the existing CFS methods, the stages are not as clearly distinguished as we might expect. This causes some confusion as sometimes authors themselves define their own method as computer-aided CFS when they refer only to the decision-making stage and others refer to the identification method when they tackle the SFO stage.

4.4.2 Face Enhancement and Skull Modeling

Let us highlight the main differences between the image of the face and the model of the skull. The face image is typically a photograph that has been acquired under fixed conditions that are usually unknown at the time of the forensic analysis. With a digital image, the only possibility is to attempt to enhance its quality. If it is not in digital format, it can be scanned and transformed into a 2D digital image. Then, it can be enhanced using digital image filters and/or processing algorithms. However, the skull is an available physical object and a model of it needs to be obtained to allow for an automatic procedure.

We will detail both face enhancement and skull modeling procedures. Regarding the image of the face, good quality is needed (Nickerson et al. 1991); therefore, enhancement techniques should be applied (Gonzalez and Woods 2008). Such techniques depend on the available format (digital camera image or scanned photographic paper) and include frequency domain filters to fix artifacts due to aliasing and sampling problems present in scanned documents, as well as removal of nonuniform illumination effects and sharpening methods to deal with blurring and problems related to movement. Notice that the choice of a proper filter and its most suitable parameters must be performed by the expert, since they depend strongly on the acquisition conditions. As explained above, approaches that use human-operated commercial software for the 2D face image enhancement will be considered nonautomatic methods. Automatic methods perform such 2D image enhancement using computer programs with almost no human intervention.

Regarding the model of the skull, recent techniques for CFS need an accurate 3D model. In the biomedical field computed tomography, scanning images are used as the starting data for reconstructing the skull (Singare et al. 2009; Fantini et al. 2008). However, the possibility of recording 3D forensic objects is limited, considering the available resources of a typical forensic anthropology lab. Indeed, nowadays, many forensic labs are exploiting the capabilities of laser range scanners. That is due to the fact that these devices present a greater availability and a lower cost. Thus, we will consider the possibility of extending the study to other devices that have also been used for obtaining a 3D model of the skull in other application domains (Nakasima et al. 2005; Enciso et al. 2003). Laser range scanners are based on the optical principle of triangulation and acquire a dense set of three-dimensional point data in a very rapid, noncontact way (Bernardini and Rushmeier 2002). Some laser range scanners are equipped with an additional positioning device, such as a rotary table, and appropriate software that permits the 3D reconstruction. Nevertheless, there are situations where the software does not provide suitable 3D models. Moreover, there are scenarios where it is not even possible to use a rotary table.

Before continuing with the 3D modeling process, every 3D view of the skull acquired by the laser range scanner must be preprocessed. This task involves the cleaning, smoothing, and filling of the view. Cleaning aims to remove those artifacts that were acquired by the scanner as part of the scene but which do not correspond to the skull. Meanwhile, smoothing is mainly concerned with the removal of any artificial vertices that could have been wrongly included by the scanner on the borders of the surface because of perspective distortion. Fortunately, this task is not needed as often. Finally, filling is used to avoid small holes appearing in the parts of the skull that are not properly scanned, because they are too dark for the scanner capabilities or because they are located in shadow regions.

Some anthropologists are skilled enough to deal with the set of 3D views and supervise the creation of the 3D model using commercial software like RapidForm^TM. Sometimes, this software does not provide the expected outcomes and the anthropologists have to stitch up every couple of adjacent views manually. Hence, 3D image reconstruction software is a real requirement in the construction of the 3D model, as it allows the views to be aligned in a common coordinate frame. This process is usually referred to as range image registration (Brown 1982; Ikeuchi and Sato 2001; Zitova and Flusser 2003). It consists of finding the best 3D rigid transformation (composed of a rotation and a translation) to align the acquired views of the object. An example of three different views of a skull and the reconstructed 3D model is shown in Fig. 4.3.

Fig. 4.3

Three different views of a skull and the reconstructed model

In this section, we will mainly focus on contributions that include an automatic3D modeling procedure, because the other methods do not consider this stage and directly acquire a 2D projection of the skull (i.e., a skull photo). As said, all the approaches that use computers but do not consider the 3D skull model will be considered as nonautomatic methods (Yoshino et al. 1995; Ghosh and Sinha 2001; Pesce Delfino et al. 1986; Ricci et al. 2006).

To our knowledge, Nickerson et al. (1991) were the first researchers to propose the use of a 3D model to tackle the CFS problem. In their work, a range scanner and a digital camera were used for 3D digitization of the skull surface mesh and the 2D ante-mortem facial photograph, respectively.

Well-known image-processing algorithms were used for image enhancement (median filtering, histogram equalization, Wiener filtering) (Gonzalez and Woods 2008). Rendering was completed through the use of computer graphics techniques. A feature-based algorithm to reduce the computational and memory complexities inherent in solid modeling was also described.

Shahrom et al. (1996) followed a similar approach based on the use of a 3D laser range scanner. The authors used a skull holder, which could be slowly rotated through 360° in a horizontal plane under computer control. The 3D model was later used in facial approximation.

A completely different approach is presented in Biwasaka et al. (2005), in which the authors examined the applicability of holography in the 3D recording of forensic objects. Holography is an optical technique capable of recording the 3D data of an object. Two types of images, real and virtual, can be recorded in a holographically exposed film or hologram. Two superimposition systems using holographic images were examined in order to evaluate the potential use of this recording method. The authors claim that the performance of holography is comparable to that of the computer graphics system, which consists of an image scanner, software, and a display unit. Moreover, they argue that it can even be superior to the computer technique with respect to the 3D reconstruction of images. The suitability of this technique requires further study. In particular, the use of an automatic superimposition method and a comparison with a reconstructed 3D range image could objectively prove the utility of holography in this field.

Galantucci et al. (2006) compared two different acquisition techniques of images of a skull. In particular, computed tomography and laser range scanners performance were compared to ascertain which enabled more accurate reproductions of the original specimen. Comparison between the original and each model yielded satisfactory results for both techniques. However, computed tomography scanning demonstrated some advantages over the laser technique, as it provided a cleaner point cloud, enabling shorter preprocessing times, as well as data on the internal parts, which resulted in the reproduction of a more faithful model.

Santamaría et al. (2007a, 2007b, 2009a) proposed a method, based on evolutionary algorithms (Bäck et al. 1997), for the automatic alignment of skull range images. Different views of the skull to be modeled were acquired by using a laser range scanner. A two-step pair-wise range image registration technique was successfully applied to such images. The method includes a prealignment stage that uses a scatter-search based algorithm (Laguna and Martí 2003) and a refinement stage based on the classical iterative closest point algorithm (Besl and McKay 1992). The method is very robust since it reconstructs the 3D model of the skull even if there is no turntable and the views are scanned incorrectly.

Fantini et al. (2008) used a laser range scanner to create a 3D model of a damaged medieval skull. The large missing part of the skull allowed scanning of both the outer and inner surfaces of the object. Thirty-three partial views were needed to complete the acquisition of the whole surface by rotating the skull. Through postprocessing of the data collected from the 3D scans, a triangular mesh was finally obtained. The operations were performed by RapidForm 2006, RE^TM commercial software.

A similar approach was followed in Benazzi et al. (2009) in order to tackle the 3D skull reconstruction of Dante Alighieri (1265–1321) as part of a project to achieve a facial approximation of the famous poet. Based on the data provided by a laser range scanner, the model of Dante’s skull was constructed using the utilities provided by the Rapidform XOS2^TMcommercial software. In particular, authors refer to operations such as registration and merging of the point clouds, as well as simplification and editing of the digital model.

Ballerini et al. (2009) presented a feature extraction algorithm, based on evolutionary computation (Bäck et al. 1997; Laguna and Martí 2003),to identify a subset of points that could improve the registration of multiple views and reconstruct an accurate 3D model of skull objects. They extracted a set of relevant features from point clouds acquired by a 3D range scanner. They overcame the trade-off between having a fully automatic method and using allow number of points located on meaningful features. The method detected regions close to boundaries and localized small and sharp features. The technique was robust when dealing with nonuniform sampled surfaces.

Ballerini et al. (2009) proposed the automatic reduction of data provided by the laser range scanner used in the skull 3D model reconstruction task. The dense point cloud corresponding to every skull view is synthesized by considering heuristic features that are based on the curvature values of the skull surface. Those features guide the automatic 3D skull model reconstruction by means of an evolutionary algorithm.

4.4.3 Skull-Face Overlay

The success of the superimposition technique requires the positioning of the skull in the same pose as the face. The orientation process is a very challenging and time-consuming part of the CFS technique (Fenton et al. 2008). Most of the existing CFS methods are guided by a number of landmarks on the skull and the face. Once these landmarks are available, the SFO procedure is based on searching for the skull orientation leading to the best matching of the set of landmarks. Scientific methods for positioning the skull had already been proposed before computers became largely available. As reviewed in Sect. 4.2, in some of the very early approaches (Glaister and Brash 1937), the enlargement factor is calculated based on linear measurements of items within the ante-mortem photograph, such as fabric patterns, buttons, ties, and other objects of known geometry (doors, chairs, etc.) (Sekharan 1993). Other scale correlation methodology has included measurement of the interpupillary distance and the size of the dentition (Austin-Smith and Maples 1994) used a geometrical method to calculate projections of anthropometric distances, angles of rotation, and inclination to define the posture of the head or skull with respect to the anatomical frontal plane. None of these methods are computer-aided, but they are somehow closer to such strategies than to the trial and error procedures. In these approaches, the skull is manually placed on a tripod; however, its pose is estimated using a mathematical procedure instead of a trial and error routine. The researchers calculated the head size and orientation in the photograph to position the skull in the same posture.

Within the group of computer-aided SFO contributions, we will differentiate between nonautomatic and automatic works as follows.

4.4.3.1 Nonautomatic Skull-Face Overlay Methods

Below, we describe SFO methods known as computer-aided methods in the literature. Nevertheless, following the Damas et al. (2011) proposal, we prefer to refer to them as computer-aided nonautomatic SFO methods. They are typical examples of the use of a digital infrastructure but without taking advantage of its potential utility as an automatic support tool for the forensic anthropologist. Notice that they depend on good visualization and overlay mechanisms to aid human operators. Hence, processes following this approach are prone to be time-consuming, hard to reproduce, and subjective.

Lan and Cai (1985) developed a CFS apparatus called TLGA-1, based on the principles of dual projection. During the following years, these authors further developed this system resulting in new subsequent versions, TLGA-2 and finally TLGA-213 (Lan and Cai 1988; Tao 1986; Lan 1990). The TLGA-213 system was composed of a TV camera, a computer, an A/D and D/A converter, a mouse, and the 213 system software library. The system calculated the pitch angle of the photograph of the face by measuring the ratio between distances in the vertical line segments glabella to nasion and gnathion to nasion. The natural head size was calculated from the distance between the ectocanthions and the deflection angle in the photograph. The latter parameters were iteratively computed and considered as a guide for the manually performed SFO.

Ubelaker et al. (1992) solved a huge number of cases submitted to the Smithsonian Institute by the FBI. Their software allows any desired combination of skeletal-photograph comparisons, including the chance to remove the soft tissue to view the underlying skeletal structure. It works on digitized images of both the face and skull and offers the possibility to assess the consistency between them. The identification procedure usually requires less than 1 h. It is not specified if this time includes the acquisition and SFO steps or only the decision-making stage. However, for the acquisition of the digital images, the authors visualize the facial photograph and trace anatomical landmarks on a plastic slide taped onto the monitor. Then, they visualize the skull and manually manipulate it to match the marked landmarks. The quality of the photograph and the proper orientation of the skull are claimed to be highly influential to the success of the technique.

The superimposition method of Lan and Cai (1993) is based on the radiographic recognition and labeling of landmarks on the face and skull. First, they placed a drop of lead on 28 landmarks on the face of the person to be examined. Then, they had the subject sit facing the radiographic film box and took X-rays in seven pitch angles and ten reflection angles, using the two ectocanthions on the skull as objective points with a certain subject-to-film box and X-ray unit-to-film plate distances. After the radiographs had been developed, measurements were taken and the distances between landmarks on the skull and face were calculated, as well as indices, and corrections for radiographic distortion of the measurements were made to make them conform to their actual values. From the above test, they obtained projectional plane data and index relationships between landmarks in related parts of the skull image and soft tissue of the face. They also calculated the displacement formulae and projectional data of different superimposed marking points at different angles.

Yoshino et al.’s skull identification system (Yoshino et al. 1997) consists of two main units, namely, a video SFO system and a computer-aided decision-making system. In the former, a pulse motor-driven mechanism and a video image-mixing device (with fade-out or wipe mode) are used to estimate the orientation and size of the skull to reach the pose in the facial photograph. Then, the skull and facial images are digitized, stored in the computer, and superimposed on the monitor.

Ricci et al. (2006) presented an algorithm to compare a facial image with a skull radiograph. They worked with pairs of 2D images and the superimposition was carried out by the human operator who marked anatomical points and adjusted them to match. The algorithm only calculates distances and thresholds in an automatic way, while SFO is completed manually.

The use of commercial software such as Adobe Photoshop™ has been reported by Bilge et al. (2003) and Al-Amad et al. (2006). They used the “free transform” tool to adjust the scale of the photograph of the face, superimposed over the skull photograph. The “semitransparent” utility allows the operator to see both images while moving, rotating, and resizing the overlaid image (see Fig. 4.4).

Fig. 4.4

Nonautomatic skull-face overlay based on Photoshop™

A similar approach was also used by both Scully and Nambiar (2002) and Ricci et al. (2006) to validate a classical method and to superimpose skull radiographs, respectively. The computer program Adobe Photoshop® 7.0 was also used in Takač and Pilija’s (2012) proposal. They reported a case of negative superimposition. Photographs of randomly selected young living woman were superimposed onto a previously discovered female skull. Digitized photographs of the skull and face were superimposed onto each other and displayed on a monitor in order to assess their possible similarities or differences. Special attention was paid to matching the same anthropometrical points of the skull and face, as well as following their contours. The process of fitting the skull and the photograph was usually started by setting the eyes in the correct position relative to the orbits. In this case, the gonions were positioned beyond the face contour and gnathion was highly placed. In positioning the chin, the mouth and nose could not be placed in their correct anatomical position. The authors detailed all the difficulties associated with the superimposition and recorded the negative superimposition results. The negative superimposition had a greater probative value (exclusion of identification) than a positive superimposition (possible identification).

Birngruber et al. (2010) presented a superimposition technique that uses a free software tool called Afloat®. The skull is positioned at a distance of 2 m in front of a camera and the image is displayed in live view mode on the computer screen. The window with the digitized portrait image is then made semitransparent and floated over the live view window showing the skull with the Afloat®(v.2.1) software. In the windows, the skull with the tissue markers can then be aligned along the anatomical axes to match the orientation of the portrait in the overlying window. The skull is then photographed to obtain an aligned skull image for the superimposition technique.

4.4.3.2 Automatic Skull-Face Overlay Methods

We have found only a small number of really interesting works that perform SFO in a fully automatic way. A few of them are based on the use of machine learning algorithms (Mitchell 1997) from artificial intelligence, as artificial neural networks (Rumelhart and McClelland 1986). However, the majority use multimodal optimization methods, that is, evolutionary algorithms (Bäck et al. 1997) and fuzzy logic (Zadeh 1965). The automation provided by these approaches represents an added value, since they are typically faster than nonautomatic methods. Moreover, they rely on quantitative measures and they can be easily reproduced. However, this type of work often involves technical concepts that are usually unknown by most forensic anthropologists. Thus, a multidisciplinary research team is required. A brief description of the methods in this group is provided below.

The method proposed by Ghosh and Sinha (2001) is an adaptation of their previous work for face recognition problems (Sinha 1998), and it was recently applied to an unusual identification case (Ghosh and Sinha 2005). Their proposed Extended Symmetry Perceiving Adaptive Neuronet (ESPAN) consists of two neural networks, which are applied to two different parts of the overlaying process. It allows the selection of fuzzy facial features to account for ambiguities due to soft tissue thickness. The system can also implement an objective assessment of the symmetry between two nearly frontal 2D images: the cranial image and the facial image, which are input as the source and the target images, respectively. The output is the mapped cranial image suitable for superimposition. The two neural networks need to be trained separately, because each can correctly map only a part of the cranial image. Two limitations are pointed out by the authors: (1) a part of the cranial image will not be properly mapped, and (2) a front view image is needed. Moreover, this method is not fully applicable for two reasons. Firstly, an important drawback is its long computation time and secondly, the need to apply two different networks separately is also a relevant flaw. Each network must deal with the upper skull contour and the front view cranial features, respectively. The superimposition found by the first network can be disrupted by the one achieved by the second network.

On the other hand, Nickerson et al. (1991) proposed a novel methodology to find the optimal fit between a 3D skull model and a 2D digital facial photograph. The most important novelty of this technique was the automatic calculation of the overlay of the skull surface mesh on the digital facial photograph. This mapping was achieved by the matching of four landmarks previously identified in both the face and the skull. The landmarks used in their work were: glabellaor nasion, the two ectocanthion points, and an upper mandibular dentition point, if present, or the subnasal point. The mappings were developed from sets of similarity transformations and a perspective projection. The parameters of the transformations and the projection that overlay the 3D skull on the 2D photograph are optimized with three different methods: heuristic, classic nonlinear optimization, and a binary-coded genetic algorithm, with the latter achieving the best results.

Ballerini et al. (2007) proposed an improvement on Nickerson et al.’s approach. The forensic experts extracted different landmarks on the 3D skull model obtained in the first stage and on the face photograph. Then, a genetic algorithm was used to find the optimal transformation to match them. The main differences between this approach and the previous one are the use of a real coding scheme and a better design of the genetic algorithm components. The method for the superimposition of the 3D skull model on the 2D face photograph is fully automatic.

Ibáñez et al. (2009a) extended the initial results from Ballerini et al. (2007) to accomplish a broader study in order to demonstrate that real-coded evolutionary algorithms are suitable approaches for CFS. In particular, the authors highlighted the good performance and high robustness of the state-of-the-art covariance matrix adaptation evolution strategy (CMA-ES) (Hansen and Ostermeier 2001). Moreover, the CMA-ES computation time was less than 15 s in the six real-world identification cases considered. It is an impressive improvement with respect to the manual superimposition performed by a forensic expert, which took several hours. An example of an automatic superimposition achieved by this method is shown in Fig. 4.5. A new SFO method based on the scatter search evolutionary algorithm was proposed a few years later by Ibáñez et al. (2012a). It was a new 3D-2D image registration optimizer for the SFO task that facilitates the integration of prior knowledge. They incorporated a method to properly initialize the algorithm and to restrict the parameter ranges using problem-specific information (domain knowledge). The “intelligent” initialization is based on the orientation of the skull to a frontal pose and the corresponding limitation of the rotation angles, which results in a significant reduction of the solution space, thus easing the problem solving. This new design exploits problem-specific information in order to achieve faster and more robust solutions. The performance of this method outperformed their previous approach (Ibáñez et al. 2009a).

Fig. 4.5

From left to right, manual and computer-aided CFS

Ibáñez et al. (2011) extended their previous approach (Ibáñez et al. 2009a), considering the uncertainty involved in the location of the cephalometric landmarks. In particular, they made use of fuzzy logic to model the difficult task of locating the landmarks (Richtsmeier et al. 1995) in an invariable place, with the accuracy needed by CFS. By the location of a larger number of cephalometric landmarks (precise and imprecise), the automatic SFO method is able to overcome coplanarity problems (Santamaría et al. 2009b) and it achieves a more robust performance. Fuzzy landmarks are regions on the face image that are provided by the forensic anthropologists when it is not possible to determine an accurate location for the cephalometric landmarks. The main drawback is the increasing computational time needed to perform an automatic SFO. In order to reduce the runtime while improving the accuracy and robustness, the same group of authors presented a novel SFO approach (Ibáñez et al. 2012b). It was based on a cooperative coevolutionary algorithm (Paredis 1995), which is able to look for both the best projection parameters and the best landmark locations (inside a given imprecise region) at the same time. Promising results were achieved, dramatically reducing the runtime required by their previous work (Ibáñez et al. 2011).

4.4.4 Decision Making

Once SFO is achieved, the decision-making stage can be tackled. The straightforward approach would involve measuring the distances between every pair of landmarks on the face and on the skull. Nevertheless, this is not advisable, because errors are likely to be accumulated during the process of calibrating the size of the images. Instead, studies based on proportions between landmarks seem to be preferred. Geometric figures like triangles or squares are good choices. It is also important to consider as many landmarks as possible and different proportions among them (George 1993). Although the methods described in the following are usually called computer-aided CFS in the literature, we prefer to refer to them as decision-making methods, since we think that the authors fail to specify the correct CFS stage in which their works are included. Indeed, the proposed automatic techniques focus mainly on the decision-making strategy as they are actually decision support systems assisting the anthropologist in making the final identification decision.⁴

These algorithms are applied on the digitized images stored in the computer, after the determination of the orientation and size of the skull by “routine” SFO techniques.

Tao (1986) developed the first procedure in which a computer was used for the decision-making stage. That decision support system aimed to replace the previously used methods based on range estimation and subjective judgment. The system provided an identification conclusion by using distances between landmarks from the superimposed images. Later, Lan and Cai proposed the use of 52 different superimposition identification indexes in the TLGA-213 system, with the same aim (Lan and Cai 1988, 1993; Lan 1990). Those indices were based on anthropometrical measures of Chinese adults, male and females, and were used together with proportions and distances between superimposed landmark lines to automatically compute the final identification decision.

Pesce Delfino et al. (1986, 1993) applied k-th-order polynomial functions and Fourier harmonic analysis to assess the fit between the outline of the skull and the face. Ten cases including positive and negative identifications were investigated. The polynomial function was used to smooth the curve representing the investigated profile. The square root of the mean square error was taken to calculate the distance between polynomial function curves obtained for the skull and the face profiles. The Fourier analysis considered each profile as an irregular periodic function whose sinusoidal contributors are found. Low-order harmonics (the first three or four) represented the basic profile shape and the high-order harmonics corresponded to the details. The sum of the amplitude differences of the sinusoidal contributors between profiles of the skull and the face represented the second independent parameter for numerical comparison. A Janus procedure (so-called by the authors because of the double-headed Latin god Janus, the bi-front) was used to evaluate the symmetry differences between the two profiles. This procedure takes into account the relationship between the total arc and the chord length and the area they delimit in the two-faced profiles. All these parameters are calculated by a computer software package called Shape Analytic Morphometry. However, this method would be only applicable when lateral or oblique photographs are available. Furthermore, their contribution requires manual repositioning of the skull for the correct superimposition.

Bajnóczky and Királyfalvi (1995) used the difference between the coordinate values of pairs of anatomical and/or anthropometrical points in both skull and face for judging the match between the skull and facial image obtained by the superimposition technique. Eight to twelve pairs of points were recorded and expressed as pixel units. Then, the final matrix, containing coordinates of measured points and calculated values, was established by computer-aided processing. Lacking the appropriate information, their model assumed that all data in that matrix was independent and followed a normal distribution with the same variance. A part of that variance was σ ², which was the square of the measurement error and was itself assumed to be the same for all the data. The model of the authors was based on assumption that

The components of the error term are independent and distributed according to

$$ N\left(0,2{\sigma}^2\right). $$

(4.1)

The authors used a presupposed value of σ as part of the model assumption. Under the assumption that the null hypothesis (Eq. 4.1) is valid, it was statistically tested using two values for σ. The authors claimed that, when a given case is evaluated, it is crucial to know what value can be considered as measurement error. One skull and two photographs were used to test the method. Both frontal and lateral face photographs were considered. They noted that their method is suitable for filtering out false-positive identifications. Although the results obtained from this method are objective and easily interpreted for lay people, the anatomical and anthropometrical consistency between the skull and the face should be assessed by forensic examiners who are well versed in the anatomy of the skull and face. The authors concluded that their method should be used only in combination with classic video superimposition and could be regarded as an independent check.

In Yoshino et al.’s skull identification system (Yoshino et al. 1997), the distance between the landmarks and the thickness of the soft tissue of the anthropometrical points are semiautomatically measured on the monitor for the assessment of the anatomical consistency between the digitized skull and face. The consistency is based on 13 criteria that they previously defined using 52 skulls (Yoshino et al. 1995). The software includes polynomial functions and Fourier harmonic analysis for evaluating the match of outlines such as the forehead and mandibular line in both digitized images. To extract the outline, gradient and threshold operations are used. Five case studies were carried out. However, they noted that this analysis could not always be applied because of the difficulties in extracting the facial contour from small and poor facial photographs offered by the victim’s family.

Skull-face overlay was guided by different crosses that were manually marked by the human operator in both the face and the skull radiograph photographs in Ricci et al. (2006). Once that stage has been completed, the algorithm calculates the distance moved for each cross and the respective mean in pixels. The algorithm considers a 7-pixel distance a negligible move. The mean value of the total distance moved represents the index of similarity between the given face and skull: the smaller the index value, the greater the similarity. The algorithm suggests an identification decision based on that index of similarity. The authors claim 100% correct identifications over 196 cross-comparisons and report that the minimal number of landmarks required is 4.

In Birngruber et al. (2010), the portrait image and the image of the aligned skull are compared at the same scale and morphological matches are searched for, such as the general length and width of the face or the deviation of the bridge of the nose. The facial proportionality of the portrait and the aligned skull should be in accordance with horizontal lines drawn through the middle of the eyes, the spina nasalis, and the mouth.