Keywords

1 Manufacturing Metrology

To support the manufacture of next-generation high-value products, increased reliance will be placed on metrology. This article will discuss an approach to metrology that, in our opinion, has the potential to significantly enhance the metrology capability in advanced manufacturing. While traditionally metrology has been applied to the inspection of the final part, after the manufacturing process is completed, nowadays the trend is to bring metrology in to the production line (Everton et al. 2016). This may just mean performing part inspection right after each step of the manufacturing process (for example, between a roughing and a finishing operation in a machine tool), or we can push integration even further by carrying out the measurement tasks during the execution of each individual manufacturing operation, where the nature of the operation allows it (for example, within an additive manufacturing process, measuring the properties of a layer while it is being fabricated). Also, when considering integration between metrology and manufacturing, considerations about the measurement data are important. One integration scenario may see a measurement triggering an alarm if something goes wrong during a manufacturing operation (i.e. detection of an out-of-tolerance condition); more complex integration scenarios may see some form of implementation of feedback mechanisms, for example, the triggering of a corrective action, or the real-time modification of some manufacturing process control parameter to bring the process back to an in-tolerance state.

Such integrated metrology needs to be compatible with the manufacturing cycle time (we do not want metrology to prohibitively slow down the process), and measurement systems need to be spatially located to be compatible with the type of integration: i.e. on the production machine, or close enough if measurement is to be performed between manufacturing stations.

Developments in integrated measurement and control have allowed significant enhancement of advanced manufacturing techniques and marked improvements in surface texture and material properties, along with a reduction in process variation and defects. Integrated measurement and control technologies can also offer cost reductions and process efficiency improvements, but the scale of up-front investment required can often seem daunting, and the benefits may not be easily quantified. Cost considerations aside, there are a number of potential barriers to integrated metrology that can prevent its wide-scale adoption in industry. These include, but are not limited to the following. Note that we will concentrate mainly on dimensional and surface metrology using optical technology in this paper, but many of the arguments can be generalised to any metrology discipline.

  1. i.

    Measure over larger dynamic ranges: all measuring instruments have a finite dynamic range, defined here as the ratio of their range to their resolution in terms of either lateral and/or height capabilities. Often the trade-off is to have either a large range or a high resolution, but rarely both (Leach et al. 2013). For example, in optical surface metrology, there are two distinct classes of instrument which are commercially available: 1. those that measure over large areas (metres squared) with low spatial resolution – a few hundreds of micrometres (for example, fringe projection, photogrammetry and moiré interferometry) (Harding 2013; Van der Jeught and Dirckx 2016 and Zuo et al. 2016) and 2. those that measure over small areas (up to a few millimetres squared) with spatial resolutions of the order of a micrometre (for example, coherence scanning interferometry, confocal microscopy and focus variation microscopy) (Leach 2011). Essentially, the former class is camera-limited (sensor resolution and depth of field issues), while the latter is objective-limited (diffraction limited through finite numerical aperture) and can be prohibitively slow due to the need for scanning in both lateral and often axial directions. In order to meet the demands of advanced manufacturing, there have been several attempts to try and combine the two classes (see for example, Kaysar et al. 2004; Weckenmann et al. 2009), but more progress is required before such hybrids can be used in earnest. This dynamic range issue is especially problematic for industries that manufacture small features over large areas in a highly parallel fashion, for example, for roll-to-roll applications such as printed electronics.

  2. ii.

    Measure higher slope angles: all optical instruments are fundamentally limited by finite slope measurement capabilities, mainly due to their numerical apertures and, in many cases this limitation is related to the dynamic range limitation above (see Leach et al. 2014 for an outline review of these limitations). This fundamental limitation will be treated in some depth in Sect. 3.2, as it is one of the key limitations that hinder progress in optical metrology.

  3. iii.

    Measure difficult materials: all surface measuring instruments have limits on the types of surface that they can measure. Also, where mathematical models of the imaging or scattering process are necessary, some instruments make assumptions about the surface being measured that may not be met in practice, for example, that the surface is a highly-reflecting metallic surface. Polymer and ceramic surfaces can cause significant issues due to translucency and rough surfaces can cause multiple-reflection and slope effects (see ii). Surfaces with mixed materials and/or produced using multiple processes are especially challenging (Mathia et al. 2011), and surfaces produced by additive manufacturing have significant material challenges for optical instruments (Grimm et al. 2015; Launhardt et al. 2016 and Townsend et al. 2016).

  4. iv.

    Measure at high speed: for integrated metrology, and especially for in-line/in-situ metrology, there must be significant increases in the speed of measurement (Allwood et al. 2015). The high-speed metrology task is increasingly limited by the fundamentals of optical interrogation of the surface, such as: the compromise between spatial resolution and field of view; the loss of effective spatial resolution due to motion blur; or the dynamic range of optical properties across the inspected region. Such limits imply that faster “brute-force” measurement of the whole surface cannot be a solution. Whilst there have been valiant attempts to speed up conventional measurement techniques (see for example, Jiang et al. 2010; Zhang 2012 and Hahn et al. 2016), often speed increases of several orders of magnitude are required for integrated metrology to become realistic. Scatterometry (Madsen and Hansen 2016) and scattering methods (Leach et al. 2013) have been applied in-process, especially in the integrated circuit and optics manufacturing industries respectively, but both methods are highly specific in the types of surface they can measure (although both methods are arguably information-rich in nature). In order to overcome the metrology speed challenges, it is essential to exploit a priori knowledge about the production task, the nature and functional significance of relevant defects, and any potential repair steps to dramatically simplify the measurement task. Development of existing measurement techniques to simplify integration, hybridisation and increased environmental tolerance will also help. A range of research strands needs to be followed including: optical system modelling and defect extraction, global control of substrate and defect location, intelligent sampling and data throughput, and fast feature inspection.

The conventional approach to address the issues above is simply to improve our existing measurement technologies. But, as has been pointed out in iv above, there are a number of reasons why this is not the whole solution. In many cases, we have come up against barriers that prevent us from significant further improvements in instrument performance. Such barriers are due to limitations in current technology and due to the fundamental laws of physics (or a combination). Examples of limitations due to physics include: the shot noise limit, either due to the discrete nature of electrical charge in electronics or to the particle nature of light in optical detectors; the diffraction limit in optics, due to the wave nature of radiation, there is diffraction caused by the limiting edges of the optical system’s aperture (this means a point will always be imaged with finite blur); the slope angle limitation in an imaging system due to its finite numerical aperture (see Sect. 3.2); and the limit on sampling due to Shannon’s theorem (all these limitations are summarised in Boreman 2001). Examples of technology limitations are the finite processor speed or memory limits of computing systems. When we attempt to produce integrated measurement solutions, such technology limits are a frequent issue and often the only way around this is to use pre-processing methods, for example adaptive or intelligent data reduction and sampling techniques, effectively trying to reduce the data overhead (Wang et al. 2012; Yu et al. 2012).

2 Information-Rich Metrology

Information-rich metrology (IRM) is a term that we introduce to refer to the use of any type of additionally-available information to improve a measurement process. Information may come from knowledge of the manufacturing process, knowledge of the object to the measured, and/or knowledge of the physical interactions/principles underlying the measurement technology itself. Additional information may be pre-existing (i.e. “a priori”), or obtained through other measurement processes, even concurrently to the measurement we are aiming to improve. An overview of the primary sources of information typically exploitable when IRM is applied to a manufacturing environment is provided in Fig. 1.

Fig. 1.
figure 1

The information-rich metrology paradigm: typical sources of information that can be exploited to improve the measurement process.

Full size image

The idea of using available information related to the product, or process, or product-measurement-instrument interaction, makes intuitive sense because metrology in manufacturing takes place in controlled and predictable conditions, with a sensible amount of information which is known in advance. Often when we manufacture something, and especially when we use digital manufacturing methods, we have a large amount of information about the object being manufactured, for example, the CAD data gives us the nominal form and we have usually characterised the surface texture to a high degree of confidence. Analogously, we do have – or we can acquire – a significant amount of information about the manufacturing process, in terms of its capability, the features and defects it generates, the materials it is designed to operate with, and the types of geometries and surfaces it typically produces. Most of the above information becomes available at product development and at manufacturing process planning, and we are asserting that such information may also bring benefit to metrology.

One of the most promising paradigms for IRM is based on using additional information about the manufacturing process and the object that is fabricated, to develop improved mathematical models that describe the interactions between the measured object and the measurement probe (see Fig. 1). In practice, mathematical models that describe physical principles and phenomena underlying many measurement technologies are already available, although one has to be careful that over-simplifications are not abused. In optical measurement, for example, many models have been developed over the decades, to support the theory of focus-variation measurement, coherence-scanning interferometry, confocal microscopy, fringe projection, photogrammetry, etc. It is safe to say that the totality of current commercial optical measurement systems are already making use of complex mathematical models to interpret raw data acquired through their probes. However, because such models aim to be general, which means that they must be applicable with little prior knowledge of the measurement scenarios, they can make very few assumptions about the nature of the surface which will be measured, the material properties they will encounter, and other factors. Thus, such models are limited in the information they can provide. A typical example is the interpretation of signals originated by light captured by the detector after multiple reflections and scattering. Trying to reconstruct what determined the patterns captured by the detector typically implies the solving of complex (often non-linear) inverse problems, which are typically unsolvable or ambiguous without resorting to additional sources of information. The advantage of working in the scenarios typically encountered in manufacturing metrology is that we often have such additional information, for example, because we have an approximate idea of the part geometry, and/or because we have a clear idea of what type of signature features a specific manufacturing process leaves on a surface.

The predictability of operating conditions, and the wealth of information available in a typical manufacturing metrology scenario explain why our research in IRM is focusing on one of its most promising paradigms, that is, to develop advanced models of probe-surface interaction in order to improve measurement quality, whilst using externally available information (from the product, process, or other) to develop such models.

One last comment must be reserved on the need for IRM to revisit the way information is aggregated and processed. As a matter of fact, the addition of a potentially high number of heterogeneous information streams raises a whole series of challenges regarding how such information should be homogenised, aggregated and finally exploited towards achieving a better measurement result overall. Challenges are in how to handle large amounts of data in increasingly shorter times (possibly verging towards big data issues), in how to data mine the relevant relationships between variables, and finally in how to obtain mathematical and statistical models that ultimately support what we call the “smart” measurement paradigm, which is meant to supersede the conventional metrology pipeline of “blind” processing (i.e. where knowledge is extracted exclusively from the raw data provided by the measurement instrument, with no help from any other sources of information). As in many other applications involving big data, we believe that artificial intelligence (AI) technology in general, and machine learning in particular, can provide significant support to the development of the smart measurement solutions of the future.

Several examples will be given in the following, together with additional examples that show that a priori information also has additional uses, other than for improving measurement models, but before proceeding, it is important to discuss why IRM is important to manufacturing metrology, and to integrated metrology in particular.

As stated earlier, central to IRM is the aim to improve measurement quality. Quality is here intended as a generic term encompassing multiple facets. Improving quality may mean: reducing measurement times, improving measurement performance indicators (accuracy, precision, etc.), expanding the range of covered scales (spatial resolution and range - in terms of spatial frequencies we will often refer to expanding the bandwidth of a measurement), and improving coverage, intended as the capability to reach surfaces which may be harder to reach, for example measuring beyond the maximum permissible slope for a given measurement technology (see below). Improving measurement quality may also mean that we can obtain the same results that we obtained before, but at a fraction of the cost, or of the time, or with smaller, cheaper, more rugged and more affordable instruments; these are all essential aspects to better integrate metrology into the production line.

IRM is nothing new; the use of additional sources of information to enhance a measurement system’s performance, and thus the quality of its measurement output, has been carried out for hundreds of years. Perhaps the first and most well-known example of IRM is that of optical super-resolution in microscopy and astronomy (Leach and Sherlock 2013). A number of further examples of IRM are given in Sects. 3.1, 3.2 and 3.3.

3 Examples of IRM Research at Nottingham

3.1 An All-Optical Dimensional Measuring System

We present here the design of an original, flexible and open-architecture, all-optical dimensional measuring system (AODMS) for measuring the geometry and surface topography of micro-scale components and components with microscale features (Fig. 2). The system is designed to operate in a cube of 100 mm sides, with micrometre or sub-micrometre measurement uncertainties. The key aspects of AODMS are flexibility and open-architecture. The system is designed to accommodate a wide array of heterogeneous optical sensors, ranging from 3D measurement to 2D imaging, from prototype to commercial sensors, and is being designed to be particularly suitable to support the investigation of multi-sensor data fusion solutions (Wang et al. 2015) – another example of IRM, where different resolution data are combined; one effectively acting as the a priori data for the other. The open nature of the architecture allows full flexibility in the design and configuration of the instrument control and communication software, as well as of the data analysis and processing software, thus presenting itself as an ideal platform to investigate IRM through the support to the development of solutions to enable knowledge-driven measurement, for example, through the interaction with CAD/CAM systems, product data-management systems and any other IT-based knowledge-management solutions.

Fig. 2.
figure 2

Schema of the all-optical dimensional measurement system.

Full size image

The AODMS prototype includes a moving stage, a support and interface to a photogrammetric system dedicated to form measurement (Sims-Waterhouse et al. 2017), and a support and interface to a hybrid prototype sensor that integrates coherence scanning interferometry (CSI) and focus variation microscopy (FVM) for surface topography measurement. The metrology frame of the AODMS is fabricated using additive manufactured lattice structures for vibration isolation within a pre-defined bandgap (Syam et al. 2017). AM fabrication technologies and lattice structures are being investigated for the AODMS also because of their potential to support modularity and increased architecture reconfigurability to accommodate different inspection scenarios. The AODMS is currently being developed to support the experimental activities of a number of IRM-related research projects: (1) fusion of photogrammetry, CSI and FVM data; (2) CAD information and speckle enhancement technologies to improve photogrammetry; (3) self-calibration solutions for CSI and FVM (Ekberg et al. 2017); (4) motion stage calibration using photogrammetry (Sims-Waterhouse et al. 2017); (5) calibration and adjustment of the CSI measurement using the ‘foil’ model of the surface (Su et al. 2017); and (6) bandwidth enhancement of CSI for high slope surface measurement based on rigorous modelling of the signal response (Coupland and Lobero 2017, see next section).

3.2 High Slope Measurement

Complex surfaces often have features which have high slope angles, either through design, for example prismatic arrays for safety signage, or due to the nature of the manufacturing method, for example in additive manufacturing (Townsend et al. 2016). The most complex surfaces may have very large ranges in terms of spatial frequencies (and hence surface slope angles), for example, a highly aspheric surface with a micro-scale Fresnel grating (Fang et al. 2013). Deterministically-textured surfaces with complex geometries are abundant in mechanical engineering fields for such applications from fluid control, adhesion, tribology and bio-compatibility, and such surfaces present significant metrology challenges (see Bruzzone et al. 2008; Malshe et al. 2013 and Thomas 2013). Instruments that use contrast to detect the surface (for example, FVM), require a certain degree of roughness on the surface (or other contrast mechanism), therefore, can detect out-of-aperture slope angles (Leach 2013; Hiersemenzel et al. 2012). However, such instruments cannot measure smooth surfaces and the reconstruction process is complex and prone to error (Nikolaev et al. 2016). With fringe projection systems, powder sprays are used to give the surface a matt coating to exploit the use of scattered light and hence capture information from slopes outside the aperture range, but can contribute several micrometres to the measurement uncertainty (Palousek et al. 2015).

The above methods to overcome the fundamental slope limitation either require the surface to have a specific nature (for example, to be rough on specific scales) or require a coating to be applied. However, another potential approach to extend the slope angle limitations of optical instruments is to use rigorous modelling of the optical interaction with the surface to solve the non-linear inverse problem of multiple reflection. For example, in CSI, the effects of multiple scatter would be considered a source of error (Gao et al. 2008; Lehmann and Xie 2015) and, therefore, neglected. However, multiple scattering can be considered as a mechanism that redirects into the instrument light that would otherwise fall outside the limits of the numerical aperture. Consequently, the effects of multiple scattering have the potential to reveal 3D features described by spatial frequencies that are outside of the usual bandwidth of the instrument (i.e. that would be described by simple linear imaging theory). For general 3D objects, a priori information is needed to distinguish the effects of single and multiple scattering. For the case of surfaces, however, knowledge that the scattering is due to the interface of two homogenous media can be sufficient. This method has been demonstrated as an iterative optimisation routine using finite element analysis to calculate the optical fields scattered from a silicon step (Coupland and Lobero 2009) and is illustrated in Fig. 3. Figure 3 (left) shows computed CSI fringes that clearly reveal the upper and lower horizontal surfaces of the step. From this data (the a priori), a more accurate scattering model can be implied, as shown in Fig. 3 (centre). The changes to this model necessary to explain the data in Fig. 3 (left) were then computed resulting in the higher order interferogram shown in Fig. 3 (right). The vertical wall of the step is now apparent and further iterations of the method can be applied. Such a method for measuring outside the numerical aperture limit can be used to measure a range of structures, for example, high aspect ratio holes and pillars found in MEMS devices or X-ray optics.

Fig. 3.
figure 3

Left: measured CSI interferogram. Centre: finite element model from CSI interferogram. Right: second-order CSI interferogram, clearly showing the vertical sidewall Coupland and Lobero 2009).

Full size image

The results shown in Fig. 3 are a good illustrative example of IRM as they? many of the essential ingredients shown in Fig. 1 in an obvious manner. However, the research effort required to realise such a technique should not be underestimated. There are significant challenges in performing the measurements (mainly in optimising the contrast of the second-order and potentially higher-order fringes), for the modelling of the second-order effects (a finite element model was used in this example and we are currently working on a more time-efficient boundary element method) and in the interpretation of the (information-rich) data. There may be opportunities to use this approach with other optical systems and for rough surfaces (perhaps using low aperture measurements as the a priori data), but the rigorous models need to be developed and optimised to operate in reasonable times.

3.3 Measurement Enhancement Using Artificial Intelligence

AI promises to be one of the most disruptive technologies of the next century, but researchers are up against critics who claim that it is nothing but another hype cycle of the same old recycled technology. The main reason for the new ‘hype’ in the commercial market today is mainly the availability of ubiquitous multi-core GPU computing, which enables the low-cost training of simple AI networks at home or in small businesses. Another reason is the proliferation of low-cost and relatively powerful computing platforms (such as the Raspberry Pi which retails for well under £100) which can take advantage of trained models to perform autonomous tasks. With the advent of the Internet of Things, AI algorithms on low-cost platforms will become even more pervasive and ubiquitous in society in the near future, and manufacturing industry is trying to position itself in order to be at the forefront of AI technology.

In terms of IRM, AI can be used to take advantage of the a priori data, the measured object (even past measurements) and, in combination with a functional measurement model, accelerate the measurement procedure and make it more efficient. An example of IRM which leverages AI to perform the measurements can be found in the microelectronics industry, where scatterometry data is used to accurately predict track resistance and, therefore, pre-empt failures in integrated circuits (Rana et al. 2015). The advantage in doing this is that silicon wafers can be discarded early in the manufacturing cycle to avoid expensive processing and testing further down the manufacturing line, thus avoiding the time required to produce and test the wafer at the end of the process.

AI can also assist within an actual measurement system by converting the various parts of the measurement system (for example, cameras, stages, light sources) into smart ‘agents’ and thus viewing the machine as a ‘perceptive agency’, with agents working collaboratively to optimise the measurement result in terms of accuracy and coverage on a specific object (Amigoni et al. 2003). At Nottingham, we are currently developing a system to measure the 3D form of complex components (our main target industry is additive manufacturing). A fringe projection system is effectively converted into a ‘perceptive agency’ that is able to predict the source and camera positions (using inverse rendering) with respect to the measured object in real time, i.e. a frameless, self-calibrating measurement system (Stavroulakis et al. 2017a).

In fringe projection, where either the camera or projector setup can change significantly between measurements or the object needs to be tracked, self-calibration has to be carried out frequently to keep the measurements accurate. It is common to use methods developed initially for photogrammetry for the calibration of the camera(s) in the system in terms of extrinsic and intrinsic parameters. To calibrate the projector(s), an extra correspondence between a pre-calibrated camera and an image created by the projector is performed. These recalibration steps are usually time consuming and involve the measurement of calibrated patterns on planes, before the actual object can continue to be measured after a motion of a camera or projector has been introduced in the setup and hence do not facilitate fast and efficient 3D measurement of objects. By employing and combining a priori information via inverse rendering, on-board sensors (Stavroulakis et al. 2017b), deep learning and leveraging a graphics processor unit, we have developed a fine camera pose estimation method which is based on optimising the rendering of a model of a scene and the object to match the view from the camera (see Fig. 4). We have found that the success of this calibration pipeline can be greatly improved by using adequate a priori information from the aforementioned sources. The ultimate plan is to have a simple-to-use projector and camera set-up that can be re-configured for different 3D object shapes and can allow for effects such as shadowing, occlusions and different textures. The complexity of the system is transferred to the AI-based software and the hardware can be cost-effective and fit-for-purpose.

Fig. 4.
figure 4

A priori data used for the fringe projection system under development at Nottingham.

Full size image

4 Discussion

We are led to believe that we are currently experiencing a new industrial revolution; one that will be data-driven, agile and autonomous. One of the key ingredients in this revolution will be the need for fast, integrated metrology. However, many of the current metrology tools have hit fundamental and/or technical barriers that result in them being seen as a process overhead rather than potentially leading to innovative and commercial successes. Information-rich metrology is essentially a marriage of metrology with information technology and makes use of all the available information sources to enhance a given measurement scenario. We believe that IRM will allow many of the metrology barriers to be leap-frogged in the future and ease in integrated metrology as an essential part of any modern manufacturing process – not as an overhead, but as something that will enhance quality, enhance efficiency and ultimately, reduce costs. By making use of all the available information resources in our measurement and manufacturing processes, we can significantly enhance the way we make things.