Aerospace Assembly Gap Measurement Using Low Cost Smart Tools with Machine Vision

Abstract

This paper details the conversion of mature machine vision technology from a fixed position automation line based device to a handheld technology and addresses the problems associated with maintaining a consistent camera distance and light source by using 3D printed hand tools. Specifically, the problem of gap measurement within aircraft wing assembly is used as an example application, however, the wider opportunities and functionalities associated with industry 4.0 are demonstrated and briefly discussed.

1 Introduction

1.1 Industry 4.0 and Smart Tools

Industry 4.0, often termed the “smart factory” is defined as the next industrial revolution utilizing the latest groups of cyber-physical systems such as cloud computing, internet of things and cognitive computing [1]. It is based on a trend of increasing automation and data exchange where the physical manufacturing process is monitored analyzed and optimized with minimal engineering input.

In order for the optimization process to take place, the system must have accurate quantitative, digitized data from the process. This is often difficult to achieve in an aerospace manufacturing production environment where many tasks are performed manually using traditional hand tools or human inspection. This currently relies heavily on operators recording the value manually to a database system. This is time consuming, subjective, error prone, and provides no direct infallible record of the product.

The scientific field for machine vision inspection and associated algorithms is mature and well developed [2]. Various functions and methods for detecting features such as geometry and faces are well developed specialist subjects, for instance several papers focus specifically on edge detection functions used in this development [3,4,5]. Commercially available technology is also widespread and is accompanied by software support with extensive functions for the detection of a multitude of features [6]. However, such systems are typically expensive £4k – 15k, require the use of a microcomputer with significant processing power, and are intended to be permanently positioned on a production line where conditions such as lighting and camera position remain consistent.

Hand tools with data driven functions are recently becoming more available and widely accepted. They can automatically record data about the process, such as the torque of fasteners, connect wirelessly and save process data to a central database. These tools are commonly termed “smart hand tools” as they perform some type of process data management in addition to completing the engineering task [7]. They may also be aware of the process and product context, via RFID, Bar or QR codes on components, and have the ability to adjust settings accordingly to suit the assembly component or the task at the point of use [8] (Fig. 1).

Fig. 1.

An example of a “smart tool” for tightening of fasteners to controlled torque values with process data logging capability [7].

Fig. 2.

Commercially available assembly gap measurement devices include: traditional feeler gauges (a), laser line gap profile scanning systems (b) and capacitive film sensor devices (c)

The ability to significantly reduce the cost and mobility of machine vision technology without compromising reliability would dramatically widen its application. The aim of this development determines the ability and reliability of the technology when used with the latest low-cost portable microprocessors such as MyRIO and to show that lighting conditions and camera positions can be controlled via the use of bespoke rapid prototype hand held tools to maintain the reliability of the technology. The paper also highlights how the productivity of the manual worker, when using such a handheld device, can be increased with greater integration into the factory 4.0 data network.

1.2 Hand Held Machine Vision Application; Gap Measurement in Aerospace

In aerospace assembly, the torque values of fasteners and the gaps are critical for product quality and safety and are required to be controlled and recorded. Fastener torque values can be controlled and logged via commercially available smart tools [7]. However, gap measurement is more difficult in practice. A number of devices and methods exist (Fig. 2) but each has associated high costs and limitations (Table 1). Therefore, measurements are typically taken by hand using feeler gauges and recorded manually. The process is slow and subject to human error requiring manual data input for industry 4.0 compliance and additional electronic proof or quality checks may be required. Additionally, none of the devices is able to capture gap closure during tightening and co-ordinate torque data with hand tools to measure the loads required to close gaps. Such data is valuable to determine assembly build stresses and fastener assembly load specifications.

Table 1. Commercially available gap measurement device types

1.3 Rapid Manufacturing, Machine Vision and IOT Devices

Rapid manufacturing and 3D printing systems are now prolific with hobbyists and design engineers. The process is particularly suited for producing low volume customized components such as prosthetics [8]. It has also been used to produce hand tools for various purposes [9]. However, the most commonly available PLA material can be brittle, soft and break easily. Therefore, it is ideally suited as a prototyping tool in the rapid development of industry 4.0 hand tools provided no significant load is experienced in use. It can also be used as a temporary prototype to perfect the design before a more rugged production model is completed. Therefore, it was selected as the most suitable method for the rapid development of smart hand tools for machine vision inspection.

Machine vision, computer vision or image-processing has been in development since the 1980s and has a number of applications within the automation industry. Typically, it is used as a quality control or sorting method where typical applications include checking label positions on consumer packaging [10]. Its use is also now common place in large robot automation cells used for finding hole centers, edges and other well defined part features. However, apart from rare specialist exceptions such as droplet measurement [11, 12], it is not currently in widespread use in hand tools. It offers great potential for handheld monitoring of processes provided the cost and size of the technology can be reduced along with simplifying the programming and deployment method.

IOT (Internet Of Things) devices are defined as a self-contained smart device or computer comprising of a micro-processor and controller with the capability to connect the internet, gather and transmit data, read physical sensors and operate switches.

A number of low cost development IOT microcomputers are now commercially available to both the hobbyist and industrialist alike, such as Arduino, Raspberry PI and Siemens IOT 2040. However, very few have machine vision capability. Raspberry PI appears to function using Open CV [13], however, it is not a native function. These low cost devices have the potential to significantly transform data acquisition and control in the production environment once they become reliable and widely available. National Instruments MyRIO supports machine vision capability, however, its cost is still relatively high in comparison to most other IOT devices and each additional device requires a software license at an additional expense, increasing the cost of deployment further. Nevertheless, it was found to be the most convenient tool with the greatest amount of customer support for the development and concept proving of self-contained programmable vision applications.

2 Research Methodology

2.1 Experimental Set-Up

The hand tool was designed to work as a self-contained casing for a commercially available low cost USB microscope. It was also required to project a high incidence light source over the measured edge to highlight the intended gap to be measured whilst eliminating as much light as possible from other sources, thus improving measurement reliability. The prototype casing also allowed the adjustment and experimentation with the camera angle. The casing was also required to be as small as possible for access within the aerospace structure. The final prototype included a clamp for the microscope wire as it was found that movement of the wire resulted in some movement of the optical sensor inside the microscope. Figure 3 shows the latest prototype design which also included a variable resistor to adjust the intensity of the high incidence LED light source and customizable buttons with indicators for initiating a measurement and storing images. The initial device is reasonably small but has excellent potential for further miniaturization for increased measurement access.

Fig. 3.

Smart hand tool development prototype for optical aerospace assembly gap measurement

The National Instruments MyRIO device was chosen as the main IOT processor due its ability to be programmed via LabVIEW with predefined vision functions with relative ease. The platform also provides wireless capability and serial/digital IOs. Although the system does not support a VDU screen onboard, the image output can be monitored using any internet browser via a wireless PC, tablet or smartphone and the saved image captures can be saved directly to the attached USB for either wireless or USB connected transfer. It also supports interfaces such as UART serial LCD displays for output of symbols, text and numbers. A Maplin 16 × 2 Serial UART was utilized in this case to display the gap dimension, chosen for its availability and low cost.

A generic USB 2–5 Mp camera with 10–300x magnification and 30FPS sampling rate was selected for imaging device due to its very low cost and widespread availability. The high incidence light source, buttons and indicators are powered and controlled via digital and analog I/O sockets provided on the MyRIO. The microscope camera was connected to the NI MyRIO device via USB hub along with a standard USB drive for image storage.

2.2 Control Algorithms and Decision-Making Environment

An example of the overall integration of the handheld device in a factory 4.0 aerospace assembly environment is shown in Fig. 4. The component being measured and the position of the gap measurement may be obtained via laser tracker and/or QR code on the component. The Gap data can then be sent via ODBC connection to the manufacturing bus, this value can then be cross checked for quality assurance. Once passed the data can then be archived against the part number. On failure, the relevant decision processes for rework or concession can be made facilitating an evolvable assembly systems architecture [15] similar to that demonstrated for aerospace assembly [16].

Fig. 4.

Integration of the handheld machine vision device into the EAS factory 4.0 system architecture

The overall approach for decision making is demonstrated in Fig. 5. A number of processes run locally on the MyRIO device such as image acquisition, image processing, edge detection algorithms and calculations to convert pixel values to actual gap via a calibration table. Decisions on image quality and edge measurement quality of points and edge line fit are made locally with the operator notified to remeasure if an edge has not been detected with adequate certainty. Once an adequate measurement has been made the image and gap value can be passed to the manufacturing execution system (MES) via wireless connection. In order to work with the MES the location of the measurement on the assembly must be known, this could be entered manually, via a QR or bar code on the assembly or by RFID or laser tracker co-ordinates. Once the measurement location is known the MES can look up the gap requirements from the product database and ensure that the gap measured is within tolerance. If so, then the operator can be notified of a successful measurement and carry on to the next measurement point. If not, additional instructions may be issued.

Fig. 5.

Integrated control architecture

The MyRIO was programmed visually using LabVIEW functions combined with the Vision builder workbench. The program works by grabbing an image from the USB camera converting to high contrast black and white. Then a linear edge contrast recognition function was used searching from top to bottom for the top edge and bottom to top for the bottom edge to output a linear average. The pixel distance between the two averaged edges is then calculated. Values obtained from a linear fit of calibration results allows the conversion of pixel distance to the actual gap in mm. This value is obtained via a calibration procedure discussed in the results chapter.

Once the gap result is obtained it is communicated to the operator via a number of solid lit LEDs if the gap is in range. If the gap is out of range or an edge has not been detected LEDs are set to flash using digital I/O functions. The actual GAP value is also displayed on the LCD display via a UART3 LabVIEW function. Images are also saved to a selected file location and the value can also be published for reading from any Wi-Fi enabled device with an internet browser.

3 System Implementation and Discussion of Results

3.1 Calibration and Variability

A typical aerospace gap measurement process was recreated using mild steel flat and channel section coated with typical green zinc chromate aerospace primer (Fig. 6). The gap was adjusted and measured using feeler gauges to various gap sizes between zero and 0.7 mm with five optical measurements taken at each gap using the newly developed device. Table 2 shows the results of the calibration procedure with an excellent linear fit to the results (Fig. 7) yielding the required conversion constants for converting pixel gap to mm within the LabVIEW code. An average range of 20 microns was found in the measurements with an average standard deviation of 7 microns, well within the gap measurement requirements for typical aerospace assemblies. However, a zero gap measurement did not necessarily result in a zero pixel measurement. This was due to the presence of a radius on the edge of flat bar at the top edge of the gap. A small radius on a gap edge is typical in most assembly situations. Therefore, a calibration may be necessary on each new gap measurement procedure for new parts and assemblies. Parts will also require consistent edge radius dimensions to give reliable measurements.

Fig. 6.

Aerospace gap measurement test set up to replicate wing rib foot to cover measurement, showing handheld device and typical output image of the gap under the top plate with automatically generated text overlay of results.

Table 2. Optical gap measurement calibration results

Fig. 7.

Linear approximation of calibration results to give results in mm

4 Conclusions

The results show that the development of machine vision technology in a hand tool is now possible using low-cost portable microprocessors and readily available machine vision software functionality. The reliability of the technology can also be maintained by the careful design of bespoke rapid prototype tools which maintain camera position relative to the measurement workpiece, shield external light sources and provide their own consistent self-contained light source. This development widens the potential application of the technology to a large range of hand held inspection processes and could greatly improve the connectivity and productivity of labor within the assembly industry.

The device developed here specifically for microscopic optical gap measurement of wing skin to rib foot has shown great promise with a maximum 20 µm error and standard deviation of 7 µm under lab conditions. The cost of equipment is estimated at £600 plus the cost of software licensing which could be further reduced by development on an open source platform. However, the method requires refinement in the following areas before reliable workshop deployment is feasible:

• Graphical interface for the operator

• Robust procedure to account for edge radius in measurements

• Rugged housing and long life battery

• Automated procedure for networking and saving data.