About Vision Sensors & Smart Cameras
Fundamentals of Machine Vision for Inspection & Quality Control
1. What is Machine Vision?
Machine Vision is a transformative technology that plays a crucial role in industrial automation. It involves the use of cameras, often referred to as Industrial Smart Cameras, and computer algorithms to conduct visual inspections and analyses of various types of manufactured goods. The primary objective is to replace task that were typically conducted with human visual inspection.
This includes applications such as performing measurements that would normally require tools, reading characters and codes for serial number tracking, identifying defects, contaminants, and other irregularities that could compromise the quality of and end product.
2. Typical Smart Camera and Machine Vision Tools
Measurement and detection tools are fundamental to the core functionality of smart cameras. These tools enable the camera to capture, analyze, and respond to visual data accurately and efficiently. They provide the essential precision needed to quantify physical attributes such as dimensions and distances, ensuring consistent quality in manufacturing processes. Furthermore, their adaptability allows the camera to fine-tune parameters, making it versatile across diverse industries.
Beyond precision and adaptability, these tools drive automation and efficiency by facilitating real-time decision-making, thus reducing labor costs and enhancing operational efficiency. They also serve as data analyzers, offering invaluable insights for process optimization. Notably, the following section will provide a breakdown of the common smart camera tools, highlighting their specific roles and applications in various industries. These tools, ranging from edge detection to pattern recognition, form the cornerstone of smart camera technology, enabling businesses to improve product quality, streamline operations, and make data-driven decisions.
i. Edge Detection
Edge Detection is a crucial feature in industrial smart cameras, enhancing their image processing capabilities. This technique identifies and highlights boundaries or edges within an image, enabling precise object recognition, quality control, and automation across industrial applications. It operates by analyzing intensity gradients, detecting abrupt changes in pixel values corresponding to object edges. This edge map forms the basis for various industrial applications.
Programming for Edge Detection in industrial smart cameras involves configuring parameters such as threshold levels, filter types, and kernel sizes. These settings fine-tune the detection process, influencing sensitivity and accuracy. Engineers and operators adapt the camera's performance to different tasks and environments using complex algorithms that efficiently process image data, extracting valuable edge information.
ii. Color and Contrast Recognition
Color and contrast detection tools are essential components of industrial smart cameras, allowing them to analyze images based on color information. These tools enable the cameras to identify, measure, and classify objects by their color properties, which is particularly important in industries such as food production for quality control and pharmaceuticals for sorting medications by color-coded packaging.
Color and contrast detection tools function by examining the color components of each pixel in an image, typically in RGB or HSL color spaces. They can recognize specific colors or color ranges, making them versatile across applications. For example, in automotive manufacturing, these tools ensure consistent paint quality by detecting variations in color shades, while in textile production, they can sort and classify fabric based on color patterns, improving quality control processes.
iii. Diameter Measurements
Diameter measurement tools are integral components of industrial smart cameras, specializing in precise measurements of object dimensions. These tools enable cameras to accurately gauge the diameter of objects, regardless of their size or shape. This capability is invaluable in industries like manufacturing and quality control, where ensuring adherence to specific size specifications is critical.
In the automotive sector, diameter measurement tools are utilized to verify the accuracy of machined parts, such as pins, dowels, drilled holes, counter sinks and chamfers, ensuring they meet precise diameter requirements.
In practical terms, diameter measurement tools work by utilizing image analysis algorithms to identify object edges or boundaries and calculate the distance between them. These tools are highly adaptable, capable of measuring the diameter of objects ranging from small components to large industrial equipment. In applications like electronics manufacturing, they can measure the diameter of micro components with utmost precision, ensuring product quality. Likewise, in the construction industry, diameter measurement tools play a vital role in assessing the dimensions of pipes, cylinders, and other structural components, enhancing accuracy and efficiency in construction projects.
iv. Circle Detection and Evaluation
Circle detection and evaluation tools are integral components of smart cameras, offering the capability to identify, measure, and assess circular objects within images. These tools find extensive use in various industrial applications, such as quality control, robotics, and medical device manufacturing. For instance, in semiconductor manufacturing, these tools are employed to ensure the precise alignment of circular components on microchips.
The algorithms behind circle detection and evaluation tools operate within the camera's computational engine by analyzing pixel patterns and gradients. They look for circular shapes by identifying points that form the contours of circles or circular objects. The algorithms measure the size, position, and orientation of detected circles and evaluate them against predefined criteria. This process allows for the detection of anomalies, the assessment of quality, and the precise alignment of circular objects, contributing to enhanced productivity and quality control in industrial processes.
The picture above is an example of a using circle detection and evaluation to determine if a bullet casing has been damaged in handling before the gun powder has been filled and the actual bullet is inserted into it. If the casing is damaged, than the bullet might not be able to be inserted and and this could jam the assembly machine. In this way, machine vision can be used as a poka-yoke to avoid damaging the tooling in a machine or even simply wasting explosive powder for a bullet that won't correctly assemble.
v. Point to Line Measurements
Point-to-line measurement tools are essential features within industrial smart cameras, designed for accurately determining distances between a specific point and a defined line in an image. These tools enable precise spatial analysis and measurement, critical for applications in robotics, logistics, and quality control. For example, in logistics and warehousing, point-to-line measurement tools are used to verify the alignment of products on conveyor belts or to ensure proper spacing between items, streamlining sorting and packaging processes.
The functionality of point-to-line measurement tools relies on sophisticated algorithms that calculate the shortest distance from a designated point to a pre-defined line or reference. This capability has diverse applications. In automotive manufacturing, it aids in precisely positioning components on assembly lines, while in agriculture, it assists in optimizing planting and harvesting processes by ensuring consistent row spacing. This versatility makes point-to-line measurement tools invaluable assets in industrial settings, where spatial accuracy and efficiency are paramount.
vi. Defect Detection
Defect detection, often referred to as exception detection, is a crucial application of smart cameras in industrial settings. In this process, the camera is trained on "OK" (acceptable) and "NOK" (not acceptable or defective) images to learn the visual characteristics that distinguish between acceptable and defective products. Deep learning algorithms are then employed to interpret target images in real-time and identify any anomalies or defects. This technology has widespread use in quality control across industries such as manufacturing, electronics, and automotive, where maintaining consistent product quality is paramount.
Deep learning algorithms used for defect detection leverage neural networks to analyze complex visual data. They can detect subtle variations in texture, color, shape, or other visual attributes that may indicate defects or anomalies. By continuously comparing incoming images to the learned patterns from the training data, these smart cameras can swiftly flag and document any deviations from the expected standard, allowing for immediate corrective action. This not only ensures product quality but also enhances efficiency by reducing the need for manual inspections and minimizing the risk of defective products reaching customers.
3. Part Location and Pattern Matching
The first and often most crucial step in any machine vision application is locating the object or feature of interest within the camera's field of view. This is achieved through pattern matching technology. The success of the entire machine vision process often hinges on this initial step. If the system's pattern matching tools fail to accurately locate the part within the captured image, subsequent tasks like guidance, identification, gauging, and inspection become unreliable or even impossible.
Smart cameras also use circle measurement to detect locating holes and gives the part location for pick and place robots. The picture below shows a camera mounted around two meters away from a conveyor belt. When the cargo door comes under the smart camera, it will use circle measurement to differentiate the upper and lower locating holes from the other holes on the door, based on their exact diameters.
The cameras will then and report the center X & Y to the Programmable Logic Controller (PLC), over a communication protocol like EthernetIP. The robot now knows the coordinates of the locator holes and now can insert pins into the holes and lift the cargo door onto the welding rack for the next process.
4. Typical Industrial Smart Camera Applications
i. Automotive Industry
In the automotive sector, the margin for error is exceedingly low, given that even a minor defect can lead to catastrophic outcomes. Machine vision systems are instrumental in maintaining rigorous quality and safety standards. They not only help in meeting quality specifications but also contribute to increased efficiency, cost reduction, and traceability of components.
In the image below there are a number of tools being used by the smart camera to determine that the part has been assembled correctly: Three edge checks ensure that there have been no chips during the handling, the presence of the retaining clip and then logic so that all tools must be OK in order for the whole part to PASS.
Since the automotive industry requires many stages of assembly, there are many intermediate quality checks during the manufacturing process. You can imagine the cost that would be incurred if a transmission was missing a gear or even a seal. When there is a missing gear, the manufacturer will find out before the transmission makes it to being installed in final assembly. A missing or damaged seal could showup after the car has been driven for a while and will lead to catastrophic damage to the engine. If there was an O-ring missing in an engine instead of a conduit, that could lead to an engine leak, which is expensive to fix.
Presence/Absence detection is one of the most commonly used functions in industrial Smart Cameras. Checking for presence of proper welds, hardware such as washers or O-rings, components such as gears, springs, bushings or pins, checking for moisture-proofing agents on ECUs and fuse assemblies, verifying proper label application, vision systems can also verify the presence of defects like rust.
Judging Appearance and Defect Detection
Judging appearance of interior components after they are assembled and before they are installed is critical to ensuring that there are no scratches, misalignments or gaps. If this part were to be installed, it would need to be removed and replaced or risk a customer being unhappy because fit and finish. Below is an example picture of a scratch on the trim on a luxury car door that could be detected by a smart camera system, using some type of deep learning algorithm.
Often there are functional reasons why a part needs to be inspected. In the below image you have a mild steel that has been machined into a gear and a hardened, stainless steel bushing is required to be welded in order for it to perform it's function. If the weld is not present then the gear will not be able to torque properly and will simple spin. By training the algorithm of the camera on a few good and bad pictures, this becomes simple for the camera to evaluate.
ii. Food and Beverage
The food and beverage sectors are governed by ever-evolving inspection standards, often requiring full traceability to uniform packaging, correct fill amounts. Machine vision systems offer the advantage of automating these inspections while storing image and result data for future reference.
For frozen food that is produced in scale, machine vision systems can be used to ensure all of the food items are in the right place and the meal will have a consistent look and roughly the same amount of each dish type.
Presence Detection: Apart from ensuring proper case counts and the adherence of multiple components like straws to drink cartons, vision systems can also confirm the presence of safety seals on medication bottles or caps on condiment jars.
Appearance Inspections: Beyond Optical Character Recognition (OCR) for lot codes and checking the integrity of food trays, vision systems can inspect the color and texture of food items for spoilage and verify the print quality on medication labels.
Measurement/Alignment Checks: Besides label placement and seal width, vision systems can measure the dimensions of medical tablets, the volume of liquid in vials, and even the alignment of packaged goods within a box.
iii. Medical Industries
Machine vision technology is pivotal in the medical industry, ensuring the utmost precision, quality control, and traceability, which are crucial for patient safety and compliance with stringent regulatory standards.
Below are specific applications of machine vision in the medical sector:
Needle Sharpness Inspection:
Machine vision systems are employed to inspect the sharpness of needles, ensuring they meet the required specifications. This is crucial as a blunt or defective needle can cause unnecessary discomfort and harm to the patient.
Correct labeling is vital in the medical industry to avoid any misinformation or mix-up of medical products. Machine vision systems perform visual inspections to verify that labels are correctly applied, ensuring that all the information is accurate and properly positioned.
Package Integrity Check:
The integrity of medical packages is inspected using machine vision to detect any damages, tears, or compromises in the sealing. This is essential to maintain the sterility and safety of the medical products contained within.
Code Scanning and Quality Checks:
Machine vision systems are used to scan 1D barcodes and 2D QR codes and also perform other quality checks to ensure traceability of medical products. This is crucial for tracking the product's journey through the supply chain and ensuring that any defective or recalled products can be quickly identified and located.
Vial and Cap Inspection:
Machine vision inspects vials and caps for proper sealing, alignment, and presence of contaminants. This ensures the integrity and safety of the liquid medications stored within, preventing leaks and contamination.
Syringe Plunger Position Verification:
The position of the plunger in pre-filled syringes is verified using machine vision to ensure the correct dosage. Any deviation in the plunger position can result in incorrect dosages, which can be harmful to patients.
Measurement of Medical Components:
Machine vision systems are used to measure the dimensions of various medical components accurately, such as tubing and implants, ensuring they conform to the specified tolerances. This is vital for the functionality and safety of medical devices.
Pill Count and Verification:
Automated vision systems count the number of pills in bottles or blister packs and verify their color, shape, and size, ensuring the right quantity and type of medication is packaged.
The integration of machine vision in the medical industry is indispensable, addressing various applications from package integrity to intricate inspections like needle sharpness and blood sample analysis. These applications not only ensure the quality and safety of medical products but also enhance traceability and compliance with regulatory standards, ultimately contributing to improved patient care and safety.
iv. Electronic Devices Industry
As electronic devices like smartphones and gaming consoles continue to miniaturize, the need for more compact and precise components grows. Machine vision systems are crucial for maintaining the high level of accuracy required in these intricate assemblies.
Presence Detection: In addition to pins, connectors, and solder, vision systems can verify the presence of tiny components like resistors and capacitors on a PCB.
Appearance Inspections: Beyond crystal oscillators, IC molds, and LEDs, vision systems can inspect the quality of screen displays, the integrity of insulation on wires, and even the application of thermal paste on processors.
Measurement/Alignment Checks: Apart from connector pin co-planarity and tray orientation, vision systems can measure the thickness of insulation layers, the diameter of wire filaments, and the flatness of semiconductor wafers. Measuring angles of components installed on circuit boards, Besides battery positioning and bent connector terminals, vision systems can also measure the thickness of brake discs, the alignment of headlights, and even the gap distances between car body panels.
v. Resin and Container Industries
In industries involving molding and resins, machine vision systems are vital for automating inspections to ensure that defective products do not reach the consumer. These systems are particularly useful for processes like injection molding, plastic forming, laminations, and labeling.
Presence Detection: Besides resin coatings and cap presence, vision systems can also check for the presence of handles on containers or the correct placement of spouts on jugs.
Appearance Inspections: In addition to container inner surface inspections and product flash, vision systems can check for color consistency in plastics and the clarity of printed labels.
Measurement/Alignment Checks: Beyond gasket center misalignment and label misalignment, vision systems can measure the thickness of plastic walls, the diameter of bottle openings, and even the uniformity of resin layers.
By understanding the specific needs and challenges of each industry, machine vision systems can be tailored to offer robust, efficient, and cost-effective solutions for quality control and inspection.
5. Understanding Key Camera Specifications
When it comes to the effectiveness of machine vision applications, the specifications of your camera—such as resolution, color vs. mono, pixel size, internal focal length, field of view, viewing angles, magnification and working distance—can have a significant impact. Whether you're utilizing a Smart Camera or a Vision Sensor, a deep understanding of these parameters is essential for achieving optimal results.
Below is an example chart with 1.6 MP (Mega-Pixel) camera showing how the focal length and Working Distance (Installation Distance) affects the pixel accuracy or resolution.
i. Field of View (FoV)
The Field of View (FoV) is the area that a camera can capture in a single frame.It is determined by the size of the camera's sensor, the lens's focal length, and the working distance between the camera and the object.
Formula: FoV = 2 * arctan ( Sensor Size / (2 * Focal Length) )
Impact on Applications: A larger FoV allows you to use fewer cameras for broader applications. For example, a single camera with a large FoV could be sufficient for detecting scratches on an entire car door after assembly, thereby reducing the need for multiple cameras and lowering costs.
Resolution refers to the number of pixels contained in the camera's sensor. It is often expressed in megapixels and is critical for the level of detail a camera can capture.
Formula: Resolution = Sensor Width (in pixels) x Sensor Height (in pixels)
Impact on Applications: In measurement tasks, a high resolution is crucial for determining tolerances with high accuracy. For calculated values like angles and gauging, higher resolution allows for more precise and reliable measurements.
iii. Internal Focal Length
The internal focal length of a lens determines how "zoomed in" your images will appear and plays a significant role in determining the FoV.
Formula: Focal Length = 1 / Magnification
Impact on Applications: The appropriate focal length should be chosen based on the size of the object you're inspecting. For smaller objects requiring detailed capture, a lens with a longer focal length would be more suitable.
iv. Working Distance
The working distance is the space between the front of the lens and the object being imaged. It is essential for practical considerations like how the camera will be mounted.
Formula: Working Distance = FoV / (2 * tan ( Angle of View / 2 ))
Impact on Applications: The working distance can influence the lighting conditions, which in turn can affect the quality of captured images. In applications where the object is moving, such as on a conveyor belt, an appropriate working distance is crucial for capturing clear, in-focus images.
By meticulously considering these camera specifications, you can significantly enhance the performance and reliability of your machine vision applications, whether you're employing Smart Cameras or Vision Sensors. These aren't just theoretical considerations; they are practical tools for optimizing your machine vision setup.
An alternative to integrated smart cameras are Machine Vision systems that are comprised of cameras, computational controllers and lenses. In this manner, the user must select the correct lens that matches with their FOV and desired accuracy. The picture below defines the key terms for selecting the right lens type for the particular application.
For applications that rely on judging extremely small features, such observing any burrs or dents a needle used for surgery, magnification lenses can be used as well. To detect sharpness, the camera will need to detect the two edges that form the needle and give it its sharpness, and ensure that the edge is extremely straight with no deviations. The lens selected for this application will be very different than the lens use in the cargo door locating application above.
6. The Importance of Lighting and Part Location
One of the most challenging aspects of machine vision applications is dealing with lighting difficulties, especially when locating parts marked with specific features like plus signs. Inconsistent lighting conditions can significantly affect the system's ability to accurately identify and locate these parts, thereby impacting the overall effectiveness of the inspection process.
7. Challenges in Real-World Environments
While it may seem straightforward to locate a part based on its pattern, real-world production environments introduce a host of variables that can make this task exceedingly difficult. Even in tightly controlled processes, there can be some variability in how a part appears, affecting the system's ability to recognize it.
8. The Need for Intelligent Part Location Tools
For a machine vision system to deliver accurate, reliable, and repeatable results, its part location tools must be highly intelligent. These tools should be capable of quickly and accurately comparing training patterns to actual objects as they move down a production line. This ensures that the system can adapt to slight variations in part appearance, thereby maintaining high levels of accuracy and reliability.
9. PC-Based Industrial Cameras vs. Integrated Smart Cameras
Industrial vision systems have become an integral part of various industries, including food and beverage, cosmetics, pharmaceuticals, building materials, metal processing, electronic manufacturing, automobile manufacturing, and more. Currently, the market is dominated by PC-based solutions, with embedded solutions like smart cameras and x86 open platforms supplementing them.
Both industrial cameras and smart cameras serve similar purposes, but their applications and usage differ significantly.
Industrial Cameras: These are primarily PC-based solutions that require external controllers, lenses, and other equipment. The software for these cameras often needs specialized personnel for programming and debugging. Their primary advantage lies in their flexibility, allowing users to combine various algorithms and design interfaces to cater to complex detection requirements.
Smart Camera: These are integrated devices that can function independently without the need for external devices. They are essentially micro machine vision systems that combine image acquisition, storage, processing, and communication. The components of a smart camera include:
- Acquisition Module: Camera, lens, and light source.
- Processing Module: Vision controller, software, and algorithms.
- Communication Module: TCP, serial port, PLC protocols, etc.
Thanks to advancements in DSP, FPGA, and mass storage technology, smart cameras are becoming increasingly intelligent, catering to more complex machine vision applications. Their algorithms are packaged into fixed modules, allowing users to apply them directly without programming. This makes smart cameras user-friendly, simple to operate, and convenient for maintenance.
10. Traditional Algorithm vs. Deep Learning Traditional Algorithm
- Relies on the operator's subjective experience and judgment, making it challenging to ensure comprehensive inspection and maintain historical records.
- Requires specialized knowledge for product selection, installation, and parameter setting.
- Switching between different products for production demands skilled operation for parameter resetting.
- Position deviation of the product and background interference can lead to numerous false detections, compromising stability.
- Traditional vision sensors lack image capture, storage, and visual judgment capabilities.
Benefits of Deep Learning
- Simplifies operations, supporting detection tasks like presence, absence, and orientation.
- Supports both traditional vision algorithms and AI detection, catering to a wide range of detection needs.
- Low operational difficulty ensures that anyone can operate it with ease.
- Surface-based detection ensures stability and is unaffected by position deviations, providing accurate judgments.
- High-resolution imaging capabilities.
- Outputs or stores judgment results to databases, facilitating easier reviews.
11. Optical Character Recognition (OCR) in Machine Vision Systems
Optical Character Recognition (OCR) is a specialized sub-field within machine vision that focuses on the identification and conversion of different types of characters into machine-encoded text. OCR is widely used in various industries for tasks such as date/lot code verification, serial number identification, and label text verification.
i. Importance of ISO Certification in OCR
When it comes to OCR in industrial applications, adhering to international standards like ISO certifications can be crucial. The ISO 15415 and ISO 15416 standards, for instance, provide guidelines for the quality of printed and direct part marked (DPM) characters. These standards help in ensuring that the characters to be recognized meet certain quality criteria, thereby increasing the accuracy and reliability of the OCR process.
Quality and Score: ISO standards often include grading criteria that assess the quality of the characters based on factors such as contrast, distortion, and sharpness. A higher grade usually translates to a higher OCR success rate.
Consistency: Adhering to ISO standards ensures that the OCR system will provide consistent results, which is particularly important in industries where traceability and compliance are critical.
ii. Factors Affecting OCR Quality
Resolution: Just like in other machine vision applications, the resolution of the camera can significantly impact the OCR's ability to accurately identify characters. A higher resolution camera can capture more details, making it easier for the OCR algorithm to distinguish between characters that look similar.
Lighting Conditions: Proper lighting is essential for achieving high-quality OCR. Inconsistent or poor lighting can create shadows or glare, making it difficult for the OCR algorithm to accurately identify characters.
Font and Size: The type of font and its size can also impact OCR accuracy. Simple, clear fonts are generally easier to recognize than stylized or decorative fonts.
Background Noise: A clean, contrasting background can significantly improve OCR accuracy. Background noise, such as additional text or graphics, can confuse the OCR algorithm.
By understanding the importance of ISO certification and the factors that can affect OCR quality, you can optimize your machine vision system for high-accuracy character recognition, whether you're using Smart Cameras or Vision Sensors. This is not just beneficial for compliance but also for improving the overall efficiency and reliability of your automated processes.
12. Graphics Hardware and Algorithms in Modern Machine Vision Inspection Smart Cameras
Modern machine vision inspection smart cameras leverage advanced graphics hardware and sophisticated algorithms to perform intricate inspections and analyses in real-time. These components are crucial for the rapid processing and analysis of visual data, enabling smart cameras to detect, identify, and assess various elements within their field of view.
Graphics Processing Units (GPUs):
- GPUs are at the core of the graphics hardware in smart cameras, allowing for parallel processing of image data, which is essential for real-time analysis. NVIDIA and AMD are among the leading manufacturers of GPUs, providing powerful processing capabilities for machine vision applications.
Field-Programmable Gate Arrays (FPGAs):
- FPGAs are integrated circuits designed to be configured by the user after manufacturing. They are crucial for machine vision as they allow for the development of custom, application-specific integrated circuits, enabling optimized performance for specific vision tasks.
Application-Specific Integrated Circuits (ASICs):
- ASICs are custom-designed chips optimized for specific applications, offering high efficiency and performance. They are used in smart cameras for specialized tasks where maximum efficiency and minimum power consumption are critical.
Convolutional Neural Networks (CNNs):
- CNNs are a class of deep learning algorithms specifically designed for image recognition and analysis. They are adept at extracting hierarchical features from input images, making them ideal for object detection, classification, and segmentation in machine vision systems.
Image Processing Algorithms:
- Traditional image processing algorithms like edge detection, thresholding, and morphological operations continue to play a vital role in preprocessing the image data, enhancing features, and reducing noise, thereby aiding in more accurate analysis.
Object Detection and Classification Algorithms:
- Algorithms like YOLO (You Only Look Once) and SSD (Single Shot Multibox Detector) are used for detecting and classifying objects within images. These algorithms are capable of processing images in real-time, making them suitable for smart cameras in industrial inspection applications.
Optical Character Recognition (OCR) Algorithms:
- OCR algorithms are used to convert different types of characters into machine-encoded text. They are crucial for applications such as date/lot code verification, serial number identification, and label text verification.
3D Reconstruction Algorithms:
- These algorithms are used to reconstruct three-dimensional models of objects from two-dimensional image data. They are essential for applications that require depth perception and volumetric analysis, such as inspecting the dimensions of manufactured parts.