INTRODUCTION
FPGAs are ubiquitous in machine vision systems; they are a part of the camera, the framegrabber, and the image processing module. This is with good reason, as they offer many advantages when it comes to image processing. FPGAs offer vast parallel processing capabilities, reliable data handling and data transfer, and flexibility in connecting to various interfaces.
FPGA for IMAGE SIGNAL PROCESSING
The raw image data signals captured by the camera sensor are conditioned before they can be used by the image processing module. These operations are for image enhancement and corrections and include processes like noise reduction, pixel correction, white balancing etc., which are executed by an image signal processor (ISP).
An FPGA can be used as an ISP, as the algorithms can take advantage of the spatial parallelism offered by the FPGA, and execute with minimal latency. For example, consider demosaicing, which is an algorithm used to reconstruct a full-colour image from the raw sensor data by applying a colour filter array (CFA). The CFA is applied by the ISP in parallel to all the pixels at the same time, which significantly reduces the execution time, thereby improving the system performance.
FPGA FOR PRE-PROCESSING
As a pre-processor, the FPGA offers two main benefits: bandwidth reduction and efficient pixel-level operations.
- Bandwidth reduction reduces the amount of data that the main processing unit (CPU or GPU) needs to process by cropping the image based on the Region of interest.
- FPGAs can be used for executing many low to mid-level algorithms such as colour space conversion efficiently. FPGAs are efficient in such pixel-level operations because they can execute the datastream operation in parallel.
The pre-processing using FPGA can be achieved by using programmable Framegrabbers or by using the FPGA inside the camera like in the case of Mikrotron EoSens.
Some of the advantages of pre-processing are –
- Reduced load on downstream subsystems: Bandwidth reduction reduces the load on downstream systems like the vision processor. Because of the reduced bandwidth, it is possible to reduce the cost by using a low or medium performance vision processor.
- Component reduction: In the case of Mikrotron Eosens, the reduction in bandwidth enables using an interface like 10GigE that does not require components like Framegrabbers. It also reduces the cost of cables used for data transfer
FPGA AS AN ACCELERATOR
FPGA can also be used alongside a CPU & GPU as an accelerator in a Machine Vision system. The CPU can be used to execute a complete image processing pipeline with FPGA & GPU as co-processors that accelerate algorithms that are part of the pipeline.
FPGA FOR COMPLETE IMAGE PROCESSING PIPELINE
The FPGA can also be used to execute the complete image processing pipeline. This is particularly useful for hard-real time applications like optical sorting. A typical pipeline for an optical sorting application is as follows: 
These systems manage large volumes of pixel data at high speeds and require a hardware architecture that has low latency and reliability. FPGAs are best suited for such applications as the image processing pipeline is restricted to pixel-level processing.
CONCLUSION
YantraVision offers custom development of image processing IPs on FPGA for a diverse range of applications including hard real-time applications. We have experience in building complete image processing pipelines on FPGA for hard real-time applications like Optical sorting of Grains.