| dc.description.abstract | The fisheries and fish processing industries have high labor costs. In part, this is due to visual inspection tasks, which are found in these industries, being so complex that humans are needed to do the inspection. One large class of inspection tasks is defect detection. This involves discerning whether a part or product is defective. Machine vision is a technology with the potential for solving many visually-based defect detection tasks, and has been successfully applied in automation systems in many industries (Hirano et.al. 2006). In this dissertation, machine vision is used successfully for defect detection in a number of specific applications in the fisheries and fish processing industries, and the particular challenges of applying machine vision, to these types of applications, are highlighted. Methods and approaches for meeting these challenges are discussed, with reference to their successful use the applications presented in this dissertation. A brief summary of each of the applications is presented in the following sections.
The first application that was addressed, and the initial topic of this dissertation, was automatic sorting of cod viscera. Cod viscera are a valuable byproduct, containing different fractions. When the cod viscera is sorted into separate fractions, such as liver, roe and stomach, its value increases substantially. Sorting of cod viscera is done manually, and is a laborious process in the cod fisheries. This was the motivation for developing a method for automatic sorting of cod viscera, using machine vision and a robot. We developed and implemented hand-eye coordination algorithms for a 5-DOF robot with a stereo camera platform. Assuming that a viscera fraction could be identified in images from both cameras, we thus had a method of directing the robot tool to the fraction. The remaining challenge was to identify the fraction in the images. Towards this end, we tested several approaches, culminating in a combination of color and texture features together with a linear discriminant analysis (LDA) classifier. We found the fraction detection, in visible light images, to be a very difficult problem to solve, and it was not solved adequately to be useful for automatic sorting of the cod viscera within the time-frame given. In light of this, the focus of this dissertation was changed – to encompass machine vision for defect detection in a number of specific fisheries and fish processing applications.
The second application is a cod fisheries application – detecting nematodes and skin remnants, in cod fillets, using multi-spectral images in the visible and near-infrared. A combination of contrast-invariant Gabor texture features and linear discriminant analysis showed promising results for detecting both nematodes and skin remnants. The pixel-wise detection rate and false-alarm rate for nematode detection was on the order of 99% and 1% respectively, and for skin remnant detection approximately 90% and 10%. Combining this with multi-spectral features gave even better results. The data set was not large enough to draw any definitive conclusions in terms of detection rate and false-alarm rate in an industrial setting, but the results are promising enough to merit further investigation into the use of Gabor texture features for detecting nematodes and skin remnants in cod fillets.
The third application is a fish processing application, and involves quality sorting of whole Atlantic salmon into the classes ‘superior/ordinary’ and ‘production’, in which case the ‘production’ are seen as being defective with respect to the ‘superior/ordinary’. Salmon characterized as ‘production’ may have several deviations from ‘superior/ordinary’ salmon, including a humpback or short tail. The approach taken, to solve the sorting problem described here, was to first segment the image in order to extract the salmon silhouette and orient it properly, independent of its original orientation. From this silhouette image, specific geometrical descriptive parameters were extracted. In the feature-space spanned by these descriptors, a linear discriminant analysis (LDA) was done to maximally discriminate between ‘production’ and ‘superior/ordinary’. This performance of this approach was analyzed using a leave-one-out (LOO) cross-validation procedure, and this showed that approximately 90% of salmon could be correctly classified as ‘superior/ordinary’ or ‘production’. This machine vision system was further developed to discriminate between the more similar classes ‘superior’ and ‘ordinary’, and tested on a larger data set. The sorting accuracy for this system was also approximately 90 %. Thus, the system is well suited as part of automated system for quality sorting of whole Atlantic salmon.
The fourth application is in the longline fisheries, and the task here is to detect defective longline hooks. In longline fishing, lines of up to 50-60 kilometers are used, each containing up to 40 000 hooks. Today, baiting, shooting (setting the line out in the sea) and hauling (pulling it back on board after soaking for some time in the sea) is almost completely automated. Between each hauling and the next shooting, the gear (line and hooks) is manually inspected, and any defective hooks are corrected or replaced. This manually-executed maintenance task is extremely labor-intensive. A machine vision system is presented, that can potentially automate this task. Combining knowledge from the fishermen, on what denotes a ‘defective’ hook, and machine vision knowledge, a set of features were automatically extracted from images of the hooks. Thousands of classifiers were automatically explored, and each classifier was plotted in the receiver-operator characteristic (ROC) plane. From all of these, the classifiers on the ROC convex hull (ROCCH) were selected. On tests with more than 400 hooks, the classifiers on the ROOCH were found capable of detecting 97.4 % of ‘defective’ hooks with zero false positives. Together with an automatic and mechanized hook replacement and repair system, this machine vision approach can automate the maintenance phase of longline fishing.
The fifth application is in the pelagic fisheries and fish processing, and is the task of weight and quality grading of whole pelagic fish. In this work, the goal was to develop a single machine vision system that could be used to automatically perform both weight and quality grading – a labor-intensive task that is done manually today. This involved finding the optics, illumination and imaging modalities that could be used to weigh the fish automatically and detect most of the important defect types that could occur. The types of defects that occur in whole pelagic fish are many, including: split fish, superficial wounds, such as cuts scrapes, and deformations due to pressure and injuries. An imaging and illumination system, that combined 3D range imaging, laser scatter imaging and diffuse gloss imaging, was developed and implemented in a prototype sorting station. Experiments showed that these imaging modalities could be used to detect all the defect types at a high processing speed. The prototype machine vision system was integrated with a conveyor belt and robot, and demonstrated the feasibility of using machine vision to automatically weigh whole pelagic and detect those fish that are defective. The main focus of this work was to develop the optical, imaging and illumination part of the machine vision system, and demonstrate proof-of-concept machine vision algorithms that could discriminate between defective and non-defective fish.
The sixth application is detection of melanin spots in Atlantic salmon fillets. One of the manual operations done in salmon processing plants is to inspect the fillets and discard or process those that have melanin spots. Melanin spots are dark spots that reduce the quality grade of the fillet, and they are present in a substantial percentage of fillets. Melanin spots are caused by an inflammatory condition most often induced by vaccination. For the melanin spot detection application, a machine vision system using linear discriminant analysis (LDA) was developed that could detect 93% of melanin spots. At this detection rate the number of false alarms was very high, but an investigation showed that these false alarms were entirely due to two main causes: 1. uneven illumination and 2. blood, viscera and fin remnants near the belly flap. By improving the illumination uniformity, and cleaning and trimming the fillets before inspection, we thus have a method for accurate detection of melanin spots. This system uses RGB images, and can thus be easily integrated in a fillet processing line where other quality parameters, such as trimming grade, pigmentation and shape, are measured using machine vision with RGB cameras.
The seventh application is automatic trimming of salmon fillets, in which fin and tail remnants, belly fat, back fat and belly membranes are detected and removed. Trimming fillets is today done manually, after rough trimming in mechanical trimming machines. This application was developed into a prototype sorting station, with conveyor belt, robot and camera and illumination housing. As part of this complete prototype, a machine vision system had to be developed that could distinguish between the ‘defective’ and ‘non-defective’ parts of the fillet. The solution that was found involved using a 3D camera with blue LED illumination. This gave use both the required 3D data needed to perform the trimming, and the illumination needed to acquire images in which the ‘defective’ parts of the fillet were clearly discriminated. Once the images clearly distinguished between objects to be trimmed off (‘defective’ parts) and the remaining fillet, the subsequent image processing was much simplified. The work in this application illustrates the importance of looking at the entire machine vision system, and its context, as a whole. The prototype was fully functioning and demonstrated all the necessary parts needed in an automatic salmon fillet trimming station.
The eighth application is automatic sorting of salmonid eggs. Salmon breeding companies can produce up to 100 million eggs per year, and these must be manually inspected before delivery to the hatcheries. This is a highly laborious and costly task, and is therefore very desirable to automate. This manual sorting is needed to remove unfertilized eggs, dead eggs and fertilized eggs with defects. A machine vision system, including optics, illumination and image processing, was developed that can sort more than 100 000 eggs per hour with a sorting error of less than 1%. This machine vision system was demonstrated in a prototype sorting machine, and then implemented in an industrialized sorting machine. These sorting machines are now successfully in use at several salmon breeding facilities.
The applications developed and described in this thesis demonstrate the effectiveness of machine vision for defect detection in fisheries and fish processing applications. The work on these applications shows that in order to be successful, all aspects of the application, and their context, must be taken into account. In each application there are specific constraints that make each application unique. Even so, there are some commonalities between many applications, in terms of the challenges involved. These commonalities have been highlighted in this dissertation, and appropriate methods and approaches have been found to meet them. Being aware of these challenges, and of the relevant methods and approaches to meeting them, increases the probability of successfully solving an application. As such, the work in this dissertation, and understanding of the lessons learnt in it, will benefit future applications of machine vision for defect detection in the fisheries and fish processing industries.
Through continuing improvements in machine vision technology, and increasing domainspecific knowledge of how to apply this technology to fisheries and fish processing applications, almost all such applications can be partially or fully automated in the future. This dissertation shows that this vision is indeed a possibility, and hints at the directions needed to realize it.
| nb_NO |
