A Dual-Stream Deep Learning Architecture for Action Recognition in Salmon from Underwater Video.

Abstract

Over half of the costs from breeding salmon in the Norwegian salmon farming industry comes from feed usage. Today the feeding process is largely a manual labor, requiring an operator to monitor the amount of feed sinking to the bottom of a breeding cage. When the amount of feed exceeds a certain threshold, the feeding process is terminated. Automation of this process and using salmon motion behavior instead of sinking feed to determine when to terminate feeding, could greatly reduce costs both in through the labor needed and amount of feed wasted. Resent developments in Human Action Recognition have shown that Deep Learning approaches are well suited to perform Action Recognition. We therefore examine the feasibility of using Deep Learning approaches to automate the feeding process. We use 76 videos of salmon collected from within a breeding cage during the month of November 2016. Using these videos for training, validation and testing, we propose three approaches to automatically classify Feeding and NonFeeding behavior in salmon. The three approaches are a Spatial Architecture, a Spatial Recurrent Architecture and a Dual-Stream Architecture. Our results show that all our proposed architectures are able to separate videos of Feeding and NonFeeding salmon with high accuracy. We also find that the Dual- Stream Architecture is the best performing architecture. It combines spatial and temporal information through the use of a Spatial Stream, a novel Temporal Stream and a Recurrent Neural Network (RNN). Our Dual-Stream Architecture is able to accurately classify 80.0% all of our testing videos, presenting state-of-the-art performance. To the best of our knowledge, both our Temporal Stream and our Dual-Stream Architecture are original and novel architectures, as is the application of Deep Learning inference models for the Salmon Activity Recognition domain in optimization of feeding operation in Norwegian Aquaculture. We hope the results presented in this thesis will contribute to achieve a higher sustainability in Norwegian salmon aquaculture, optimize feeding operations, and consequently reduce potential waste. Future work beyond the results presented in this thesis concerns research on understanding of what our the Deep Learning architecture have learned and visualizing this learning process.