Object Detection and Tracking With UAVs: A Framework for UAV Object Detection and Tracking Using a Thermal Imaging Camera

Abstract

In recent years there has been a drastic increase in the commercial availability of small UAVs for civil and commercial applications. These applications includes, but are not limited to, inspections of structures, search and rescue, monitoring and surveillance of oil spills and autonomous border control. All of these applications emphasize the UAV’s role as a remote sensing platform with the main focus on visual sensors and data collection, and also includes an object detection and tracking component. Object detection and tracking in UAVs using visual sensors can generally be divided into two subproblems, where the first problem is that of controlling the path and orientation of the UAV in such a way that the UAV is able to capture useful data with its on-board sensors. The second problem is that of making the UAV able to automatically detect and track objects of interest using the UAV’s on-board sensors and autopilot. The former subproblem is considered a problem of path planning, while the second subproblem is considered a problem of machine vision. This monograph is motivated by the lack of a commercial agile and adaptable object detection and tracking framework for UAVs equipped with a visual sensor. Existing solutions are often application specific and implemented ad hoc, resulting in rigid object detection and tracking systems which are difficult to adapt and reconfigure for use in other similar applications. This thesis presents a novel framework for object detection and tracking in UAVs equipped with a visual sensor where the focus is to make it as configurable, modularized and as hardware independent as possible. Chapter 2 presents a modularized object detection and tracking framework consisting of a path planner, machine vision and object handler modules. Each module solves a specific sub-problem of the overall object detection and tracking problem, and each module has a wide range of applicable solutions which could be implemented and mixed with the remainder of the system. Chapter 2 further describes a software toolchain which provides a bridge and abstraction layer for the object detection and tracking framework’s architecture and the UAV’s on-board hardware components. This makes the software toolchain compatible with the idea of easily being able to configure the individual modules and hardware of the overall UAV object detection and tracking system. This is exemplified by the implementation of the hardware and software modules of the object detection and tracking framework in two different UAV platforms. In Chapter 3, a path planner module based on model predictive control (MPC) is developed. This controller uses a kinematic model of the UAV with certain constraints on the UAV’s motion to generate a feasible object tracking path for the UAV. That is, the MPC assumes that the visual sensor is mounted in a gimbal with two degrees of freedom and supplies the on-board autopilot with waypoints and gimbal control input which enables the UAV to track an object or group of objects. The gimbal is controlled in a way to ensure that the gimbal is pointing towards the tracked object’s estimated position. The MPC is demonstrated through field experiments in Chapter 3 and 5 to be able to successfully make both of the UAV platforms developed in this thesis able to track objects. Chapter 4 develops a solution for the machine vision module as an object detection, recognition and tracking module compatible with the developed object detection and tracking framework. The module enables the UAV to perform on-board image processing of thermal images in order to automatically segment the images into either background or objects of interest. Using the on-board telemetry data indicating the UAVs position and attitude coupled with the segmented images, the module is able to use a Kalman filter to track and estimate the position and velocity of the detected objects of interest. Chapter 5 demonstrates that the developed object detection and tracking framework is agile and adaptable by replacing the machine vision module of the system with a module more suitable for use in a sea ice management application. Furthermore, the nominal tests of the alternative modules developed in this chapter show promising results for the use of UAVs in the area of sea ice management. A full scale test of the object detection and tracking framework where all of the modules of the framework are activated and interacting simultaneously is still a test that has to be conducted, although the nominal testing in Chapter 5 demonstrates the viability of the framework and its implemented modules.