Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis

Abstract

Iconography studies the visual content of artworks by considering the themes portrayedin them and their representation. Computer Vision has been used to identify iconographic subjectsin paintings and Convolutional Neural Networks enabled the effective classification of charactersin Christian art paintings. However, it still has to be demonstrated if the classification resultsobtained by CNNs rely on the same iconographic properties that human experts exploit whenstudying iconography and if the architecture of a classifier trained on whole artwork images canbe exploited to support the much harder task of object detection. A suitable approach for exposingthe process of classification by neural models relies on Class Activation Maps, which emphasize theareas of an image contributing the most to the classification. This work compares state-of-the-artalgorithms (CAM, Grad-CAM, Grad-CAM++, and Smooth Grad-CAM++) in terms of their capacityof identifying the iconographic attributes that determine the classification of characters in Christianart paintings. Quantitative and qualitative analyses show that Grad-CAM, Grad-CAM++, andSmooth Grad-CAM++ have similar performances while CAM has lower efficacy. Smooth Grad-CAM++ isolates multiple disconnected image regions that identify small iconographic symbols well.Grad-CAM produces wider and more contiguous areas that cover large iconographic symbols better.The salient image areas computed by the CAM algorithms have been used to estimate object-levelbounding boxes and a quantitative analysis shows that the boxes estimated with Grad-CAM reach55% average IoU, 61% GT-known localization and 31% mAP. The obtained results are a step towardsthe computer-aided study of the variations of iconographic elements positioning and mutual relationsin artworks and open the way to the automatic creation of bounding boxes for training detectors oficonographic symbols in Christian art images.