Coen D. Needell, a recent Brain Bridge Lab graduate, has published his first paper! The article, entitled “Embracing New Techniques in Deep Learning for Estimating Image Memorability,” describes ResMem, the newest and best performing deep neural network for estimating image memorability. Previous DNNs attempting to estimate the intrinsic memorability of images have achieved fairly high correlations with human behavioral scores, but rely on now-outdated techniques. In the current work, Coen and Dr. Bainbridge evaluated a mix of old and novel deep learning approaches to estimating memorability. They propose ResMem and M3M, two residual neural networks that use semantic information to estimate image memorability, in addition to perceptual features. ResMem and M3M were able to estimate ground truth scores with Spearman rank correlations of 0.67 and 0.68, respectively, greatly outperforming the earlier models. ResMem’s distribution of memorability predictions also closely matched that of ground truth memory performance. When broken down by image category, ResMem performed the best on images of landscapes (Spearman correlation = 0.81). Network feature analysis revealed that earlier layers of the model rely on basic perceptual features, while later layers rely on objects like body parts, as well as more complex conceptual features. These results demonstrate the ability of residual networks using semantic information to mimic human memory performance, and open the door for further exploration of what makes an image memorable. Read the article, published in Computational Brain & Behavior, here.
ResMem is available as a Python package, with code for the network here. Anyone can upload their own image and instantly get a memorability score on our web app. If you have tried ResMem on your own database, please let us know here!