Computer Vision for Memorability

Info

Reimplemementing and testing Khosla et al.’s MemNet[1], and beyond.

Goals

The original MemNet has proved difficult to use and access in recent years. We would like to implement it so that we can use their work for our own research. The short term goal is to simply get it working. In the long term we’d like to improve upon the architecture using pre-training, and possibly some other progress in neural network architecture that has been made since 2015. In addition we’d like to get a tool running for in-lab use. This will be implemented in PyTorch.

A decent amount of progress has been made on this project so far. The progress will be chronicled in a number of blog posts.

  1. MemNet and Similar Models
  2. ResMem Release
  3. ResMem and M3M
  4. Comparing ResMem and MemNet

A release version of ResMem, the best performing model I’ve made so far (at least that isn’t too large to run on personal computers) is publicly available! The source code can be found at github. But there’s an easy-to-install version on PyPI, with a web interface coming soon. Just type into your console:

pip install resmem

Resources

1 : Khosla, Aditya, Akhil S. Raju, Antonio Torralba, and Aude Oliva. 2015.“Understanding and Predicting Image Memorability at a Large Scale.” In2015 Ieee International Conference on Computer Vision (Iccv), 2390–8. IEEE.https://doi.org/10.1109/ICCV.2015.275.1

2 : Bordelon, B., Canatar, A., & Pehlevan, C. (2020). Spectrum dependent learning curves in kernel regression and wide neural networks. ArXiv:2002.02561 [Cs, Stat]. http://arxiv.org/abs/2002.02561

3 : L. Goetschalckx, A. Andonian, A. Oliva, and P. Isola. GANalyze: Toward Visual Definitions of Cognitive Image Properties. , 2019. http://ganalyze.csail.mit.edu/

4 : https://docs.daft-pgm.org/en/latest/

Research Assistant (Pre-Doctoral)

Student, Data Scientist, Philosopher, Mathematician, Freelancer. Currently working as a Research Assistant at Microsoft Research, pontificating about video games, and learning to play the classical guitar.