aunnnn / ProjectsA Corner-based Saliency Model
A bottom-up computational visual attention model built on a simple but intriguing hypothesis: our eyes usually rest on areas with abrupt changes in color, which I call corners.
I turned this idea into a visual saliency prediction model.
This is the sample result:
Paper, Full thesis, Presentation video, Slides
General Problem
The field of computational visual attention tries to find algorithms that predict where we look. The general model takes an image as input and returns a grayscale map that indicates how much attention people will pay to each pixel. This field is very multidisciplinary, drawing on approaches from information theory, computer science, biology, psychology, and more.
Overview
There are two types of visual attention models: bottom-up and top-down.
Bottom-up models deal with low-level features such as luminosity, color, and orientation. Imagine a single red dot on a white paper. Your eyes are drawn to that area immediately. The dot "pops out" and you can't help but look at it, as if our eyes and brains are wired that way. Another example is a paper filled with a pattern of vertical bars where one bar is horizontal. Your eyes are drawn to the horizontal one.
Top-down models deal with high-level features that come from the individual and differ from person to person, like internal goals. They're much more complicated than bottom-up models. For example, if a paper is filled with dots of various colors and you're told to count the blue ones, your brain becomes "primed" to detect that single color.
A unified model would address both low-level and high-level features, but such an algorithm is still hard to find today. There's little existing research in psychology or biology to base the model on, so most papers run various experiments hoping to uncover the secret sauce behind the mechanism.
My contribution
I propose a bottom-up model called the corner-based saliency model, or CORS. It's based on the fact that corner information often attracts our eyes.
Try looking at a line. Did you spend the same amount of time on each segment of it? Probably not. You likely looked quickly at the starting and ending points, since the middle section isn't as important to you. I call these areas corners; for a rectangle, they're its four corners. Or look around where you are right now and notice how your eyes tend to rest on areas with abrupt changes in color or intensity.
CORS detects corners in an image and treats them as the areas with the highest visual attention. According to the MIT Saliency Benchmark, this simple idea turned out to be competitive.
What I learned
- Unlike software engineering, where we can estimate progress as we work, research is unpredictable, especially when the goal is to find something new.
- It's easy to fool yourself into thinking you're making progress just by reviewing more and more past work. I spent more than 90 percent of my time reading others' papers, and kept going until the deadline was only a couple of weeks away. That's when I realized I needed to stop and focus on my own model.
- Most of the time, a great idea turns out not to work in some particular case. There are endless edge cases to cover in this field, since our brains are so complex.
- A number of approaches nowadays use deep learning instead, since it lets researchers focus on higher-level problems rather than fine-tuning every detail.
- Sometimes you can't tell whether an idea is wrong or just needs more polishing.
- The evaluation metrics used in this field can be flawed, so you need to understand how they work first. There are various tricks to inflate a model's benchmark score. For example, a saliency map with a big white blob in the center usually scores well on most test images.
- This project took me a full year, so it's easily my proudest achievement so far!