One researcher, Jennifer A. Mather, wondered: how do octopuses navigate? Do they rely on chemotactile sensory information (i.e. movement based on chemicals in the environment), or do they orient towards visual landmarks?
Octopuses occupy “homes” for several days or in some instances for several weeks, and when they go out looking for food, they are sometimes gone for several hours at a time. Therefore, they must use some sort of memory to find their way back home.
Many molluscs use trail-following, and follow their own mucus trails, or the trails of others. You might expect that octopuses use trail-following as well, since they forage by using chemotactile exploration – at least four different types of receptors on their suckers gather chemical and tactile information as they move along the rocky seafloor.
However, many other species use visual scene recognition to aid in navigation: ants, bees, gerbils, hamsters, pigeons, and even humans, use visual landmarks to navigate around their environments. Since octopuses use visual information to distinguish among different objects, they could use visual landmarks to get home as well.
To find out the researchers performed a series of experiments, testing chemicals, memory, and navigation. See what they learned - Scientific American.
** The article is kinda long so you can just read the last paragraph for the summary