Goldilocks and the Three Experiments
Once upon a time, during a usability test in a low and Nether-land, a seemingly meaningless pinch of the fingers by a middle-aged Dutch man named Geert sparked a series of three experiments by a girl with golden hair.
While looking through the search results on the tablet website of Booking.com, Geert decided he wanted to see the hotel photos in a little more detail, but when he zoomed in all he got was a pixelated mess of colors! This observation lit a fire in Goldilocks, and she enthusiastically thought, “Geert and other customers definitely want high-quality hotel photos that are beautiful and sharp!”
My mission became as clear as an SVG icon on a retina-quality screen. I would fight for Geert and his desire to zoom. And so, during the team’s next planning meeting, I included a task to improve the experience of how the photos on the search results and landing pages were displayed.
I began by digging through the treasure chest of goodies we refer to as the “tag dump” to see what sizes of images were available to use.
At the time, on both the search results and the landing pages, the website displayed performance-optimized 150 by 150 pixel hotel photos with a padding of ten pixels on each side used at their original dimensions.
I discovered that we also had images in other dimensions such as a maximum height and width of 500 pixels, a maximum height and width of 300 pixels, and a 200 pixel square were all available.
First, I coded the square 200 pixel images in place of the square 150 pixel versions and reviewed the quality of these slightly larger images on a variety of tablet devices at different resolutions.
“Square 200 pixel photos are still much, much too blurry,” said Goldilocks. “That doesn’t solve the problem at all!”
So I moved on to the 300 pixel tall/wide images. They looked fine and appeared to be more nicely cropped, but they still weren’t amazing.
At this point, my “designer-know-it-all sensibilities” wanted something really, really perfect looking. After all, Geert and our other customers deserved the best that was available.
When I coded up the 500 pixel photos and refreshed the page, I was blown away. I was certain that these photos were the best solution to the problem.
“Wow!” said Goldilocks. “500 pixel photos are cropped so nicely and are so sharp they will cut through the glass of the iPads and jump out at Geert and our other customers!”
Of course, I realized that 500 pixel photos have much larger file sizes than square 150 pixel photos and that this fact would inevitably increase the time it took for the page to load. So I decided to experiment with both the 500 pixel and the 300 pixel versions, and to keep a very close eye on the page load time.
I coded up two different experiments. One experiment I created on the search results page replacing the square 150 pixel images with 500 pixel ones. And I created another experiment on the landing pages replacing the square 150 pixel with 300 pixel images.
After running both experiments for a couple of weeks the results were finally in! Both of the experiments were absolutely, conclusively, and beautifully — neutral? Our customers seemingly didn’t mind.
I went straight to the analytics to understand what went wrong. Initially thinking that the page load time would be a key factor in these experiments that was the first thing I checked. I found that the page load time on search results using the 500 pixel photos increased by more than half a second.
I wasn’t horribly surprised. What actually surprised me was that the experiment still ended up being neutral despite the incredible performance hit. That discovery led me to wonder whether higher quality images could, at the very least, balance out the negative impact of the increased page load time.
I moved on to the landing pages to see how page load time was affected there. The increase in page load time was negligible, but the results were still neutral. This baffled me.
I stepped back from it for a couple of days and let everything subconsciously marinate.
After a few days, I was working on something in the search results completely unrelated to images when it hit me. This ten pixel padding around these photos serves absolutely no purpose. I’m going to get rid of it to make the images a little larger and clearer. It was the epiphany I needed.
Excitedly, I mentioned the idea to my design friends.
Phil suggested, “You could use ‘background-size: cover’ to make the images fit even better! They can get bigger or smaller depending on the size of the block.” Brilliant, Phil!
And Catalin said, “Decreasing the overall height of the content blocks is typically a good thing from what I’ve noticed.” Great insight, Catalin!
Finally, Goldilocks chimed in, “And if I use the 300 pixel tall/wide photos the pictures will be cropped more nicely, but page load time shouldn’t be a big issue!”
And that was it. I coded up the experiment with smaller content blocks and the higher-quality images. Then I released it to customers. After a couple of weeks the results were conclusive. It was positive!
“150 pixel squares were much too blurry. 500 pixel wide photos were far too heavy. But maximum dimensions of 300 pixel was just right!” declared Goldilocks as she enabled the experiment for all customers.
Reflecting on the process that led Goldilocks and her friends to the point of solving Geert’s problem, I realized that the initial “ideal solution” to the problem of using the 500 pixel photos because I liked the way they looked best could have been useless or even actively harmful. Without measuring how much our customers approve or disapprove of our application by using A/B testing and without the collaboration of my peers, I might have easily just changed the image quality based on my instincts and opinion.
Good design at Booking.com isn’t just about making something that seems to be better. It’s about measuring results, understanding the impact on the full customer experience, and actually making things better.
Getting feedback from our customers through A/B testing plays a key role in making things better. Couple that with the ability to collaborate with incredible colleagues who provide insight and potential solutions, and you have a winning combination for improvement.
The ability to abandon ego, truly listen to both customers and colleagues, and process this feedback is all part of the fairy tale magic.
Goldilocks would like to thank Phil Hammel and Catalin Bridinel for always humoring her ridiculous conversation topics about design-and-the-like and offering incredible feedback, solutions, and insights. She’d also like to thank “Geert” wherever he may be.
Originally published on Booking.com’s tech blog in August 2014.
Want to work with us? Check out www.workingatbooking.com
