Why Are Web Sites So Confusing?

Do you sometimes get the feeling that Internet portals, search pages, social networks, e-commerce, and other Web sites are not necessarily designed in order to maximize user convenience and benefits? We do, too.

Why—you might ask? For a fundamentally similar reason to why some retail stores place the most popular items (e.g., bread, milk) in the furthest possible place from the entrance; that shopping malls seem designed to make sure you get lost at every single visit; and that popular magazines drown the content they carry in a sea of advertising with no clear table of contents and split stories.

Indeed, all of these intermediaries are in the business of matching consumers with products. Trouble is, prior to visiting an intermediary, consumers are interested only in some products, which may not necessarily be the ones that yield the highest margins for the intermediary. If the latter was offering a perfect information service (i.e. one that enabled consumers to find what they want most quickly and efficiently), it would be losing valuable potential revenues. Hence the incentive to attract users with products that they want a priori and then divert them towards products that they might be interested in ex-post (i.e., once there).

Thus, consumers coming to the supermarket to buy daily staples (say, bread and milk) might be induced to also get expensive chocolate if they have to walk past the corresponding aisle anyway. Shoppers visiting a mall for its anchor store (say, Macy's) may decide to stop by a small design store while walking around the mall. And while flipping through the pages of a magazine in search of the article promised on the cover, readers are exposed to advertising, which produces most of the revenues.

In the same way, Google faces a subtle issue in designing its search result pages: consumers are mostly interested in the "objective" (i.e., middle) search results, but all revenues come from the sponsored search ads on the right hand side. The result is a compromise between what users want and what produces more revenues. For any given search, the 11th objective search result might be more relevant than any of the sponsored search results displayed on the right; yet it will be displayed on the second search page only—well beyond the reach of most users.

Most e-commerce sites nowadays (e.g., Amazon, iTunes, Netflix) use recommendation systems to suggest to each individual user products or content which might interest him/her, as inferred from their past behavior or the behavior of users with similar profiles. How much should you trust those recommendations? Not entirely, of course: while consistently irrelevant recommendations would eventually drive people away, the sites have an incentive to steer users to the products that yield them the highest margins—which may not always coincide with the ones that best correspond to users' preferences.

So how bad is this apparently insidious form of "search diversion"? (It started in the brick-and-mortar world but the digital economy provides many more subtle ways to divert search.) Well, it may appear that users lose whereas some vendors (and the intermediaries) benefit, but in a world without perfectly efficient search, potentially valuable product-consumer matches might go unrealized. Thus, enabling those matches through search diversion may, to a certain extent, provide a net benefit to society.

Turns out, there are benefits which go beyond that. A recent and thought-provoking Science article ("Electronic Publishing and the Narrowing of Science and Scholarship") shows that as library search has gotten vastly more efficient with the advent of digitized libraries and online search tools, the depth of scientific research has suffered. Scholars in a variety of fields tend to reference fewer articles and cast a narrower net when conducting their background research.

The old, inefficient search method, which relied on index cards and inevitably entailed flipping through pages of not necessarily relevant journals, had the benefit of exposing scientists to a wider range of ideas, which could potentially also widen the scope of their research. This kind of serendipity can turn out to be a vastly more valuable consequence of search diversion than stumbling upon new products at the supermarket or while browsing aimlessly through Amazon.com.

This article was provided with permission from Harvard Business School Working Knowledge.