Learning from Amazon’s culture of metrics

1. Context

Why a case study on Amazon? Surely everyone knows about who Amazon are and what they do? Yes, well, that may be true, but this case goes beyond the surface to review some of the ‘insider secrets’ of Amazon’s success.

Like eBay, Amazon.com was born in 1995. The name reflected the vision of Jeff Bezos to produce a large- scale phenomenon like the River Amazon. This ambition has proved justified since, just 8 years later, Amazon passed the $5 billion sales mark – it took Wal-Mart 20 years to achieve this.

By 2008 Amazon is a global brand with over 76 million active customers accounts and order fulfilment to more than 200 countries. Despite this volume of sales, at 31 December 2007 Amazon employed approximately 17,000 full-time and part-time employees. In September 2007, it launched Amazon MP3, a la carte DRM-free MP3 music downloads, which now includes over 3.1 million songs from more than 270,000 artists.

2. Vision and strategy

In their 2008 SEC filing, Amazon describes the vision of their business as to:

Relentlessly focus on customer experience by offering our customers low prices, convenience, and a wide selection of merchandise.

The vision is to offer Earth’s biggest selection and to be Earth’s most customer-centric company. Consider how these core marketing messages summarizing the Amazon online value proposition are communicated both on-site and through offline communications.

Of course, achieving customer loyalty and repeat purchases has been key to Amazon’s success. Many dot-coms failed because they succeeded in achieving awareness, but not loyalty. Amazon achieved both. In their SEC filing they stress how they seek to achieve this. They say:

We work to earn repeat purchases by providing easy- to-use functionality, fast and reliable fulfillment, timely customer service, feature rich content, and a trusted transaction environment. Key features of our websites include editorial and customer reviews; manufacturer product information; Web pages tailored to individual preferences, such as recommendations and notifica­tions; 1-Click® technology; secure payment systems; image uploads; searching on our websites as well as the Internet; browsing; and the ability to view selected interior pages and citations, and search the entire contents of many of the books we offer with our ‘Look Inside the Book’ and ‘Search Inside the Book’ features. Our community of online customers also creates feature-rich content, including product reviews, online recommendation lists, wish lists, buying guides, and wedding and baby registries.

In practice, as is the practice for many online retailers, the lowest prices are for the most popular products, with less popular products commanding higher prices and a greater margin for Amazon. Free shipping offers are used to encourage increase in basket size since customers have to spend over a certain amount to receive free ship­ping. The level at which free shipping is set is critical to profitability and Amazon has changed it as competition has changed and for promotional reasons.

Amazon communicates the fulfilment promise in several ways including presentation of latest inventory availability information, delivery date estimates, and options for expedited delivery, as well as delivery ship­ment notifications and update facilities.

This focus on the customer has translated to excellence in service with the 2004 American Customer Satisfaction Index giving Amazon.com a score of 88, which was at the time the highest customer satisfaction score ever recorded in any service industry, online or offline.

Round (2004) notes that Amazon focuses on customer satisfaction metrics. Each site is closely moni­tored with standard service availability monitoring (for example, using Keynote or Mercury Interactive) site availability and download speed. Interestingly, it also monitors per-minute site revenue upper/lower bounds – Round describes an alarm system rather like a power plant where if revenue on a site falls below $10,000 per minute, alarms go off! There are also internal perform­ance service-level agreements for web services where T% of the time, different pages must return in X seconds.

3. Competition

In its SEC (2005) filing Amazon describes the environment for its products and services as ‘intensely competitive’. It views its main current and potential competitors as: (1) physical-world retailers, catalogue retailers, publishers, vendors, distributors and manufacturers of products, many of which possess significant brand awareness, sales volume and customer bases, and some of which currently sell, or may sell, products or services through the Internet, mail-order or direct marketing; (2) other online e-commerce sites; (3) a number of indirect competitors, including media companies, web portals, comparison shopping web sites, and web search engines, either directly or in collaboration with other retailers; and (4) companies that provide e-commerce services, including web site development; third-party fulfilment and customer service.

It believes the main competitive factors in its market segments include ‘selection, price, availability, conven­ience, information, discovery, brand recognition, personalized services, accessibility, customer service, reliability, speed of fulfillment, ease of use, and ability to adapt to changing conditions, as well as our customers’ overall experience and trust in transactions with us and facilitated by us on behalf of third-party sellers’.

For services offered to business and individual sellers, additional competitive factors include the quality of their services and tools, their ability to generate sales for third parties they serve, and the speed of perform­ance for their services.

4. From auctions to marketplaces

Amazon auctions (known as zShops) were launched in March 1999, in large part as a response to the success of eBay. They were promoted heavily from the home page, category pages and individual product pages. Despite this, a year after launch they had only achieved a 3.2% share of the online auction compared to 58% for eBay and it only declined from this point.

Today, competitive prices of products are available through third-party sellers in the ‘Amazon Marketplace’ which are integrated within the standard product list­ings. The strategy to offer such an auction facility was initially driven by the need to compete with eBay, but now the strategy has been adjusted such that Amazon describes it as part of the approach of low pricing.

Although it might be thought that Amazon would lose out on enabling its merchants to sell products at lower prices, in fact Amazon makes greater margin on these sales since merchants are charged a commission on each sale and it is the merchant who bears the cost of storing inventory and fulfilling the product to customers.

As with eBay, Amazon is just facilitating the exchange of bits and bytes between buyers and sellers without the need to distribute physical products.

5. How ‘the culture of metrics’ started

A common theme in Amazon’s development is the drive to use a measured approach to all aspects of the busi­ness, beyond the finance. Marcus (2004) describes an occasion at a corporate ‘boot-camp’ in January 1997 when Amazon CEO Jeff Bezos ‘saw the light’. ‘At Amazon, we will have a Culture of Metrics’, he said while addressing his senior staff. He went on to explain how web-based business gave Amazon an ‘amazing window into human behavior’. Marcus says: ‘Gone were the fuzzy approximations of focus groups, the anecdotal fudging and smoke blowing from the marketing depart­ment. A company like Amazon could (and did) record every move a visitor made, every last click and twitch of the mouse. As the data piled up into virtual heaps, hummocks and mountain ranges, you could draw all sorts of conclusions about their chimerical nature, the consumer. In this sense, Amazon was not merely a store, but an immense repository of facts. All we needed were the right equations to plug into them.’

James Marcus then goes on to give a fascinating insight into a breakout group discussion of how Amazon could better use measures to improve its performance. Marcus was in the Bezos group, brainstorming customer-centric metrics. Marcus (2004) summarizes the dialogue, led by Bezos:

‘First, we figure out which things we’d like to measure on the site’, he said. ‘For example, let’s say we want a metric for customer enjoyment. How could we calculate that?’

There was silence. Then somebody ventured: ‘How much time each customer spends on the site?’

‘Not specific enough’, Jeff said.

‘How about the average number of minutes each customer spends on the site per session’, someone else suggested. ‘If that goes up, they’re having a blast.’

‘But how do we factor in purchase?’, I [Marcus] said feeling proud of myself. ‘Is that a measure of enjoyment’?

‘I think we need to consider frequency of visits, too’, said a dark-haired woman I didn’t recognize. ‘Lot of folks are still accessing the web with those creepy-crawly modems. Four short visits from them might be just as good as one visit from a guy with a T-1. Maybe better.’

‘Good point’, Jeff said. ‘And anyway, enjoyment is just the start. In the end, we should be measuring customer ecstasy.’

It is interesting that Amazon was having this debate about the elements of RFM analysis (described in Chaffey et al., 2009, Chapter 6) in 1997, after already having achieved $16 million of revenue in the previous year. Of course, this is a miniscule amount compared with today’s billions of dollar turnover. The important point was that this was the start of a focus on metrics which can be seen through the description of Matt Round’s work later in this case study.

6. From human to software-based recommendations

Amazon has developed internal tools to support this ‘culture of metrics’. Marcus (2004) describes how the ‘Creator Metrics’ tool shows content creators how well their product listings and product copy are working. For each content editor such as Marcus, it retrieves all recently posted documents including articles, inter­views, booklists and features. For each one it then gives a conversion rate to sale plus the number of page views, adds (added to basket) and repels (content requested, but the back button then used). In time, the work of editorial reviewers such as Marcus was marginalized since Amazon found that the majority of visitors used the search tools rather than read editorials and they responded to the personalized recommendations as the matching technology improved (Marcus likens early recommendations techniques to ‘going shopping with the village idiot.’)

7. Experimentation and testing at Amazon

The ‘culture of metrics’ also led to a test-driven approach to improving results at Amazon. Matt Round, speaking at E-metrics 2004 when he was director of personalization at Amazon, describes the philosophy as ‘data trumps intuitions’. He explained how Amazon used to have a lot of arguments about which content and promotion should go on the all-important home page or category pages. He described how every category VP wanted top-centre and how the Friday meetings about placements for next week were getting ‘too long, too loud, and lacked performance data’.

But today ‘automation replaces intuitions’ and real­time experimentation tests are always run to answer these questions since actual consumer behaviour is the best way to decide upon tactics.

Marcus (2004) also notes that Amazon has a culture of experiments, of which A/B tests are key components. Examples where A/B tests are used include new home page design, moving features around the page, different algorithms for recommendations, changing search rele­vance rankings. These involve testing a new treatment against a previous control for a limited time of a few days or a week. The system will randomly show one or more treatments to visitors and measure a range of parameters such as units sold and revenue by category (and total), session time, session length, etc. The new features will usually be launched if the desired metrics are statistically significantly better. Statistical tests are a challenge though, as distributions are not normal (they have a large mass at zero for example, of no purchase). There are other challenges since multiple A/B tests are running every day and A/B tests may overlap and so conflict. There are also longer-term effects where some features are ‘cool’ for the first two weeks and the oppo­site effect where changing navigation may degrade performance temporarily. Amazon also finds that as its users evolve in their online experience the way they act online has changed. This means that Amazon has to constantly test and evolve its features.

8. Technology

It follows that the Amazon technology infrastructure must readily support this culture of experimentation and this can be difficult to achieve with standardized content manage­ment. Amazon has achieved its competitive advantage through developing its technology internally and with a significant investment in this which may not be available to other organizations without the right focus on the online channels. As Amazon explains in SEC (2005),

using primarily our own proprietary technologies, as well as technology licensed from third parties, we have implemented numerous features and function­ality that simplify and improve the customer shopping experience, enable third parties to sell on our plat­form, and facilitate our fulfillment and customer service operations. Our current strategy is to focus our development efforts on continuous innovation by creating and enhancing the specialized, proprietary software that is unique to our business, and to license or acquire commercially-developed technology for other applications where available and appropriate. We continually invest in several areas of technology, including our seller platform; A9.com, our wholly- owned subsidiary focused on search technology on www.A9.com and other Amazon sites; web services; and digital initiatives.

Round (2004) describes the technology approach as ‘distributed development and deployment’. Pages such as the home page have a number of content ‘pods’ or ‘slots’ which call web services for features. This makes it rela­tively easy to change the content in these pods and even change the location of the pods on-screen. Amazon uses a flowable or fluid page design unlike many sites, which enables it to make the most of real estate on-screen.

Technology also supports more standard e-retail facilities. SEC (2005) states:

We use a set of applications for accepting and vali­dating customer orders, placing and tracking orders with suppliers, managing and assigning inventory to customer orders, and ensuring proper shipment of products to customers. Our transaction-processing systems handle millions of items, a number of different status inquiries, multiple shipping addresses, gift­wrapping requests, and multiple shipment methods. These systems allow the customer to choose whether to receive single or several shipments based on avail­ability and to track the progress of each order. These applications also manage the process of accepting, authorizing, and charging customer credit cards.

9. Data-driven automation

Round (2004) said that ‘data is king at Amazon’. He gave many examples of data-driven automation including customer channel preferences, managing the way content is displayed to different user types such as new releases and top-sellers, merchandizing and recommen­dation (showing related products and promotions) and also advertising through paid search (automatic ad generation and bidding).

The automated search advertising and bidding system for paid search has had a big impact at Amazon. Sponsored links were initially done by humans, but this was unsustainable due to the range of products at Amazon. The automated programme generates keywords, writes ad creative, determines best landing page, manages bids, and measures conversion rates, profit per converted visitor and updates bids. Again the problem of volume is there, Matt Round described how the book How to Make Love like a Porn Star by Jenna Jameson received tens of thousands of clicks from pornography-related searches, but few actually purchased the book. So the update cycle must be quick to avoid large losses.

There is also an automated e-mail measurement and optimization system. The campaign calendar used to be manually managed with relatively weak measurement and it was costly to schedule and use. A new system:

• Automatically optimizes content to improve customer experience
• Avoids sending an e-mail campaign that has low clickthrough or high unsubscribe rate
• Includes inbox management (avoid sending multiple e-mails/per week)
• Has a growing library of automated e-mail programmes covering new releases and recommendations
• But there are challenges if promotions are too successful if inventory isn’t available.

10. Your recommendations

‘Customers Who Bought X …, also bought Y’ is Amazon’s signature feature. Round (2004) describes how Amazon relies on acquiring and then crunching a massive amount of data. Every purchase, every page viewed and every search is recorded. So there are now two new versions, ‘Customers who shopped for X also shopped for Y and ‘Customers who searched for X also bought Y’. They also have a system codenamed ‘Goldbox’ which is a cross-sell and awareness-raising tool. Items are discounted to encourage purchases in new categories!

He also describes the challenge of techniques for sifting patterns from noise (sensitivity filtering) and clothing and toy catalogues change frequently so recommendations become out of date. The main chal­lenges though are the massive data size arising from millions of customers, millions of items and recommen­dations made in real time.

11. Partnership strategy

As Amazon grew, its share price growth enabled part­nership or acquisition with a range of companies in different sectors. Marcus (2004) describes how Amazon partnered with Drugstore.com (pharmacy), Living.com (furniture), Pets.com (pet supplies), Wineshopper.com (wines), HomeGrocer.com (groceries), Sothebys.com (auctions) and Kozmo.com (urban home delivery). In most cases, Amazon purchased an equity stake in these partners, so that it would share in their prosperity. It also charged them fees for placements on the Amazon site to promote and drive traffic to their sites. Similarly, Amazon charged publishers for prime-position to promote books on its site which caused an initial hue- and-cry, but this abated when it was realized that paying for prominent placements was widespread in traditional booksellers and supermarkets. Many of these new online companies failed in 1999 and 2000, but Amazon had covered the potential for growth and was not pulled down by these partners, even though for some such as Pets.com it had an investment of 50%.

Analysts sometimes refer to ‘Amazoning a sector’, meaning that one company becomes dominant in an online sector such as book retail such that it becomes very difficult for others to achieve market share. In ad­dition to developing, communicating and delivering a very strong proposition, Amazon has been able to consolidate its strength in different sectors through its partnership arrangements and through using tech­nology to facilitate product promotion and distribution via these partnerships. The Amazon retail platform enables other retailers to sell products online using the Amazon user interface and infrastructure through their ‘Syndicated Stores’ programme. For example, in the UK, Waterstones (www. waterstones.co.uk) is one of the largest traditional bookstores. It found competition with online so expensive and challenging that eventually it entered a partnership arrangement where Amazon markets and distributes its books online in return for a commission. Similarly, in the US, Borders, a large book retailer, uses the Amazon merchant platform for distrib­uting its products. Toy retailer Toys ‘R Us has a similar arrangement. Such partnerships help Amazon extend its reach into the customer base of other suppliers, and of course, customers who buy in one category such as books can be encouraged to purchase from other areas such as clothing or electronics.

Another form of partnership referred to above is the Amazon Marketplace which enables Amazon customers and other retailers to sell their new and used books and other goods alongside the regular retail listings. A similar partnership approach is the Amazon ‘Merchants@’ programme which enables third-party merchants (typi­cally larger than those who sell via the Amazon Marketplace) to sell their products via Amazon. Amazon earns fees either as fixed fees or as sales commissions per unit. This arrangement can help customers who get a wider choice of products from a range of suppliers with the convenience of purchasing them through a single checkout process.

Finally, Amazon has also facilitated formation of part­nerships with smaller companies through its affiliates programme. Internet legend records that Jeff Bezos, the creator of Amazon, was chatting at a cocktail party to someone who wanted to sell books about divorce via her web site. Subsequently, Amazon.com launched its Associates Program in July 1996 and it is still going strong. Googling www.google.com/search?q=www. amazon.com+-site%3A www.amazon.com for sites that link to the US site shows over 4 million pages, many of which will be affiliates. Amazon does not use an affiliate network which would take commissions from sale, but thanks to the strength of its brand has developed its own affiliate programme. Amazon has created tiered performance-based incentives to encourage affiliates to sell more Amazon products.

12. Marketing communications

In their SEC filings Amazon state that the aims of their communications strategy are (unsurprisingly) to:

1. Increase customer traffic to their web sites
2. Create awareness of their products and services
3. Promote repeat purchases
4. Develop incremental product and service revenue opportunities
5. Strengthen and broaden the Amazon.com brand name.

Amazon also believes that their most effective marketing communications are a consequence of their focus on continuously improving the customer experi­ence. This then creates word-of-mouth promotion which is effective in acquiring new customers and may also encourage repeat customer visits.

As well as this, Marcus (2004) describes how Amazon used the personalization enabled through technology to reach out to a difficult-to-reach market which Bezos originally called ‘the hard middle’. Bezos’s view was that it was easy to reach 10 people (you called them on the phone) or the ten million people who bought the most popular products (you placed a superbowl ad), but more difficult to reach those in between. The search facilities in the search engine and on the Amazon site, together with its product recommendation features meant that Amazon could connect its products with the interests of these people.

Online advertising techniques include paid search marketing, interactive ads on portals, e-mail campaigns and search engine optimization. These are automated as far as possible as described earlier in the case study. As previously mentioned, the affiliate programme is also important in driving visitors to Amazon and Amazon offers a wide range of methods of linking to its site to help improve conversion. For example, affiliates can use straight text links leading direct to a product page and they also offer a range of dynamic banners which feature different content such as books about Internet marketing or a search box.

Amazon also uses cooperative advertising arrange­ments, better known as ‘contra-deals’ with some vendors and other third parties. For example, a print advertise­ment in 2005 for a particular product such as a wireless router with a free wireless laptop card promotion was to feature a specific Amazon URL in the ad. In product fulfil­ment packs, Amazon may include a leaflet for a non­competing online company such as Figleaves.com (lingerie) or Expedia (travel). In return, Amazon leaflets may be included in customer communications from the partner brands.

Amazon’s associates programme directs customers to its web sites by enabling independent web sites to make millions of products available to its audiences with fulfilment performed by Amazon or third parties. It pays commissions to hundreds of thousands of participants in its associates programme when its customer referrals result in product sales.

All figures in millions:

In addition, it offers everyday free shipping options worldwide and recently announced Amazon.com Prime in the US, its first membership programme in which members receive free two-day shipping and discounted overnight shipping. Although marketing expenses do not include the costs of free shipping or promotional offers, it views such offers as effective marketing tools.

Source: Dave Chaffey (2010), E-Business and E-Commerce Management: Strategy, Implementation and Practice, Prentice Hall (4th Edition).