Once again, enterprises, vendors, consultants, and industry analysts got together to ponder the state of Web analytics at the fourth Emetrics conference (Santa Barbara, CA, June 2 - 4).  Organized by Jim Sterne, the President of Target Marketing of Santa Barbara and author of Web Metrics, the conference continues to grow in attendance: in 2002, or year one, it was a cozy affair; in 2003 it drew 75 attendees; this year Jim ran two conferences, one in London and one in Santa Barbara, with the Santa Barbara soiree (it was held at the Four Seasons Biltmore) drawing 100 attendees.

The themes this year were several:  measuring and metrics; things to look for when purchasing a solution; usability and conversion issues; the nastiest problems to solve; and stories from the Web analytics front lines, including presentations from Amazon.com, Avaya, Carat Interactive, Hewlett-Packard, InterContinental Hotels, SAP, and SmartDraw.com.  A summary of the conference, and best practices highlighted at the conference, follow.

Jim Sterne began the conference with a lively history of Web analytics, pointing out how data capture methods evolved from transforming server log files, to sniffing packets on the network, to tagging Web pages with Javascript, to online surveys, to monitoring visitor eye movements -- and with many stops in-between.  However, as Jim noted at the end of his talk, "All of this is useless unless it's actionable -- and that means (1) the business goals must be clear, (2) technology, analytics, and the business must be aligned, and (3) the feedback loop must be complete."

Next, Robert Nelson presented on behalf of the CMO [Chief Marketing Officer] Council, a private non-profit organization with 800 members and representing more than $40 billion in marketing spending.  He specifically discussed the Council's Marketing Performance Measurement (MPM) Initiative, a study of how the industry is measuring marketing effectiveness with hard numbers, rather than gut feel.  Polling practitioners via both telephone and online surveys, the Council's findings were that although  over 90% of companies rate MPM a High or Moderate priority, few companies have a formal MPM system, with almost half using an informal system.  The result is that less than a quarter of respondents are Satisfied or Very Satisfied with their company's ability to measure marketing performance.  Web analytics served as a bright spot in the survey, as it was one of three areas that was easiest to measure (the other two being direct mail/e-mail campaigns and telemarketing/contact management).

Troy Skabelund of the Walt Disney Internet Group then reported on the progress of the IAB (Interactive Advertising Bureau) Measurement Task Force.  The Task Force is working on a global approach to counting and measuring on online advertising impression by reducing counting discrepancies between ad publishers and third-party ad servers.  These newer guidelines will supercede the current January 2002 guidelines.  One of the ways the Task Force will make sure the new guidelines are adopted is by using the stick of demanding that agencies accept only certified impression counts.

Things To Look For

Terry Lund of Terry Lund Consulting, formerly responsible for managing Kodak.com, discussed how to evaluate Web analytics vendors.  He outlined five steps for vendor selection:

1. Involve the right people -- E.g., business, marketing, product, and IT managers; customer support; legal; and business research
2. Understand Web analytics basics -- Pages, visits, sessions, referrers, etc.
3. Set business goals and budgets -- Define the purpose of the Web site (e.g., e-Commerce, lead generation, product information) as well as some total cost of ownership (TCO) calculations
4. Select vendors -- Taking into account price, implementation details, Web site traffic, and the size/complexity of the Web site
5. Final selection and negotiation -- Narrow the list down to 2 or 3 solutions, and then run a pilot using your data

Eric Peterson of Jupiter Research, under the title of "Making Analytics Actionable," then discussed the use of key performance indicators (KPIs) in Web analytics.  A KPI, by distilling behavior into a single metric -- such as Percent Returning Visitors or Percent of Visits Under 90 Seconds --  makes it easy for business managers not intimately involved with the Web site to understand how the Web site is helping or hindering the business.  By distributing such "nuggets" to managers throughout the company, Web site monitoring breaks out of being the province of IT and Marketing, and instead gets woven into the business.

Jared Spool, Founding Principal of User Interface Engineering, stirred things up a bit by announcing in the middle of his presentation entitled, "Land of the Lost Revenue," that Web sites would best improve their conversion rates if they "Stopped Marketing Immediately!"  Distilled, his point was that (1) all revenue comes from a very small percentage of visitors, (2) we focus practically all our resources on people who will never purchase, and (3) what would happen if we only focused on the people who purchase?  Therefore, the best strategy is to (1) focus on the few who contribute the most, (2) aim to improve their experience dramatically, and (3) everyone else's experience automatically improves.

Jim Novo of The Drilling Down Project, former VP of Programming and Marketing for Home Shopping Network, talked about segmenting customers into three groups:

1. Those you want to keep -- High Potential Value/High Current Value)
2. Those you want to grow -- Low Potential Value/High Current Value, High Potential Value/Low Current Value)
3. Those you want to jettison -- Low Potential Value/Low Current Value).

Although it's easy to calculate Current Value, Potential Value is more problematic.  However, a proxy for Potential Value is Recency -- happy customers in a buying mood typically have a more "active conversation" with a company than lukewarm or dissatisfied customers.  Accordingly, by plotting Recency (e.g., last visit to the Web site, last purchase on the Web site), companies can rank their customers by Potential Value, thereby understanding which customers to keep and which ones to woo just as they are about to defect.

The Top Ten Nastiest Problems to Solve

With representation from enterprises, vendors, consulting groups, and analyst firms, Jim put the attendees to work by asking them what they thought were the Top Ten Nastiest Web Analytics Problems.  The audience quickly blew past the 10 limitation and came up with almost 40.  However, they clustered into a smaller number of issues:

1. Site Complexity -- Site and product complexity; site segmentation; usability and accessibility analysis; privacy; benchmarking
2. Technology -- People expect expect answers but just get a tool; need multiple tools; difficult implementation and setup; complexity of delivery of analytics reports; non-traditional browsers
3. Resources -- Not enough humans, etc./resources; have tools but no analyst; attributing value to the Web (allocating budget); cost of visitor support
4. Politics -- Inaction; internal politics; lack of executive education and sponsorship; no Chief Web Officer (no coordination); from reactive to proactive; dependence on IT to get things done; "more is better"
5. Metrics -- Identifying TCO and getting surprised; using metrics for strategy optimization; clear links between strategy, objectives, and tactics to metrics; identifying KPIs; KPIs are hard to record; getting rid of legacy metrics; measuring value of content; how to score leads
6. Data Integrity and Quality -- data collection integrity and consistency; inconsistent tools and processes and numbers; cross domain analysis (single view of the customer); onsite to offline data and lead gen and back; timely access to data; relevant data to the right people; expectations of accuracy; data overload; integrating data (e.g., clickstream and surveys); people are deleting or not accepting cookies

Stories from the Front Lines

Throughout the conference, enterprises talked about their Web analytics strategies and processes:

Amazon.com, one of the few Internet success stories, is now a Fortune 500 company and has over 41 million active customer accounts.  Ever mindful of the corporate goals of low prices, convenience, and a wide selection of merchandise, the company continually monitors and improves its six Web sites.  For example, the company has set per minute site revenue goals, and rings automatically rings pagers to pull personnel out of meetings if those bounds are violated.  The Data Mining and Personalization Group helps the company decide on site changes largely through the use of A/B tests.  By randomly showing alternative treatments (e.g., a new home page design, different feature locations) to visitors, the group is able to discern whether the change is statistically for the better.  It does not require a large percentage improvement to have a decisive bottom line impact:  for example, if Amazon.com is able to improve shopping cart conversion by 1%, it gains additional revenues of $50 million.  On any given day, Amazon.com will be running four or five tests simultaneously.

The Web analytics group at Avaya, a provider of communications systems with offices in 50 countries and fielding 800+ Web sites, described how it consciously morphed itself from a Web analytics group to a Web optimization group.  By shifting the emphasis from "analyze" to "optimize," the group made its work more action-oriented and was better able to explain its value to the corporation at large, thereby gaining senior management buy-in and increased usage of Web metrics.

Carat Interactive, a large interactive advertising agency, described several client engagements where the following were key:  (1) clarifying the appropriate metrics to develop and track, (2) segmenting customers as a way to optimize the site for high-spending customers, and (3) taking a phased approach.

Hewlett-Packard, maintaining 1,500 Web sites and receiving 30 million unique visitors a month, discussed its Web analytics philosophy and infrastructure.  Confronted with legacy systems and processes (the result of DEC being bought by Compaq and then Compaq being bought by HP), the Web analytics group has concentrated on deploying common metrics and analytics tools worldwide.  Clickstream, survey, order, 800 call data, media, and firmographic data is all consolidated in a data warehouse, which then serves as a data source for dashboards, standardized reports, ad hoc reports, and predictive modeling for the business.

InterContinental Hotels Group offers 530,000 hotel rooms through its Crowne Plaza, Holiday Inn, and InterContinental brands.  The company, generating 10% of its room revenue from the Internet via 25,000 online bookings a day, utilizes a central analytics group to help the business understand Web site visitor behavior.  This 23-person group leverages a wide variety of tools -- clickstream analysis, online surveys, usability testing, and competitive intelligence -- to ascertain how to best optimize the site to meet visitors' goals.  One example of a customer-driven change was InterContinental's realization that visitors wanted to be able to see hotel rates at a glance when shown a list of hotels within a specific geographic area, rather than have to click on each individual hotel entry to get pricing.

SAP, the third largest independent software supplier with revenues of 7 billion Euros, manages 55 country sites and maintains 75,000+ Web pages.  With business goals such as branding, education, lead generation, and sales improvement, SAP asks questions such as, "What content drives or influences conversion?", "How do customer segments differ?", "What is the health of our lead qualification process?", and "What attributes describe our best customers?"  In the course of answering these questions, the company discovered that no one tool can do it all, using Clickstream Technologies for clickstream data collection and analysis, SAP Business Warehouse for data management, mySAP CRM for lead management, VBIS for visual interpretation of traffic, and WebEx as a way to share best practices and intelligence.

SmartDraw.com, an online provider of business graphics software, uses four systems to help it understand Web visitor behavior:  (1) log file analysis to monitor software downloads, (2) an in-house developed system for tracking marketing expenses and experiments, (3) Urchin for general traffic metrics, and (4) ClickTracks for page navigation analysis.

Ballardvale Research Conclusions:  Best Practices

Although the presentations by enterprises such as Amazon.com and Hewlett-Packard were heartening, based on dinner conversations at the conference it's clear that many companies are still struggling with how to do Web analytics well.  Emerging best practices highlighted at the conference include:

• A Chief Web Officer -- Whatever the title, a single person responsible for a company's online presence.  This "buck stops here" office helps crystallize a company's online vision as well as resolve disputes such as which department gets the most real estate on the home page this week.
• Clear goals for both the business and the Web site -- Without clear directives, optimizing a Web site is futile -- the business endorses conflicting business goals, the Web analytics group cannot identify the appropriate KPIs, and confusion reigns.  If an enterprise can agree on the top priorities for both the business and the metrics, it -- by default -- agrees on what to ignore.  And given the complexity of today's Web sites and metrics, that is a huge victory.
• The use of multiple technologies -- Companies successful at Web analytics no longer solely depend on a clickstream analysis solution for all of their insight.  Although such a system is initially helpful at identifying major Web site gaffes, over time as the required changes become more subtle, the need for online surveys, user interface studies, and other tools grows.  Cross-checking and double-checking from multiple points-of-view becomes the rule.
• Iterative improvement -- Finally, rather than going for the "big bang" -- a wholesale rewrite of the Web site, for example -- cutting-edge enterprises improve their Web site in an iterative fashion.  Only then can they keep the variables to a minimum and prove that this specific site change improves conversion by 2%, for example.