comScore and the issue of misleading internet research in Asia

A recent conversation I had over Twitter with a few switched-on digital types in Asia brought up the topic of comScore and the reliability of its data in Asia Pacific. This is a topic I’ve blogged about before in a post I wrote back in March suggesting that the inaccuracy of the company’s data is losing it trust in Asia.

First, to assess the validity of comScore’s data, a question: how popular are social networks in China?

Answer 1, from comScore: social media take 5.5% of time online, receives visits from 17.8% of those online

Greater China internet usage led by web portals as social networking remains a smaller segment of the market. The report [from which the data is taken] found that Portals commanded the largest share of online minutes, accounting for 24.4 percent of total time in Greater China, followed by the Entertainment, Search/Navigation, Social Networking and Retail site categories. These five categories combined accounted for half of all time spent online in Greater China during April.

***

UPDATE: comScore has clarified that it’s statistics

The chart is showing that only Oak Pacific Interactive Sites (which includes Renren.com) reaches 17.8% of Internet users in Greater China. The reach of the Social Networking category is 50.3% in China, 93.3% in Taiwan and 93.7% in Hong Kong.

For whatever reason this charts were not shown in the press release, which makes things fairly confusing, but the revised data appears to make more sense but – with these new statistics that clearly show social network reaches more than half of Chinese online, I don’t understand how comScore can play down the significance of social media in saying that “Greater China Internet Usage Led by Web Portals as Social Networking Remains Smaller Segment of the Market” which plays down the impact of social media despite its strong reach versus time sent online (versus other media).

Wouldn’t the fact that social media reaches 50% of Chinese online but accounts for 5.5% of time online make for an interesting comparison?

***

Here are comScores figures in data charts below, the first being based on time spent while the second looks at reach:

What is clear from these figures is that social networks are one part of the Chinese online experience, but – with just 17.8% reach of online users, and accounting for just 5.5% of time spent online, comScore concludes that social network is a niche in China.

Answer 2, from Data Center of the Chinese Internet (DCCI) via The Next Web Asia: social media is used by 41% of Chinese online

Chinese now spend 41% of their time online on social networks, which indicates a major shift in how Chinese netizens are no longer purely consumers of web content, but are now more open to communicate, share, and engage online as well.

Reading news, which used to top online activity in 2008 at 49%, has plummeted to a mere 13% in 2010 while time spent on social networking sites saw a boom coming into 2009 and by 2010, ranked first at 41%. Watching online videos also saw an increase from 8% to 23% while search and e-commerce remained pretty much consistent over the years.

The chart below offers a visualisation of the data:

Thoughts

While we cannot be entirely sure what factors account for the vastly different projections of China’s digital landscape, and the role of social media, from these two reports (and it true that I do not have the raw data from DCCI) it is clear that the data from comScore does not include any internet traffic from public networks – such as internet cafes – or mobile devices.

This is crucial, as I pointed out in my last post discussing comScore stats:

Currently, as is clearly marked in its charts and press releases, comScore data internet data in Asia counts private, mainline internet access only, i.e. it does not capture data from anyone using the internet on a mobile phone or at an internet cafe – both of which are hugely used access points for internet users across a number of countries.

[Let us not forget that Asia is a continent where mobile internet usage (compared to fixed-line) exceeds all other global areas]

Take for example Indonesia, a much heralded and developing digital landscape, where according to this infographic from analyst firm Saling Silang 83% of Indonesians use internet cafes with 22% access the web through a smartphone.

Yet none of this traffic registers with comScore’s system.

Though many users access the web through multiple channels at different times, comScore’s data is missing out on recording additional activity from a huge amount of people, particularly when smartphones and internet cafes play a key part enabling internet access as internet penetration is as low as 12%.

Missing out on this data makes it questionable as to whether the data is useful or accurate.

Once again, I wonder how comScore data can be accurate anywhere? Particularly in Asia where mobile and public internet are such crucial access points.

Yet comScore research is commonly the basis for research, blog posts, media articles and countless other projects, like this Asia social media infographic from Edelman below.

Impressive though the infographic is, it is based entirely on comScore’s ‘no public or mobile internet data’ research it is “by no means an accurate reflection of actual usage patterns in the region” as I said when I blogged about it on its release in May.

So precisely how does comScore generate its data, and why doesn’t it include mobile and public internet access?

comScore’s response

I contacted comScore who kindly came back with the following responses.

The comScore panel is based on a 2 million personal panel globally that opt in to letting us measure their online usage from home and work locations. Through this method we are able to accurately measure user visitation, engagement and specifically user demographics.

Due to the multi-visitor usage of shared public computers and the complex ecosystem of mobile, collecting information from these users via panel is more difficult. We will actually be rolling our Total Universe report in Asia shortly. This view will provide a look at combined mobile, tablet and shared use traffic. You can read more on this in this announcement.

comScore data are accurate for the universe that we measure in Asia, which currently is home and work computer usage for those visitors age 15+. The reality of any market research method is that may be limitations to what can be measured, but comScore data is the global standard for digital media measurement and uses the most robust methodology on numerous accounts.

Once a cynic…always a cynic?

I am particularly struck by the final sentence which I whole-heartedly agree with but for me, comScore’s data has quite clear limitations in Asia, to the extent that they can mislead.

Of course, picking on comScore is drawing attention to one of a number of research firms whose analysis has obvious issues. In general the web-measurement science itself is imprecise and based (largely) on estimates and analysing segmented behaviour.

But I with comScore the issues in Asia are more pronounced, significant and obvious.

Though comScore’s upcoming Total Universe report marks a step towards including mobile in web traffic analytics – as its many impressive testimonials emphasis – I can’t help but be almost pre-programmed to anticipate more issues and trouble in validating the accuracy of the reportage.

Perhaps I am overly skeptical, but when messages are a mixed as the example of China’s digital space, how anyone help but look at the small print and scrutinise all research reports?

Unfortunately, too many online, in house or in agencies are too quick to republish a pretty infographic or passing on a favourable headline without providing a thought on its source data, and at least specifying that some of the data’s conclusions may be influenced by its method of collection.

comScore, and the web analytics industry in general, should be working to integrate mobile and public internet data as soon as possible, to help paint a clear picture of what is happening in Asia’s fast-changing, vibrant online community. But I can’t promise that, when they do, I’ll refrain from assess the short-comings of their data collection – they have a responsibility and obligation to be accurate.

Note: comScore did also add that it is looking to release reports for Thailand “soon”, which I look forward to seeing in due course.