By Mark Esposito and Terence Tse
In the past two decades, the number of performance indices has proliferated substantially. There are good reasons why performance indices have blossomed: they catch attention and they make information easily understandable. In this article, Terence Tse and Mark Esposito discuss the abilities of indices and glean insights into how they work by highlighting their strengths and weaknesses.
It is no secret that statistics have are often have been manipulated to alter perceptions. Just consider the following example: In the late 1970s, the Mexico City government increased the capacity of its four-lane motorway by re-painting lines to make it six lanes wide, which, in effect, gave it a 50% rise in capacity. Unfortunately, this also resulted in an increase in fatal accidents. So, after a year it was changed back to a four-lane road, representing a 33% capacity reduction. Yet, the government, in an attempt to support its claims of social progress, subtracted that 33% drop from the 50% increase to claim a net increase in capacity of 17%.1
[ms-protect-content id=”544″]Through such manipulation, not surprisingly, statistics has become perhaps one of the most unloved subjects. Popularised by Mark Twain, the phrase “There are three kinds of lies: lies, damned lies, and statistics” describes the persuasive power of numbers, particularly when used deliberately to bolster weak arguments.2 Others in history often show no more leniency. For instance, in his book “How to lie with statistics”, a guide that shows how to use figures to say whatever you want them to, Daniel Huff lamented “Crooks already know these tricks. Honest men must learn them in self defence”.3
Numerical indicators can often speak louder than qualitative evidence. In our view, the worst culprit of them all is the calculation of Gross Domestic Products (GDP). Even though different governments, research institutes as well as economists frequently estimate GDP differently, its fundamental compositions always include the following:
GDP = Consumption + Investment + Government spending + (Export – Import)
Out of this formula, it is, for the first time in history, possible to establish how much a nation produces. And the equation is only matched by its mathematical elegance. Yet, when Simon Kuznets first proposed this method to measure a country’s income, he expressively pointed out that many aspects of our social and economic lives are difficult, if not impossible, to quantify. He stated that,
“The valuable capacity of the human mind to simplify a complex situation in a compact characterization becomes dangerous when not controlled in terms of definitely stated criteria. With quantitative measurements especially, the definiteness of the result suggests, often misleadingly, a precision and simplicity in the outlines of the object measured. Measurements of national income are subject to this type of illusion and resulting abuse, especially since they deal with matters that are the center of conflict of opposing social groups where the effectiveness of an argument is often contingent upon oversimplification”.4
Despite such warning, over time, we have focused solely on the part that are quantifiable and duly neglected those that are not. It has become the sine qua non for economic progress. The grand result of such bamboozle with numbers is now felt: in the pursuit of GDP growth, the Chinese government has for years ignored pollution, from which the environmental damage can now be hardly reversed. Russia’s stellar GDP growth in the past decade did not reveal the problem of solely depending on oil and gas export,5 the dire consequence of which has led to the country’s current economic plight.
Indices: Old Tricks in New Packaging
Befuddling with numbers has lately – and unfortunately – gone on a new height: in the form of indices and rankings, which weaves data on lots of measures into a single easy-to-understand ranking.6
In the past two decades, the number of performance indices has proliferated substantially (see Figure 1 on the next page). There are good reasons why performance indices have blossomed: they catch attention and they make information easily understandable. But perhaps the most potent of them all: backed by the appearance of scientific expertise and rigorous methodology, an index serves as a named, rank-ordered representation of a past or projected performance that uses numerical data to simplify otherwise very hard-to-describe social phenomenon. This, in turn, allows for direct comparison between countries or persons and to evaluate their performance by reference to one or more standards.7
Indices – of which there is now seemingly one for everything – can therefore come in as a handy tool to exert social pressure, especially on governments and politicians to take (corrective) actions.
In this article, we first discuss their characteristics and abilities to apply pressure as well as shortcomings. Subsequently, we turn to look at four examples of indices to glean insights into how indices work by highlighting their strengths and weaknesses. This article ends with some concluding remarks as well as suggestions.
Indices I Love You
Indices – also called rankings and composites – compile numerically-rendered data and repackage statistical information. Given the numerical nature of indices, they have the characteristics that follow:
Simplification of complex reality. A major benefit of rankings is that they are often numerical representations of complex phenomena. The result is that they are capable of presenting a picture as well as telling a story in numbers and orders of seemingly unquantifiable human and social activities that range from state qualities (e.g. “transparency”) to state policies (e.g. “press freedom”) to prevalent social practices (e.g. “corruption”).8
Admittedly, boiling down a great deal of contextual information to some single numbers or “scores” as well as stripping down meaning from the observed phenomenon may sound like a self-defeating exercise. But indices do offer something that are easy-to-understand, convenient and easy-to-use. Hence, they provide an instant idea of the success of countries and policies in the areas of concern. Moreover, they repackage statistical information but do not necessarily engage in statistics, their users need nothing more than basic arithmetic to make sense of results. Viewing from this perspective, rankings are an excellent tool that everybody can understand and that everybody can use for a wide range of purposes.
It is also necessary to note that such simplification of very complex reality is often reinforced by the name of the indices themselves: the name is often designed in a way to stress what the index claims to measure or rank. Calling an indicator a measure of “transparency” or “human development” purport that such phenomena are directly observable and the numerical representation that measures them are in existent. An indicator may even create the phenomenon it claims to measure, as IQ tests came to define what intelligence is.9 Indeed, names are sometimes deliberately designed to create controversy, thereby used for grabbing more attention, faster.
Conciseness and comparability. The beauty of rending vague issues into precise scores, in addition to making indices easy reading, is that the economic, social and political aspects of our lives can now be rated: indices have become “psychological rules of thumb” that are clear and simple symbols, which in turn help the examiners, the examinees and third parties deal with the complex reality.10 In addition to simplicity, concise indicators can add processing speed as they can be scanned in moments, while reading the underlying reports on which they are based would involve substantially more time.11
Another benefit of index scores in indices is that they can be placed in orders, making it possible for countries to be ranked and explicit comparison among units to be fostered. Furthermore, scores can be averaged, thereby establishing “norms” and “standards”, against which units can be compared.12 With the scores calculated and complied periodically, time-series analysis that shows how countries perform on a certain issue over time can also be conducted.
Subject matter expertise (to the index builder). While some indices are constructed based on rigorous methodologies and robust data, others are founded on a little more than “guesstimation”. Even so, these latter indices can deploy various techniques to generate an appearance of scientific objectivity, so much that it got The Economist to publish an article called “How to lie with statistics”.13 Quantification and numerical figures certainly help: by appearing precise and certain, the rankings seem so persuasive that they cannot be ignored, which in turn reinforces the expertise.14
This may explain why performance index builders often rely on respect and credibility that are afforded to them, thereby boosting their social influence. For instance, an index developed in Switzerland is likely to carry a great deal of weight than a similar index that is created in Russia. This is not necessarily because the Swiss have more coercive power; instead, they could have a much higher credibility for the purposes of such a rating.15
You’ve Got the Power
Given the ease of processing and understanding, as well as the perception of expertise, most performance indices, especially the prominent ones, can assert influence on the examinees through systematic monitoring: continuously observing and checking the progress or quality of a policy, practice or condition. The result is that they can mobilise and inform countries, berate specific policy makers, and facilitate transnational pressure. Explicit comparisons, once promulgated, could be difficult to dislodge from public discourse.16 Rankings can therefore be a particularly potent lever for creating social pressure.
Governments that find themselves low in ranking often feel the heat. Media coverage of trafficking grew dramatically in countries covered by the index. Moreover, there is a fear of international opprobrium. A 2010 press release from Pakistan’s Interior Ministry described “significant efforts” to get off the Trafficking in Persons list, effectively vowing to “improve the stature of Pakistan before the world”. Similarly, Germany’s underperformance in OECD’s Programme for International Student Assessment (PISA), which compares the academic performance of pupils of 15 years of age in dozens of countries, pushed the country to reform school as well as the creation of a €4 billion education support programme.17
What We Should be Aware of with Indices
In 2008, we heard the following statements18:
• The director-general of the Malaysian Industrial Development Authority insists that “Malaysia aims to move from the 24th to a top 10 position in Doing Business index”;
• Akylbek Zhaparov, Minister of the Economic Development in Kyrgyzstan, expressed the wish that the country “shall rank among the top twenty countries in the Doing Business rating in three years”, from its current position of 94th;
• Macedonia placed a one-page advert in The Economist’s annual forecasting report “The World in 2008” to announce that it had improved their position on the Doing Business Index.
While these aspirations are undoubtedly commendable, raising awareness does not always lead to actions. In the 2015 ranking, Malaysia and Kyrgyzstan holds the 18th and 102th position, respectively.19 Granted, there are plenty of reasons that prohibit countries from improving their ranking. Yet, even for those who are able to go up on the list, increasing in ranking does not necessarily mean improving the real performance. Countries can choose to focus their efforts on those criteria with the goals of boosting their ranks.20
The “attention-grabbing” power of indices may also have been over-estimated. One of the major weaknesses, in particular with those rankings that include a large number of countries, is that media is likely to only be interested in those that are at the top and bottom – how those countries in-between have performed are often overlooked. The result is that indices may not have as much social power as many people believe.
Another problem is that the ordinal structure of ranking could lead to over-reaction. Each year when a ranking is published, much public attention focuses on the relative rank of a country to its rank in prior years and to the rank position of competing countries. For instance, when Pakistan (ranked 138th on the Human Development Index) bypassed India (ranked 139th) in 1998, The Tribune wrote: “Pak beat India, both lose!” In 2001, when Canada lost the number one position, it prompted The National Post to say: “We’re not No. 1! Canada drops in UN rankings… Prime Minister Jean Chretien often refers to the report in public statements and speeches”. A past study shows that they do not necessarily mean that countries have performed less well in absolute terms. Yet, the order of things has created the wrong perception.21 This shows that statements based on ordinal comparisons have to be interpreted with great care.
Conclusions and Implications
Indices, especially the most prominent ones, are often tools of social power that can affect the behaviour of policy makers and governments. This is not the least because they can turn complex matters into concise indicators that can be scanned in a matter of seconds. Such quantification of human and social issues allow for immediate and direct comparisons and benchmarking.
But as much as indices could tell a story, these stories are often told in a distorted way. Some indices are created just to put forward a subjectively and deliberately chosen message. Others are often made up with a limited set of perspectives. To Michael Porter and his team, for instance, there are other aspects in the society that matter at least as much as national competitiveness. They include access to opportunities, suicide, property rights, school attendance, attitudes toward immigrants and minorities, opportunity for women, religious freedom, nutrition and electrification, to name a few. The result is the newly launched Social Progress Index.22
We believe that this is a step forward in the right direction, as existing indices are often overwhelmingly one-dimensional. In a world where most indices delve into more narrow spectra of our lives, it makes more sense for us to widen our scope. Surely, access to data is often a constraint. Yet, we see that supra-national organisations such as the United Nations hold a great number of very rich datasets. As the different agencies have opened up their databases, we can see that there is much to be gained for everyone.
Yet here we are not suggesting any “super index”; we simply subscribe to the view that if we can combine more of the different and diverse facets of our lives into single composites, governments and policy makers can be better informed. With the social power that many current indices have potently projected, it is possible to create a better future in this way for all of us. After all, like in any relationship, we should not simply be asking “what’s wrong with you”. A more constructive and beneficial question is instead “how we can make everything around us better”. Our passion for indices should not be different.
About the Authors
Dr. Mark Esposito is a Professor of Business and Economics, teaching at Grenoble Ecole de Management, Harvard University Extension and IE Business School. He serves as Institutes Council Co-Leader, at the Microeconomics of Competitiveness program (MOC) at the Institute of Strategy and Competitiveness, at Harvard Business School. Mark consults in the area of corporate sustainability, complexity and competitiveness worldwide, including advising to the United Nations Global Compact, national banks and NATO through various Executive Development Programs. From 2013-14, Mark advised the President of the European Parliament, Martin Schulz, in the analysis of the EU systemic crisis and worked as cross theme contributors for the World Economic Forum reports on Innovation Driven Entrepreneurship. Mark holds a PhD in Business and Economics from the International School of Management, in a joint program with St. John’s University in New York. He tweets as @Exp_Mark
Dr. Terence Tse is an Associate Professor of Finance at ESCP Europe Business School. He is also Head of Competitiveness Studies at i7 Institute for Innovation and Competitiveness, an academic think-tank based in Paris and London. In addition to working in governmental advisory capacity, Terence writes extensively and appears on television programmes in China, France, Greece and Japan, discussing the subjects of competitiveness and economic affairs. Before joining academia, he worked in mergers and acquisitions at Schroders, Citibank and Lazard Brothers in Montréal and New York. Terence also worked as a business consultant both independently and at Ernst & Young focusing on UK financial services. He obtained his PhD from Cambridge in Management Studies. He tweets as @Terencecmtse.
References
1. http://www.economist.com/node/160593
2. The origin of the quote remains to this day uncertain.
See http://www.york.ac.uk/depts/maths/histstat/lies.htm
3. Huff, Daniel (1991) How to Lie with Statistics, Penguin Business
4. Kuznets, Simon (1934) National Income 1929–1932. A report to the U.S. Senate, 73rd Congress, 2nd Session, Washington, DC. US Government Printing Office, 5-6
5. http://www.worldfinancialreview.com/?p=2881
6. http://www.economist.com/news/leaders/21631025-
learn-ruses-international-country-rankings-how-lie-indices#mm
7. Davis, Kevin E., Kingsbury, Benedict and Merry, Sally Engle (2012) “Indicators as a Technology of Global Governance”, Law & Society Review, 46 (1), 71-104
8. Kelly, Judith G. and Simmons, Beth A. (2014) “Politics by Number: Indicators as Social Pressure in International Relations”, American Journal of Political Science (forthcoming)
9. Davis, Kevin E., Kingsbury, Benedict and Merry, Sally Engle (2012) “Indicators as a Technology of Global Governance”, Law & Society Review, 46 (1), 71-104
10. Sinclair, Timothy (2005) The New Masters of Capital: American Bond Rating Agencies and the Politics of Creditworthiness, Cornell University Press
11. Löwenheim, Oded (2008) “Examining the State: A Foucauldian Perspective on International ‘Governance Indicators’, Third World Quarterly, 29(2) 255-274
12. Weisband, Edward (2000) “Discursive Multilateralism: Global Benchmarks, Shame, and Learning in the ILO Labor Standards Monitoring Regime”, International Studies Quarterly, 44, 643-666
13. http://www.economist.com/news/leaders/21631025-learn-ruses-international-country-rankings-how-lie-indices#mm
14. Høylanda, Bjørn, Moeneb, Karl and Willumsenb, Fredrik (2012) “The tyranny of international index rankings”, Journal of Development Economics, 97(1), 1-14
15. Kelly, Judith G. and Simmons, Beth A. (2014) “Politics by Number: Indicators as Social Pressure in International Relations”, American Journal of Political Science (forthcoming)
16. Andreas, Peter and Greenhill, Kelly M. (2010) Sex, Drugs, and Body Counts: The Politics of Number in Global Crime and Conflict, Cornell University Press
17. http://www.economist.com/news/international/
21631039-international-comparisons-are-popular-influentialand-sometimes-flawed-ranking-rankings
18. These examples are drawn from Høylanda, Bjørn, Moeneb, Karl and Willumsenb, Fredrik (2012) “The tyranny of international index rankings”, Journal of Development Economics, 97(1), 1-14
World Bank (2015) Doing Business
19. Høylanda, Bjørn, Moeneb, Karl and Willumsenb, Fredrik (2012) “The tyranny of international index rankings”, Journal of Development Economics, 97(1), 1-14
20. Wolff, Hendrik, Chong, Howard and Auffhammer, Maximilian (2010) Classification, Detection and Consequences of Data Error: Evidence from the Human Development Index, NBER, Working Paper 16572
21. http://www.socialprogressimperative.org/
[/ms-protect-content]
