Who’s in control of your reputation?
It might seem like a rhetorical or even ridiculous question, but it’s at the center of an emerging trend which may soon shape our fundamental social and economic institutions. As we voluntarily and involuntarily share enormous amounts of personal data, companies, governments, and even other individuals are increasingly interested in using this information to produce a pixel-perfect picture of who we are.
In some sense this isn’t new. Credit scores have been used for decades to help institutions manage risk. In turn, credit scores have dictated who can own a home as well as a myriad of other financial opportunities and limits. But what happens when this score extends beyond the financial? What happens to a society when opaque algorithms weigh vast troves of non-financial personal data?
China is about to find out.
Meet China’s “Social Credit” System
BBC News recently reported on China’s ongoing efforts to develop an enormous “social credit system.” Designed to “rate each citizen’s trustworthiness,” the government has enlisted eight companies to pilot the program. The highest profile driver of the program is Sesame Credit, a branch of China’s immense online shopping platform, Alibaba.
In this program, people are literally buying into Big Brother. Purchase history contributes to algorithmic value judgments about a person’s character. Because Alibaba also administers a payment platform, it has incredible visibility into a wide range of services beyond its own marketplace, including China’s equivalent of Uber or Lyft, Didi Kuaidi.
Sesame Credit’s scoring system provides access to privileges which may not otherwise be available to the scoreless, including preferred reservations for cars and hotels as well as freedom from onerous cash deposits.
The company is working hard to promote transparency about scores. Unlike the U.S., where credit scores are privately shared between individuals and the institutions which require them, Sesame encourages people to share scores, even placing them publicly on dating profiles.
BBC reports that China hopes to have all of its citizens in the system within five years. There are no plans to allow people to opt-out.
Could it Happen in the U.S.?
In a sense, it already has. Companies are eager to expand systems designed to provide shortcuts into decision making about who we are and how much we should be valued.
Take, for example, Klout. Klout is a company dedicated to measuring an individual’s ability to influence or “drive action” online. At the core of Klout is the “Klout score.” According to the company, “When you share something on social media or in real life and people respond, that’s influence. The more influential you are, the higher your Klout Score.”
Klout claims to “use more than 400 signals from eight different networks to update your Klout Score every day.” It seeks for its users to feed all of their social networks and other information sources (such as Wikipedia) into their system in order to provide a score of 1 to 100, indicating how influential a particular person is online. Klout may be voluntary, but the idea behind it illuminates the ways in which our social sharing and online behavior could be compiled without our consent.
While it may not be surprising that governments use this sort of intrusive tracking under the mantle of security, you might be surprised that insurance companies are using social media to underwrite clients, investigate claims, and manage premiums.
It’s becoming increasingly clear that sharing information without our explicit permission represents a real threat to our ability to maintain our freedom and even safety. The rise of doxing demonstrates how malicious use of our personal data can be weaponized against us. According to Wikipedia, doxing “is the Internet-based practice of researching and broadcasting private or identifiable information (especially personally identifiable information) about an individual or organization” for a variety of ends, including “extortion, coercion, harassment, online shaming, and vigilante justice.”
What sort of future might these systems portend?
Visualizing Our Scored Future in Black Mirror
For a bracing look at how our future could be shaped by a composite reputation score, look no further than the dystopian television series Black Mirror. The series addresses this very phenomenon in Season 3’s episode “Nosedive”:
“In a society where one’s social rating with others vastly influences their lives, a woman tries to improve her own rating to afford an ideal apartment by giving an appealing speech at her best friend’s wedding.”
You can catch up on The Atlantic’s recap of the episode here (but be warned: there are spoilers ahead!). As Sophie Gilbert writes,
“As with so many Black Mirror episodes, the horror lies in imagining all too clearly how such a situation might feel.”
The show is an excellent example of how art gives us the ability to examine the human implications of privacy and information security.
Sharing vs. Autonomy
So what’s at stake when we consider the pros and cons of the information we generate being used in systems like this? Knowledge is, indeed power. The more a system knows, the more the system exercises control over what we can and can’t access in society. In a sense, our autonomy depends on our ability to exercise our privacy.
We cannot always foresee how our personal information will be used. This is especially true when the way algorithms function are not transparent, and the institutions which scrape our data may make interpretive errors or incorrect assumptions. What recourse will we have?
We must be mindful of what we share. This means pausing before we post and remaining vigilant about the privacy policies of products and services we choose to use. More than just our reputation depends upon it.