Updated with additional details about the experiment via Christian Rudder.
Ever since the big kerfuffle over Facebook's emotion manipulation study -- and the defense that this happens all over the Web all the time -- we've been wondering what other experiments we may have been part of without knowing it. OkCupid came forward Monday with another one: it shot falsehood-tipped arrows through users' hearts as an experiment. The dating site exhumed its three-year dormant “OkTrends” blog which used to share insights into online daters' behavior, but went silent after the company was bought by IAC for $50 million. In a flippant entry that announces his upcoming book on data, OkCupid co-founder Christian Rudder defends Facebook, brags about experiments OkCupid's done in the past, and reveals that at some point the site told people who were poor matches for each other that they were perfect pairs, and vice versa. The site wanted to see if OkCupid's matching algorithm actually predicted whether people would go gaga for each other, or if they were just slaves to an algorithm and would fall in love (or lust) because the data told them they should. In other words, it wanted to know if it had blinded users with data science.
"Guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work," wrote Rudder in the post titled, "We Experiment on Human Beings." Yes, but do we expect sites to lie to us about how they work as a test? This is that nebulous gray zone wherein lies the discomfort about how we're treated as users. OkCupid's privacy policy does warn that it does research to test the effectiveness of its site, but it's a little surprising to see the company brag about deceiving users. Facebook wanted users to have crappy days for science; OkCupid hoped they'd have crappy dates for science. What else are companies doing to us for the sake of experimentation?
OkCupid ran two experiments, involving its matching algorithm, which much like Facebook's Newsfeed algorithm is a bit mysterious to most users, but presumably reveals the degree to which you have things in common with another user, from books to sexual practices. In the first experiment, OkCupid "took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match)." Unsurprisingly, the data-crossed lovers were more likely to email each other when OkCupid told them they were compatible. "But we took the analysis one step deeper," writes Rudder. "We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does. When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other."
OkCupid based that on the fact that those users sent each other multiple messages as opposed to their convo petering off after that first one. That freaked OkCupid out, because it meant the matching algorithm might be BS. So it did the experiment again, reversing it to tell people who were near-perfect for each other (according to the algorithm) that they were not a particularly good match. And that reassured OkCupid, because those people still tended to have long conversations with one another, on average.
A footnote on the blog entry says that after the "experiment was concluded, the users were notified of the correct match percentage," but it doesn't say whether they were told they were part of an experiment or not. I emailed Christian Rudder about it. He says the experiment was "short" and involved fewer than 1,000 users. This is the message they got a few days after the experiment was over.
Dear [nameA]
Best,
Because of a diagnostic test, your match percentage with [nameB] was misstated as [%]. It is actually [%]. We wanted to let you know!OkCupid
"Because 'experiment' has become such an emotionally loaded word, we used the more neutral phrase 'diagnostic test,' which we felt had the same meaning," Rudder said by email.
