Intelligence Squared debates: voting data

Intelligence Squared US is a series of public debates on a wide variety of important, contentious topics.  A motion is proposed and two experts who support the motion debate two experts who oppose the motion. The debates take place in front of a live audience and are later broadcast on NPR, released as podcasts, and shared through other means.

Examples of past motions include: “Millennials Don’t Stand a Chance,” “Snowden Was Justified,” The Constitutional Right to Bear Arms has Outlived Its Usefulness,” “Affirmative Action on Campus Does More Harm Than Good,” etc.

Before the debate, the live audience is privately polled on their current support for the motion. After the debate, the audience is again polled. During both sets of votes, audience members can select “For,” “Against,” or “Undecided.” The side that has increased their share of the votes by the highest amount is declared the “winner” of the debate.

Consider, for example, this fictional debate:

BEFORE AFTER
FOR 31 42
AGAINST 44 49
UNDECIDED 25 9

In this debate, the side arguing FOR the motion would be declared the winner. The FOR side increased its votes by 9 percentage points (31 to 42) while the AGAINST side only increased its votes by 5 percentage points (from 44 to 49). Note that while FOR won the debate, they actually had a smaller percentage of total votes in the AFTER round of voting–it is the increase in votes that matters, not the total number.

Who won?

It is interesting to look at the data* from past debates to see how the voting changed during the course of the debate. For example, sometimes there is a very clear winner: in “More Clicks, Fewer Bricks: The Lecture Hall is Obsolete” (4-2-14), the FOR side increased by 26 points while the AGAINST side decreased by 12 points.

Sometimes, there is no clear winner: in the “Cutting the Pentagon’s Budget is a Gift to Our Enemies” debate (6-19-13), the FOR side increased by 7 percentage points and the AGAINST side increased by 8 points. However, it is important to note that the final tallies were actually 29% FOR and 65% AGAINST–quite far apart.

The winner of the BEFORE or AFTER vote could be very different than the winner of the debate (the side that increased more). Across all n=92 debates, the overall winner has only matched the winner of the initial vote 44.6% of the time. The overall winner has matched the winner of the final vote 73.9% of the time.

It actually makes sense that losing the initial vote may be an advantage: since the winner is determined by who increased their votes by the highest amount, starting with fewer votes than the other side provides more potential converts. If you start with 20% of the vote, that is 80% of the audience you can pull your increase from. If the other team started with 40% of the vote (thereby winning the first vote), they only have the remaining 60% of the audience to pull to their side. If you start with a smaller vote count, this also gives you fewer of your own voters to potentially lose in the final vote!

As you may have noticed, the tallies for FOR and AGAINST can both go up from the beginning to the end of the debate–and often do–due to the presence of UNDECIDED voters. A quick glance at the results shows that the number of undecided voters tends to go down substantially from before to after.

No more UNDECIDEDs?

The data certainly supports a major decrease in UNDECIDED voters. In every single one of the n=92 debates, the number of UNDECIDED voters does indeed decrease from the initial vote to the final vote. Across all of the initial votes, the median percentage of UNDECIDED voters is 30, while across all of the final votes, the median percentage of UNDECIDED voters has dropped to 8.

This is a noteworthy phenomenon. When spending an evening hearing thoughtful and well-articulated arguments for both sides of a motion, I can imagine two plausible scenarios:

  1. Audience members conclude that there are actually compelling and mutually valid reasons to favor BOTH sides of the argument and they gain a greater appreciation for the complexity and nuance of the issue and, unable to settle on a side, they vote UNDECIDED.
  2. Upon hearing good arguments on both sides of an issue, audience members weigh both sides and conclude that the slight (or not so slight) leaning they now have to one side of the issue must be sufficient to declare a side which they then vote for…. or some single argument happens to particularly resonate with them for whatever reason and they pick the side associated with that argument regardless of the rest of the debate.

I’ll set aside the philosophical question of which scenario is preferable in which situations, set aside the psychological question of the conditions under which people are more likely to choose one or the other, and set aside the political/sociological question of how to nudge people towards one scenario or the other and instead simply report what people actually did in this Intelligence Squared context.

Where did all of the UNDECIDEDs go? More data!

It certainly clear from the large decrease in UNDECIDED voters overall (30 percentage points initially to 8 at the end) that many people fall into scenario 2: they pick a side.

However, it is not yet clear what this actually looks like in practice…. Maybe most of the undecided people tend to flock to one side (presumably a “better” side that has won the debate). Alternatively, maybe the UNDECIDED people were simply uniformed and, upon learning more about the issue during the debate, will pick sides in the same proportion as the rest of the audience who had already previous picked a side.

Also, while it is clear that the total number of undecided voters is much lower, indicating that lots of the previously undecided voters picked a side by the end, what is not yet clear is if some people who previously had an opinion became undecided during the course of the debate and, if so, how many.

Finally, it seems reasonable that, when spending an evening hearing thoughtful and well-respected experts make carefully reasoned arguments on both sides of an issue,  at least some audience members would realize that they are, in fact, undecided about where they stand on that issue (or even on the wrong side entirely).  But is that actually the case?

Luckily, in addition to publishing the starting and ending “scores” for debates, Intelligence Squared has also published the breakdown of votes AFTER the debate grouped by audience members’ votes BEFORE the debate for n=26 recent debates. This allows us to see, for example, how many FOR voters stayed at FOR, switched to AGAINST, or switched to UNDECIDED.

For “More Clicks, Fewer Bricks: The Lecture Hall is Obsolete” (4-2-14), here is the data:

AFTER the debate
For Against Undecided
BEFORE the debate
For 12 7 1
Against 14 39 6
Undecided 13 5 3

The 14 above can be interpreted as “14 percent of the audience voted AGAINST the motion BEFORE the debate, but FOR the motion AFTER the debate.” The 5 can be interpreted as “5 percent of the audience was UNDECIDED on the motion BEFORE the debate, but voted AGAINST the motion AFTER the debate.”

Purple represents voters whose initial and final votes were the same (they didn’t change their minds at all). Orange represents voters who had an opinion to start with, but ended with the opposite opinion. Yellow represents voters who were initially UNDECIDED but ended up picking a side. Green represents voters who had an opinion to start with, but whose final vote was UNDECIDED.

Intuitively, you can also think about this in terms of rows or columns. The UNDECIDED row adds to 31: this means that in the initial vote, 31 percent of the audience was UNDECIDED. Among these 31 percent of the audience, in the final vote 13 of these percentage points voted FOR, 5 of these percentage points voted AGAINST and 3 of these percentage points voted UNDECIDED. Similarly, looking at the UNDECIDED column, for example, 10 percent of the audience was UNDECIDED in the AFTER vote, and 1, 6, and 3 percentage points out of these 10 had voted FOR, AGAINST, and UNDECIDED, respectively, in the BEFORE vote.

How did votes change from BEFORE to AFTER?

The table above only lists data for a single debate. Let’s look at trends across all n=26 debates for which full voting data is available. When looking at multiple debates, FOR and AGAINST are pretty arbitrary and depend on the wording/structure of the motion being debated. Thus, in the initial votes, I’ll combine FOR and AGAINST into the “had an opinion” bucket.

graph2

To get our bearings here:

  • 51 percentage points (purple) of the total audience had an opinion (FOR or AGAINST) in the BEFORE vote and voted in exactly the same way in the AFTER vote.
  • 14 percentage points (orange) had an opinion BEFORE, but switched their vote to the opposite AFTER. [Might these be strong partisans for a particular side who want their side to win–by having the biggest increase–and therefore vote for the opposite side initially and their own side in the final vote in order to maximize the increase in votes for the side they favor?….Or do they legitimately change their minds?]
  • 26 percentage points (yellow) were UNDECIDED BEFORE, but picked a side AFTER.
  • 4 percentage points (green) had voted either FOR or AGAINST in the BEFORE vote, but became UNDECIDED in the AFTER vote.
  • 5 percentage points (purple) were UNDECIDED in both votes.

Quick observations:

On average…..

  • Only 56% of the audience voted exactly the same BEFORE and AFTER!
  • Some people changed their mind! Among people who had picked a side in the BEFORE vote, 20% of these people voted for the opposite in the final vote and 6% became undecided.
  • The vast majority (84%) of people who were initially UNDECIDED ended up picking a side in the AFTER vote. However, those remaining 16% of the initially-UNDECIDED voters who stayed undecided made up 57% of the total number of UNDECIDEDs in the AFTER vote. Only 9% of the full audience was UNDECIDED in the AFTER vote.

But who exactly did the UNDECIDEDs all vote for?

For ease of comparison in a moment, let’s define people who have “picked a side” (P.A.S.) as those who voted either FOR or AGAINST (not UNDECIDED). We’ll look at the ratio of FOR/P.A.S: the proportion of FOR votes out of all FOR and AGAINST votes.

Considering the “yellow” voters who had been UNDECIDED, but eventually picked a side, how do their final votes compare to those of the rest of the audience? Let’s look at the proportion of these people who picked FOR in the AFTER vote (the FOR/P.A.S. ratio).

Which of the following do you think is most closely associated with the AFTER votes of these UNDECIDEDs?

  1. The BEFORE votes of the rest of the audience (FOR/P.A.S. in BEFORE vote). Maybe the population as a whole forms opinions on the issue in a certain proportion (equal to the proportion in the initial vote). Once audience members become informed about the issue through the debate, they’ll vote in a similar proportion as had people who had already been well enough informed to pick a side.
  2. The AFTER votes of the rest of the audience (FOR/P.A.S. in AFTER vote, among those who had picked a side initially). Based on the debate, maybe a certain percentage will vote FOR regardless of whether they were UNDECIDED to start with.

Here is the data on both of those possibilities:

 

 

          1                                                                           2graph3

A diagonal line (slope=1) would indicate that the proportion of UNDECIDEDs voting FOR in the AFTER vote exactly matched the proportion of FOR votes BEFORE (graph 1) and proportion of FOR votes AFTER (graph 2) among those who had picked a side.

A glance at the graphs shows that the AFTER vote of the UNDECIDEDs is more closely related to the AFTER vote of everyone else (graph 2) than it is to the BEFORE vote of everyone else (graph 1). The math backs this up. Doing a single-variable linear regression on each scenario, scenario 2 comes out ahead: bigger R^2 (.12 vs. .45), higher actual slope (.32 vs. .67), and lower p-value on the slopes (.09 vs. .00017).

Everyone else’s AFTER votes are a better predictor of the UNDECIDEDs’ votes than everyone else’s BEFORE votes! A possible explanation for this is that that the final vote by people who had previously picked a side is indeed related to the final vote of the previously-UNDECIDEDs, due to the substance of the debate.  It could also be partially due to the rest of the audience simply being more honest in their voting in the second vote (recall that people who come in strongly supporting one side have a strong incentive to lie in the BEFORE vote in order for their side to increase its share of votes by more in the AFTER vote). If there is some universal proportion of people voting FOR on a particular motion once they are fully informed, the (now-assumed-to-be-voting honestly) voters who picked a side initially are apparently more connected to the votes of the previously-UNDECIDEDs.

To sum up….

  • Most people who start with an opinion (FOR or AGAINST) end with the same opinion: 74%.
  • Most people who are initially UNDECIDED do pick a side: 84%. The side they pick is more closely related to the AFTER vote of the others than the BEFORE vote of the others.
  • A few people who start with an opinion actually change their minds during the debate: 20%. It is not clear how many of these people authentically change sides and how many are simply gaming the voting system to benefit a favored side. Conceivably, a voting system could be designed that can’t so easily be gamed, but such a system may have other downsides, and would almost certainly have the downside of being harder for the audience to understand than the current “winner= the side that increased more” system.
  • A few people who were UNDECIDED stay UNDECIDED: 16%.
  • A (very) few people who had an opinion to start with become UNDECIDED: 6%.

Apparently, hearing arguments on two sides of an issue (arguments that audience members perceive to be high-quality) leads most people to settle on an opinion, even when given the option be UNDECIDED. In this situation, where lots of focus in put on who will win (as in many other contexts), people could feel socially uncomfortable being UNDECIDED.

However, if an issue is complex and nuanced enough to be debated in this context, there certainly seems to be value in realizing through this process that you are, in fact, UNDECIDED. I wonder what would happen if a third pair of panelists was included in the debate: panelists who would argue for UNDECIDED–they “win” if the UNDECIDED vote increases by the greatest amount!

Do you agree that there is value in people being UNDECIDED? Do you have other ideas (in this context or elsewhere) for how to empower more people to be UNDECIDED?

 

*Here is the data. The first tab includes the BEFORE and AFTER voting data for all n=92 debates that have taken place as of today (7-27-14). The second tab includes data for the n=26 more recent debates for which complete data (the exact mappings from people’s BEFORE votes to FINAL votes) is available.

HT to Ian Simon for posing an initial question about the decrease in UNDECIDED voters in Intelligence Squared debates.

Advertisements
This entry was posted in Convopointer and tagged , , . Bookmark the permalink.

One Response to Intelligence Squared debates: voting data

  1. Lauren Meyer says:

    This is fascinating stuff! Great job explaining potentially confusing concepts in a clear way.

    “It actually makes sense that losing the initial vote may be an advantage…” >> You outline a great reason why this is true statistically, but it may also be true politically/based on the debate itself. Having a topic where your position initially sounds more implausible or less popular gives you more room to convince people that your controversial side actually has some merit and isn’t as crazy as it originally sounded during the debate.

    Like

What do you think? Anything to add? Anything you disagree with? Please add your comments below....

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s