The principal survey the military has relied on for its data regarding sexual assault in recent years has been, at best, extremely flawed. Its sample size was too small, it was taken on a volunteer basis—which, any statistician will tell you, means that those who actually fill out the survey are more likely to be highly motivated about its subject, thus skewing the results—and the phrasing of its most important line of questioning was too vague.
For example, the 2012 Workplace and Gender Relations Survey of Active Duty Members—the most recent of the biannual surveys available—never asked its respondents about "sexual assault." Instead, it asked whether the respondents had been victims of "unwanted sexual contact." And a relatively high proportion of survey respondents said in 2012 that they had experienced unwanted sexual contact: 6.1 percent of female respondents, and 1.2 percent of males. But the number of those who actually reported sexual assaults in the military during the same period of time was dramatically lower—an order of magnitude lower, in fact.
Does the difference between these sets of numbers exist because sexual assault is an underreported crime? It seems reasonable that this could be a factor. But to what extent? How is one to compare the apples of questions about "unwanted sexual contact" to the oranges of data about reported sexual assaults? I wrote in September, "Perhaps the [2014] survey currently underway will correct this obvious methodological problem."
Does it ever. Here’s the AP on this year’s survey:
Shocked and offended by explicit questions, some in the U.S. military are complaining about a sexual-assault survey that hundreds of thousands have been asked to complete.
The survey is conducted every two years. But this year's version, developed by the Rand Corp., is unusually detailed, including graphically personal questions on sexual acts.
Some military members told The Associated Press that they were surprised and upset by the questions; some even said they felt re-victimized by the blunt language. None would speak publicly by name.
Pentagon officials confirmed receiving complaints that the questions were "intrusive" and "invasive."
The Defense Department said it made the survey much more explicit and detailed this year in order to get more accurate results as the military struggles to reduce sexual assaults while also encouraging victims to come forward to get help.
The survey questions, obtained by The Associated Press, ask about any unwanted sexual experiences or contact. They include very specific wording about men's and women's body parts or other objects, and kinds of contact or penetration.
A sample question, one of a series of 11 graphic questions out of 34 (some are even more detailed):
"Before 9/18/2013, had anyone made you insert an object or body part into someone's mouth, vagina or anus when you did not want to and did not consent?"
"We've had a number of complaints," said Jill Loftus, director of the Navy's sexual assault prevention program. "I've heard second- and third-hand that there are a number of women, officers and enlisted, who have gotten to the point where they've read the questions and they've stopped taking the survey. They found them to be either offensive or too intrusive — 'intrusive, invasive' — those are the words they used."
One shudders to imagine the wording of those questions that are "even more detailed."
Activists and politicians who have made sexual assault in the military a major agenda item in Washington are expressing concern about this new approach. On Monday morning, Anu Bhagwati, the executive director of the radical feminist Service Women’s Action Network, described the questions reported in the AP’s story as a "blunder." The AP itself quotes Senator Kirsten Gillibrand, who leads the charge on this issue on Capitol Hill, expressing concerns about changes to the survey’s methods:
Sen. Kirsten Gillibrand, D-N.Y., who has pressured the military to deal with its sexual assault problem, said changing the survey questions could skew the study over time.
"I am concerned the new survey was done in a manner that not only prevents comparing apples to apples from previous years. ... I hope this isn't a case of, 'If you don't like the answer, change the question.'"
There is certainly a case to be made that the new survey’s questions are creepy, offensive, intrusive, and disgusting. But Gillibrand’s argument is risible. The earlier versions of the survey were so poorly constructed that the "apples to apples" comparisons it provided were self-evidently unreliable: For example, the 2010 survey indicated that there had been 19,000 cases of unwanted sexual contact, but the 2012 survey indicated that the number had dramatically swung up to 26,000. Swings like this are unsurprising in an online survey conducted on a volunteer basis with a small sample size, and are obviously only loosely tethered to reality—but the results were convenient for Gillibrand, so she would prefer that the methods be unchanged.
The 2014 version makes some effort to correct the problem of sample size, inviting five times the number of servicemen and women to take the survey online. Of course, this does nothing about the volunteer problem, which is likely to continue to skew the results.
The Pentagon, for its part, is defending the more specific language:
"This is a crime of a very graphic nature," Galbreath said. "For us to improve our understanding, it sometimes requires asking tough questions."
He said the Defense Department hired Rand to develop and conduct the survey this year, based on new direction from Congress that the effort be fully independent of the Pentagon. He was aware of the complaints but said that the more succinct the questions are, the more accurate the results will be.
"Research has told us, if I ask someone, 'Have you ever been raped?' they will say, 'No,'" Galbreath said. "If I ask that same person, 'Have you ever been forced to engage in sexual activity against your will?' they might say 'Yes.' It's because of the loaded terms like rape and sexual assault, that it's not very clear to a lot of people what we may be asking about."
So the Pentagon’s concern is that the old language was too vague, but simply substituting the term "sexual assault" for "unwanted sexual contact" would depress the results in a way that doesn’t represent the actual severity of the situation—which seems like a reasonable concern. Thus their plan of action is to avoid loaded umbrella terms and ask specific, graphic, personal, and extraordinarily offensive questions of the survey-takers.
Those survey-takers who are offended by the questions are within their rights—but the DOD’s argument has merit, too. What Gillibrand and Baghwati seem to want is to pass laws and make policy based on self-evidently unreliable data. It is a bit rich for them to complain about the intrusive and creepy nature of the atmosphere they themselves are creating in the military.