Nineteen Sixty-four is a research blog for the Center for Applied Research in the Apostolate (CARA) at Georgetown University edited by Mark M. Gray. CARA is a non-profit research center that conducts social scientific studies about the Catholic Church. Founded in 1964, CARA has three major dimensions to its mission: to increase the Catholic Church's self understanding; to serve the applied research needs of Church decision-makers; and to advance scholarly research on religion, particularly Catholicism. Follow CARA on Twitter at: caracatholic.

3.28.2011

A Volatile Mix: Research, News, and Advocacy Groups

I should have known better (I was a reporter once)…

I was interviewed last week by a reporter from the Catholic News Agency (CNA) for a story regarding the Public Religion Research Institute’s (PRRI) “Catholic Attitudes on Gay and Lesbian Issues: A Comprehensive Portrait from Recent Research.” In this post I'll pull back the curtain a bit and describe how news about research can go astray—especially when competing advocacy groups are reading and weighing in on every number and word. I'll also provide a look at some data that was not part of the PRRI report on this topic.

I am quoted several times in the CNA story in my capacity as a survey researcher and I have no issue with the accuracy of my quotes but I believe the story is written in a way to give the impression that I doubt the results of the PRRI study, which indicate many Catholics are supportive of legal recognitions of same-sex relationships. CNA may have its doubts about the results and I believe they have written a story that reflects this impression; however these doubts are just not generally shared by me.

I do have some issues with the presentation of the PRRI results in their report and press release. These did not contain information which is typically reported in documents like this, such as the number of interviews and margin of error for sub-groups (these are reported in a separate document). When specifically making statements about “majority” opinions this information is essential for people to be able to evaluate the precision of estimates. Thus, it is true that in my opinion, the PRRI study was missing some information in presentation in the results report but the consistency of these results with other studies leads me to believe it is not some sort of outlier.

Eventually the CNA author of the story notes, “Gray said that some numbers in the report are ‘pretty consistent’ with publicly available data.” If any comment I made was to be used as a lead that should have been the one. My comments about the study’s presentation should not have been used as the feature or the headline of this story, which was Info missing from survey claiming Catholic support for gay ‘marriage.’

Thus, I became concerned about the presentation of my statements in the CNA story. I expressed this concern to CNA the day the story came out. I did not receive a direct response [update: I did receive an email from the editor on Tuesday 3/29]. However, they did add some additional content to the quote above that reads “…and that he considers the study to be accurate within its own margin of error.”

On all sides of the debate regarding this research there seems to be so much discussion now of “is it a majority or is it not?” I can’t be clear enough about this: It simply is what it is, given the margin of error. Surveys are not referendums. No votes are taken of a population. These are interviews with a sample. More so, what if it is a majority? Does this imply that the Catholic Church needs to change to better fit the attitudes of the Catholic population in the United States? There are more than a billion members in the Catholic Church worldwide. Why would this global Church alter its teachings to meet 50% plus one of the current preferences of some 70 million people in North America? Even if the U.S. bishops did want to do this they can’t exactly go to Rome and say “We’re going to go in a different direction on this teaching because it is not playing well in recent polling data in the United States.” The Church is not a corporation that must be responsive to customers nor is it a democratic government that must represent voters. Instead the Church is a religious institution. The whole point is that it stands for a particular system of belief.

At the same time, it is still useful to understand public opinion and the implications it may present. And the best method of estimating something with greater precision is to triangulatecomparing the results of multiple surveys. This is some of what PRRI does in its report. One can further compare to trends in other surveys including the widely used and trusted General Social Survey (GSS). However, this source also has limitations with sub-group margins of error.

The GSS question on this issue is specific to marriage (i.e., excluding civil unions). In 2010, 20% of adult Catholics “strongly agreed” that “homosexual couples should have the right to marry one another.” An additional 28% “agreed.” Thus, overall 48% of Catholic respondents indicated some level of agreement with this statement. The margin of error for the 2010 Catholic data is ±5.8 percentage points. Thus, the point estimate for agreement could range as high as 54% or as low as 42%. Only 13% of Catholics expressed agreement with this statement in 1988 when the question was first asked (margin of error was ±5.8 percentage points). Levels of agreement with the statement have grown as disagreement has diminished. A consistent percentage of Catholics state they “neither agree nor disagree” with the statement.

The point again is that we just don't know if it is exactly a majority or not now. The point estimates in the GSS are too close to the 50% mark and the margins of error are too large to say it is one way or the other. Question wording only provides an additional complication. I was quoted in the CNA story as stating a preference for the three option question including civil unions (see the discussion on pg. 8 of the PRRI results report). I think this likely best reflects the opinion of Catholics whereas the two option question artificially constrains preferences/opinions (which may be a political reality in a real world voting/referendum situation; yet not in a survey measuring attitudes). It is the case that there is a majority of Catholics who would support some form of legal recognition in the PRRI data that is beyond margin of error.

There is another GSS question that has a longer history that is related to the results of the marriage question shown above. The GSS asks respondents if “sexual relations between two adults of the same sex” is wrong. In the 2010 GSS, for the first time, the percentage of adult Catholics indicating this is “not wrong at all” outnumbered those who said it was “always wrong” (44% compared to 42%). Yet here again margin of error prevents us from knowing if this is a precise distribution in the population in 2010 (margin of error was ±5.9 percentage points). 

As one can see from the trend in the figure below, the real point of change occurs somewhere in the early 1990s and has continued to evolve to this day. Responses to this question differ by age with younger Catholics being more likely than older Catholics to say this is “not wrong at all.” However, the sharp change in the population overall in the early 1990s cannot be explained by generational replacement alone.


Of those Catholics who think that “sexual relations between two adults of the same sex” is “not wrong at all,” 79% agree that “homosexual couples should have the right to marry one another.” Sixty-nine percent of Catholics who think this is “always wrong,” disagree with the right to marry statement.

It is the case that frequency of Mass attendance correlates with responses to these questions. PRRI pointed this out in the data (pg. 7 of their results report) and many who have been critical of the report have argued that this is “what really matters.” The only complication to this point of view is that Mass attendance varies by age and generation (something again that CNA did quote me on at the end of the story). So is it Mass attendance that makes one more likely to oppose civil unions or marriage for same-sex couples or is it something generational? It is likely both but which matters more? And again the figure above indicates some sort of “period effect” in the early 1990s that is also likely important. Margin of error for sub-groups is the biggest obstacle to understanding and disentangling these effects.

Among Catholics age 30 or younger at the time they were surveyed in the GSS from 2004 to 2010 (all available years for this question pooled together), 55% agreed that “homosexual couples should have the right to marry one another.” The margin of error for this group is ±5.9 percentage points. Is that a majority? Perhaps, but we still can’t be sure. Among those older than 30 at the time of their interview, 36% agreed with the statement. The margin of error for this group is ±5.7 percentage points. Is this a minority? Absolutely. There is a 19 percentage point gap (beyond margin of error) between these two groups. Attempting to dice this question further by age and Mass attendance leads to unacceptable sub-group sizes and margins of error in the GSS. However, it is the case that there is a 33 percentage point gap in the PRRI report between weekly Mass attenders and those with less than monthly attendance on the marriage question (26% supporting compared to 59%).

By the way I have no information about nor knowledge of the sources of funding for the PRRI research other than what they cite in their report. I was not asked by CNA to comment on this. I do wonder if CNA ever spoke to PRRI on this issue (or anything else)? Why wasn't any PRRI researcher quoted in their story? How did I become the focus? I can attest to the fact that I have not received (nor would accept) any funding for the comments and research I have shown above. As I have stated previously on this blog I do not do advocacy campaigns. Just the data/facts. But then again that is what got me into this whole thing. I may just care way too much about margins of error!



[A final aside if you've made it this far! ...When I was looking for sub-group margins of error in the PRRI report I told the CNA reporter that I did see a source with some informationCatholic News Service (CNS). They had interviewed and quoted a PRRI researcher (a good journalistic practice). I read the quote from their story: "based on interviews of about 600 Catholics, had a margin of error of plus or minus 6 percent, according to Cox." I then explained this margin to the CNA reporter. It was not until I read the CNA story that the numbers ran through my head and I realized a sample size of 600 has a margin of error of ±4 percentage points not ±6 percentage points. So as far as I can tell either the PRRI researcher misspoke or was misquoted, which I then quoted, to a reporter who then wrote a story that I consider misleading. I hope this sets the record straight.]

Search This Blog

Loading...
© 2009-2013 CARA, Mark M. Gray. Background image courtesy of muohace_dc.