A Volatile Mix: Research, News, and Advocacy Groups

I should have known better (I was a reporter once)…

I was interviewed last week by a reporter from the Catholic News Agency (CNA) for a story regarding the Public Religion Research Institute’s (PRRI) “Catholic Attitudes on Gay and Lesbian Issues: A Comprehensive Portrait from Recent Research.” In this post I'll pull back the curtain a bit and describe how news about research can go astray—especially when competing advocacy groups are reading and weighing in on every number and word. I'll also provide a look at some data that was not part of the PRRI report on this topic.

I am quoted several times in the CNA story in my capacity as a survey researcher and I have no issue with the accuracy of my quotes but I believe the story is written in a way to give the impression that I doubt the results of the PRRI study, which indicate many Catholics are “supportive of legal recognitions of same-sex relationships.” CNA may have its doubts about the results and I believe they have written a story that reflects this impression; however these doubts are just not generally shared by me.

I do have some issues with the presentation of the PRRI results in their report and press release. These did not contain information which is typically reported in documents like this, such as the number of interviews and margin of error for sub-groups (these are reported in a separate document). When specifically making statements about “majority” opinions this information is essential for people to be able to evaluate the precision of estimates. Thus, it is true that in my opinion, the PRRI study was missing some information in presentation in the results report but the consistency of these results with other studies leads me to believe it is not some sort of outlier.

Eventually the CNA author of the story notes, “Gray said that some numbers in the report are ‘pretty consistent’ with publicly available data.” If any comment I made was to be used as a lead that should have been the one. My comments about the study’s presentation should not have been used as the feature or the headline of this story, which was “Info missing from survey claiming Catholic support for gay ‘marriage.’”

Thus, I became concerned about the presentation of my statements in the CNA story. I expressed this concern to CNA the day the story came out. I did not receive a direct response [update: I did receive an email from the editor on Tuesday 3/29]. However, they did add some additional content to the quote above that reads “…and that he considers the study to be accurate within its own margin of error.”

On all sides of the debate regarding this research there seems to be so much discussion now of “is it a majority or is it not?” I can’t be clear enough about this: It simply is what it is, given the margin of error. Surveys are not referendums. No votes are taken of a population. These are interviews with a sample. More so, what if it is a majority? Does this imply that the Catholic Church needs to change to better fit the attitudes of the Catholic population in the United States? There are more than a billion members in the Catholic Church worldwide. Why would this global Church alter its teachings to meet 50% plus one of the current preferences of some 70 million people in North America? Even if the U.S. bishops did want to do this they can’t exactly go to Rome and say “We’re going to go in a different direction on this teaching because it is not playing well in recent polling data in the United States.” The Church is not a corporation that must be responsive to customers nor is it a democratic government that must represent voters. Instead the Church is a religious institution. The whole point is that it stands for a particular system of belief.

At the same time, it is still useful to understand public opinion and the implications it may present. And the best method of estimating something with greater precision is to triangulate—comparing the results of multiple surveys. This is some of what PRRI does in its report. One can further compare to trends in other surveys including the widely used and trusted General Social Survey (GSS). However, this source also has limitations with sub-group margins of error.

The GSS question on this issue is specific to marriage (i.e., excluding civil unions). In 2010, 20% of adult Catholics “strongly agreed” that “homosexual couples should have the right to marry one another.” An additional 28% “agreed.” Thus, overall 48% of Catholic respondents indicated some level of agreement with this statement. The margin of error for the 2010 Catholic data is ±5.8 percentage points. Thus, the point estimate for agreement could range as high as 54% or as low as 42%. Only 13% of Catholics expressed agreement with this statement in 1988 when the question was first asked (margin of error was ±5.8 percentage points). Levels of agreement with the statement have grown as disagreement has diminished. A consistent percentage of Catholics state they “neither agree nor disagree” with the statement.

The point again is that we just don't know if it is exactly a majority or not now. The point estimates in the GSS are too close to the 50% mark and the margins of error are too large to say it is one way or the other. Question wording only provides an additional complication. I was quoted in the CNA story as stating a preference for the three option question including civil unions (see the discussion on pg. 8 of the PRRI results report). I think this likely best reflects the opinion of Catholics whereas the two option question artificially constrains preferences/opinions (which may be a political reality in a real world voting/referendum situation; yet not in a survey measuring attitudes). It is the case that there is a majority of Catholics who would support some form of legal recognition in the PRRI data that is beyond margin of error.

There is another GSS question that has a longer history that is related to the results of the marriage question shown above. The GSS asks respondents if “sexual relations between two adults of the same sex” is wrong. In the 2010 GSS, for the first time, the percentage of adult Catholics indicating this is “not wrong at all” outnumbered those who said it was “always wrong” (44% compared to 42%). Yet here again margin of error prevents us from knowing if this is a precise distribution in the population in 2010 (margin of error was ±5.9 percentage points). 

As one can see from the trend in the figure below, the real point of change occurs somewhere in the early 1990s and has continued to evolve to this day. Responses to this question differ by age with younger Catholics being more likely than older Catholics to say this is “not wrong at all.” However, the sharp change in the population overall in the early 1990s cannot be explained by generational replacement alone.

Of those Catholics who think that “sexual relations between two adults of the same sex” is “not wrong at all,” 79% agree that “homosexual couples should have the right to marry one another.” Sixty-nine percent of Catholics who think this is “always wrong,” disagree with the “right to marry” statement.

It is the case that frequency of Mass attendance correlates with responses to these questions. PRRI pointed this out in the data (pg. 7 of their results report) and many who have been critical of the report have argued that this is “what really matters.” The only complication to this point of view is that Mass attendance varies by age and generation (something again that CNA did quote me on at the end of the story). So is it Mass attendance that makes one more likely to oppose civil unions or marriage for same-sex couples or is it something generational? It is likely both but which matters more? And again the figure above indicates some sort of “period effect” in the early 1990s that is also likely important. Margin of error for sub-groups is the biggest obstacle to understanding and disentangling these effects.

Among Catholics age 30 or younger at the time they were surveyed in the GSS from 2004 to 2010 (all available years for this question pooled together), 55% agreed that “homosexual couples should have the right to marry one another.” The margin of error for this group is ±5.9 percentage points. Is that a majority? Perhaps, but we still can’t be sure. Among those older than 30 at the time of their interview, 36% agreed with the statement. The margin of error for this group is ±5.7 percentage points. Is this a minority? Absolutely. There is a 19 percentage point gap (beyond margin of error) between these two groups. Attempting to dice this question further by age and Mass attendance leads to unacceptable sub-group sizes and margins of error in the GSS. However, it is the case that there is a 33 percentage point gap in the PRRI report between weekly Mass attenders and those with less than monthly attendance on the marriage question (26% supporting compared to 59%).

By the way I have no information about nor knowledge of the sources of funding for the PRRI research other than what they cite in their report. I was not asked by CNA to comment on this. I do wonder if CNA ever spoke to PRRI on this issue (or anything else)? Why wasn't any PRRI researcher quoted in their story? How did I become the focus? I can attest to the fact that I have not received (nor would accept) any funding for the comments and research I have shown above. As I have stated previously on this blog I do not do advocacy campaigns. Just the data/facts. But then again that is what got me into this whole thing. I may just care way too much about margins of error!



[A final aside if you've made it this far! ...When I was looking for sub-group margins of error in the PRRI report I told the CNA reporter that I did see a source with some information—Catholic News Service (CNS). They had interviewed and quoted a PRRI researcher (a good journalistic practice). I read the quote from their story: "based on interviews of about 600 Catholics, had a margin of error of plus or minus 6 percent, according to Cox." I then explained this margin to the CNA reporter. It was not until I read the CNA story that the numbers ran through my head and I realized a sample size of 600 has a margin of error of ±4 percentage points not ±6 percentage points. So as far as I can tell either the PRRI researcher misspoke or was misquoted, which I then quoted, to a reporter who then wrote a story that I consider misleading. I hope this sets the record straight.]

Emerging U.S. Catholic Trends: GSS 2010

There are some surprises and interesting trends regarding the U.S. Catholic population in the newly released 2010 General Social Survey (GSS) data. This post is the first in a series describing some of the CARA analyses of this survey. To begin with, the 2010 GSS indicates (again) that the U.S. Catholic population continues to grow. One in four adults in the United States self-identifies their religious affiliation as Catholic.

This percentage has remained consistent (within margin of sampling error) in major surveys all the way back to the 1950s. As the total population grows, that 25% represents more Catholics in real numbers each year. The 2010 GSS represents the second consecutive estimate in the series where a minority of Americans self-identifies as Protestant (47%). The unaffiliated "Nones" continue to grow and are now estimated to make up 18% of the U.S. adult population.

The margin of sampling error for the Catholic sample of the GSS varies by year as the total number of respondents has changed over time. Overall, in the 2010 GSS 2,044 individuals were interviewed resulting in a margin of sampling error of ±2.2 percentage points (e.g., the table above; click to enlarge). This included interviews with 482 Catholic respondents resulting in a margin of error of ±4.5 percentage points when looking within this sub-group specifically.

The GSS indicates that the Catholic retention rate is continuing to slowly decline. In the 2010 GSS, it is estimated that 68% of adults in the U.S. who were raised Catholic, continue to self-identify as Catholic now.

In 1973, the Catholic retention rate was 84%. The current 68% estimate is identical to the Pew Religious Landscape study in 2007. If the current rate of decline continues (click the table below to enlarge), the Catholic retention rate is expected to be 54% in 2050.

Many assume that the only way the Catholic population could be maintaining its 25% share of the adult population is through immigration. Yet, clearly something more complex is going on. For one there is not enough immigration to fill the gap (more on this in a future post). Second, the GSS indicates that growth in the foreign-born Catholic population may have unexpectedly halted for a time.

The percentage of Catholics in the GSS reporting that they were foreign-born has dropped from 29% in 2006 to 23% in 2010. This difference is within the margins of error for the two surveys but there is a consistent downward trending observation in 2008. More so, these percentages have been expected to increase with real growth. It is too early to say this is a real decline but it certainly indicates a potential stalling in the growth of the foreign-born Catholic population.

In retrospect there was some evidence that this could be occurring. The Pew Hispanic Center has documented recent declines in segments of the non-citizen population. The sharpest decline in immigrant numbers occurred between 2007 and 2008—just as the U.S. entered a severe economic recession. Estimates of the non-citizen population have yet to reach 2007 levels again.

There is additional evidence of these changes in the responses to ethnicity and ancestry questions in the GSS. For example, the percentage of Catholics who self-identify their ethnicity as Hispanic or Latino has dropped from 38% in 2006 to 32% in 2010 (click the table below to enlarge; note the best data for analysis are from 2006 to 2010 when the GSS began to employ Spanish language interview options). Again, these results are within margin of error but a trending pattern is emerging and this is another figure that was expected to show some growth at this time. More specifically we can see this data pattern among those who have emigrated from Mexico.

Some of what is occurring is likely related to recent immigration trends. However, some of it can also be attributed to declines in Catholic affiliation among Hispanics/Latinos. The GSS estimated that 70% of Hispanics/Latinos in the U.S. self-identified their religion as Catholic in 2006. This has dropped slightly to 63% in the 2010 survey.

The GSS does not show any drop in the Catholic affiliation percentages of non-Hispanic white adults in the United States during this period (22% in the three surveys since 2006). 

The GSS is the primary social science source for representative data on the adult population in the United States (the American National Election Study series is another but this is limited to adult citizens). The GSS, based on face-to-face interviews with representative samples of the U.S. population, has been conducted since 1972. It is often featured in trend analyses in major studies of religion such as Pew’s report for the Religious Landscape Survey (example: pg. 18) and in Robert Putnam and David Campbell’s recent book American Grace (example: pg. 11). 

Unfortunately, when the GSS is released there is no big press coverage or report of findings to read. Instead it is rather quietly made available to researchers (and the public). Literally, thousands of publications including journal articles, dissertations, and books have and will continue to cite these data. If you are interested in looking at the GSS data visit the Survey Documentation and Analysis (SDA) site from the University of California, Berkley. One note of caution, as shown in the last table above, when looking specifically at the results for Hispanic/Latino respondents it is best to focus on the surveys done in 2006, 2008, and 2010 when Spanish language interviews were available.

Sunday Morning: Deconstructing Catholic Mass attendance in the 1950s and now

Social science is not just about surveys, trend data, or focus groups. Some of my favorite types of research involve content and historical analysis. The study of art, for example, can provide some amazing insight that is not visible on a spreadsheet. Norman Rockwell’s “Sunday Morning,” as shown below (click to enlarge), has always caught my eye as an interesting piece because it brings to life some of the survey results we have been seeing for many years and it did it on the cover of The Saturday Evening Post all the way back in May of 1959 (for more see: “Sunday Morning Slackers” from the Post).

Sunday Morning; Norman Rockwell; Published: May 16, 1959; © 1959 SEPS.

Rockwell illustrated this piece five years after appearing in the national print ad shown below, which reads, “Light their life with faith. Bring them to worship this week.” This advertisement was sponsored by Protestant, Catholic, and Jewish lay people in an effort to encourage weekly religious service attendance—especially among parents with their children. This should be the first sign of something amiss with our recollections of this time…

  
They needed a national ad campaign to get people to take their children to church? Wait this is the 1950s right? Didn’t everyone go to church in the 1950s?

According to his biographer (Norman Rockwell: A Life, by Laura Claridge, Random House, New York, 2001), Rockwell lived a life much more like his cover illustration than the text of the ad campaign. He was said to rarely make it to church on Sunday during his first marriage (to a Catholic woman he later divorced) or during his second marriage (to a woman of his own Episcopalian affiliation). Rockwell did have his children baptized from the second marriage but was not a regular at church services otherwise.

In itself this may come as a shock to those who view Rockwell as the visual soul of conservative, small town, religious America. As “Sunday Morning” indicates Rockwell was not shy about pointing out the realities of American religious life even in the 1950s. Take special notice of the father’s unusual morning-bed hair-style and the color of his robe. When one realizes from his biographer that Rockwell may have spent many similar Sundays in his pajamas it is more difficult to typecast the illustrator into some of his other more famous and pious illustrations regarding religion.

It is likely that the family in the illustration is Protestant. Thus, it is not the case that Rockwell was making any comment about Catholicism in this work. But the fact that it made it on to the cover of America’s magazine of record at the time indicates that it resonated with the culture of this period. This issue of the Post was published at a time when weekly Catholic Mass attendance was peaking, as measured in Gallup telephone surveys (74% in 1958 and 72% in 1959). In 2008, Gallup surveys estimated Catholic Mass attendance in any given week had fallen to 42%. Don’t giggle. I know you don’t believe that 42% of Catholics nationally attend Mass in any given week and you’re right. But why do we believe 74% did in 1958?

I am sure there will be some reading this who says, “I remember, I was there.” But what we did see in the 1950s is not important. It is what we did not see… the people who were not in the pews. You can only get an attendance percentage by dividing the Mass attendance count (numerator) by the number of self-identified Catholics in the parish boundaries that could have attended (denominator). All of this is unlikely to be found in your memories! (Note: even Robert Putnam who notorious for highlighting the declining trend lines from the 1950s says the following in American Grace: “research on the accuracy of reporting church attendance… suggests that we should take these self-reports with a grain of salt,” pg. 571).

A piece of art is one thing. Perhaps more can be said by taking a second look at a researcher who was in many Catholic parishes studying Mass attendance in the 1950s. Joseph H. Fichter, S.J., (granduncle to current CARA research associate Fr. Stephen Fichter) famously studied parish life by going door to door and taking censuses, making Mass attendance head counts, observing parish life, and documenting everything possible both qualitatively and quantitatively. 

In his 1954 study, Social Relations in the Urban Parish (University of Chicago Press), Fichter estimates Mass attendance levels based on the number of individuals registered with the parish. But he also provides the counts for what he calls “dormant Catholics” from his census within parish boundaries. These are people who self-identify their religion as Catholic but who do not attend Mass. Thirty-eight percent of the Catholics within the parish boundaries he studied in this book were dormant. Thus, at the outset we know that typical weekly attendance by the measure of this study could have been no more than 62%. But what about among the “active” Catholics? About 79% of the non-dormant Catholics attended Mass on a typical weekend. So overall, the total percentage of self-identifying Catholics attending Mass in this study was estimated to be about 49%.

This is almost exactly what we get in the early 1950s if we “correct” the Gallup trend numbers down for “over-reports” in each year by about 12 percentage points (Why 12 percentage points? See “The Nuances of Accurately Measuring Mass Attendance” and more recent research cited below). The figure below shows this correction for the entire Gallup series.

Attendance over-reports occur as people being interviewed over the phone respond to their interviewer with answers about their behavior that they believe to fit socially desirable expectations. So typically the respondent has just told the interviewer their religion and then they are asked how often they attend services. Many respond in a way that they believe is socially acceptable—even if it does not fit their actual pattern of attendance.

We have some early evidence of this in the Americans’ Use of Time Study, 1965-1966. Here, 57% of Americans when asked directly about their church attendance reported that they had attended in the last week. However, only 39% of these respondents actually indicated attending religious services when recording their time use hour by hour in diaries (i.e., an indirect measurement). For more on this see Philip Brenner’s excellent recent article “Identity Importance and the Overreporting of Religious Service Attendance” in the Journal for the Scientific Study of Religion (March, 2011; pgs 103-115). Brenner not only estimates this phenomenon in the United States but in Europe as well, where this issue is less of a problem. These estimates are in “Exceptional Behavior or Exceptional Identity?” in Public Opinion Quarterly (Spring, 2011; pgs. 19-41). For example, in Italy (2003) time diaries estimate church attendance to be 25% weekly but surveys measure this at 30%. In Spain (2003) attendance is estimated at 16% in time diaries compared to 19% in surveys. In Ireland (2005) these numbers respectively are 42% (time diary) and 46% (survey-based). Here, Brenner also shows that from 1975 through 2008 the average annual over-report in U.S. religious attendance is very stable and much higher than in Europe, at an average of 13.4 percentage points.

In another of Father Fichter's 1950s studies, Southern Parish: The Dynamics of a City Church (Volume I, University of Chicago Press, 1951), he showed that there was also considerable variation in Mass attendance week to week (see figure below). This again is quite similar to some indirect measurements made now. Just as today, Lent brought more Catholics to Mass than during other periods of the year.

Joseph H. Fichter; Southern Parish: The Dynamics of a City Church; Volume I, University of Chicago Press, 1951, pg. 151.

Father Fichter’s observations also indicate that some of the Mass attendance of the 1950s was not as “active” as we might remember it. Here is a passage that likely still resonates with your observations of parish life today:

“A measure of the parishioners’ devotion to the Mass and of their fulfillment of this obligation is seen in the numbers who arrive late and who leave early. By actual count it was noted that, at all Sunday Masses, 8.37 per cent of the congregation arrived after Mass had started and that 6.35 per cent left before it was completed. … Although we have no accurate count, we have noticed that many of these persons are duplicated in both categories. In other words, those who come late also tend to leave early. … The younger males constitute the majority of those who omit part of the Mass, while older females make up the majority who arrive in church well in advance of Mass” (1951, pg. 138).

“By actual count, 35.08 per cent of the congregation read the missal all during Mass, while another 22.08 per cent read some sort of prayer-book while following the priest’s reading of the Gospel. … The remaining persons simply stare off into space, although several men in the last pews sometimes read a copy of Our Sunday Visitor during Mass” (1951, pg. 138).

Over a year of Masses, on average, attenders were much more often female (about 7 in 10 or more) than male—a composition that can only result from some men, perhaps like the man in the Rockwell illustration above, staying home.

Today, CARA’s national surveys use a methodology that minimizes social desirability pressure on respondents to get the most accurate measurements of Mass attendance possible. Many cite our weekly Mass attendance figure in the low 20 percent range. Some also then cite Gallup’s figure from the 1950s and attempt to argue that Mass attendance has fallen from nearly 80% to just above 20%. This is misleading and inaccurate. First, as shown above, the Gallup numbers for the 1950s are inflated by over-reports just as they are in the 1970s or now. Second, CARA and most other survey-based estimates of Mass attendance measure general frequencies of attendance such as “every week” or “at least once a month.” Gallup’s church attendance question measures whether a respondent has attended in the last 7 days. Depending on the week in which this question is asked, one will get very different results. Thus, the best use of the Gallup data is in taking the average for the year in response to this question.

Currently, CARA surveys indicate that 23% of self-identified adult Catholics attend Mass every week. Yet, in any given average week, 31% of Catholics are attending (almost identical the “adjusted” 30% estimate from the Gallup trend). Note there is considerable local variation in Mass attendance levels with higher levels in the Midwest and lower in coastal urban areas). During Lent and Advent, Mass attendance increases into the mid-40 percent-range and on Christmas and Easter, an estimated 68% of Catholics attend.

Thus, if one is seeking to make a comparison of Mass attendance in the 1950s to now, the drop is not 80% to 20%. Instead it is from a peak of 62% in 1958 to about 31% now. This is still a remarkable decline. It means that the Mass attendance you see at Christmas and Easter is a lot like the attendance you might have seen in a typical week in the late-1950s. Yet, even then, as now, there is a significant number of Catholics like the father in Rockwell’s “Sunday Morning” who choose to do something else.

The slope of the Gallup declining trend is accurate. It’s the levels that are off. If you are going to use the Gallup data from the 1950s and make comparisons to CARA data in the 2000s and beyond you’ll need to adjust the Gallup trend (and our collective memory of the 1950s) down to reach reality. And this is of course is not just an artistic endeavor. As I have argued here I believe it to be statistically valid.