The ethics of modern social science require that we "do no harm." But I have a confession to make. I have done some harm and I can prove it.
About five years ago CARA began to transition from random digit dial telephone polls to self-administered surveys taken with probability-based samples (households randomly selected by telephone or mail) where respondents take their surveys on a screen (i.e., either on a computer, tablet, or television). In 2005, we ended up doing a telephone poll and a self-administered poll just weeks apart. Although the samples for both surveys were demographically very similar some of the content questions produced shockingly different outcomes. It was then that I realized in my first few years here conducting the CARA Catholic Poll (CCP) that I had been encouraging a bit of lies (i.e., sin). This post is an atonement of sorts.
How many U.S. adult Catholics go to Mass each week? If you look around the survey research world you'll find estimates in the 35% to 45% range. Survey researchers and the Church have known for quite some time that these polls are off the mark to say the least. There are other methods of estimating Mass attendance from headcounts to time diaries that produce more accurate results which come in at the mid-20% range. In CARA's 2005 polls we saw this disparity:
If we took the telephone results literally we'd estimate a third of Catholics go to Mass every week. But realistically, the self-administered survey was much closer to the mark of a headcount or time-diary study at 23%. At the other end of the spectrum only 19% said they rarely or never attended Mass compared to 35% in the self-administered survey. What was going on? Social desirability bias. When taking a survey over the phone one is most often speaking to another human being. Even though the respondent knows this will be a short conversation, their responses are confidential or anonymous, and that they will never speak to this person again, some still feel shame in answering honestly. Instead they "over-report" their attendance. This is good and bad. The fact that they feel shame shows the cultural norm of attending church is still alive and well among Catholics but it also leads to dishonesty which results in a distorted view of religious practice. What is different in the self-administered survey? The respondent is not interacting with a human being. They feel more comfortable reporting their actual behavior to a computer.
We've written about this issue before but recently going through the CARA archives I was able to identify the types of religious practice questions that most often lead to over-reporting in telephone interviews. The winner of the "biggest bias" award goes to financial giving to one's parish. Three-fourths of Catholics in a telephone poll say they regularly give to their parish's weekly offertory collection. Yet only half say they do when the interviewer is removed and the survey is self-administered. This question parallels the same type of over-reporting social scientists see in national surveys that ask about giving to charity, volunteering, or voting.
Of course once one lies to a survey interviewer they may be inclined to make a future visit to their parish for the Sacrament of Reconciliation... especially since they probably lied about how often they go to confession! As shown below, 12% of adult Catholics say they go to confession at least once a month when interviewed by telephone. This falls to 2% when the survey is self-administered (I like to claim that CARA has reduced the necessity for some Catholics to go to confession by adopting these newer, more accurate methods).
These three types of questions make up a trinity of sorts. Catholics consistently over-report the frequency of going to Mass, confession, and giving to their parish when speaking to a human being. There is little evidence of social desirability bias affecting responses to most other questions. This is telling in that this trinity is what many Catholics feel guilt and embarrassment about not doing.
For example, Catholics apparently do not feel the same guilt about not giving to their annual diocesan appeal. As shown below, 29% said they did so in the 2005 telephone poll compared to 25% in the self-administered survey just weeks apart. The difference between the polls is within margins of error.
Catholics also are not embarrassed to say they don't register with their parish. As shown below, the difference between the two types of surveys is within margins of error.
CARA has gone through its archives and adjusted results from telephone polls to remove the effects of over-reporting by carefully comparing the results of 24 surveys, some conducted by telephone and some self-administered, to establish averages of over-reporting (see a related example of applying these methods to Gallup's trends here). We've reconstructed our best estimates of trends for two of the three measures most distorted by social desirability bias (this is not possible for Reconciliation as this question has not been asked in a sufficient number of surveys). CARA's trends for Mass attendance can always be found here from our Frequently Requested Statistics. The trend for Catholics giving to their parish are shown in the figure below:
Just as with Mass attendance, there has been very little change in giving to one's parish in the last decade. If one only had telephone poll results to look at this would present a much more positive outlook on attendance and giving. But the honest view above is best both for the Church and its respondents! Perhaps people would be even more honest in the confessional if they could type their sins rather than speak them? On second thought, removing the interviewer from the survey process was easy and beneficial. Removing the priest from the confessional is certainly no parallel.
Image above courtesy of emilio labrador at Flickr Creative Commons.