The Nuances of Accurately Measuring Mass Attendance

How many Catholics are at Mass on a weekly basis? This percentage varies depending on if one is interested in those who attend every week and those who are at Mass on any given weekend.  There are also important differences in how one conducts the poll that generates these numbers.

CARA has conducted 19 CARA Catholic Polls (CCP), national surveys of adult self-identified Catholics, since 2000. Some of these have been by telephone and others have been conducted online using Knowledge Networks national panel. There are important and interesting differences between the results of Mass attendance questions from CARA’s online self-administered polls and CARA's telephone polls that are strongly related to the effects of the presence of an interviewer.

These differences are not limited to Mass attendance and generally are observable for any socially desirable activity from financial giving and frequency of prayer and confession. CARA’s self-administered surveys consistently show lower levels of Mass attendance than what is exhibited typically in CARA’s telephone polls.  Survey researchers have long understood that the personal interaction between interviewer and respondent can create over-reports of certain activities (such as voting or giving to charity) that are considered socially desirable.  Responses to questions regarding attendance at religious services are known to be biased toward estimates higher than actual attendance as measured by head counts.   Below we show varying estimates of Catholic weekly attendance using telephone polls and other methods such as head counts and time diaries.

CARA's self-administered surveys use methods that do not require a human interviewer--respondents are answering questions as they appear on-screen (either through a computer or on their television using MSN TV).  Self-administered surveys, such as this, are known to reduce over-reports for questions that have socially desirable response sets (encompassing attitudes people believe they “should” have or behaviors they feel they “should” do), producing results that more closely reflect actual behavior than estimates derived from telephone polls.  

The estimated percentage of Catholics attending Mass every week using the responses to CARA's self-administered surveys are more consistent with what is found in estimates of the attendance of Catholics derived from other methods, such as sample-based head counts and time diary studies.  As the figure below shows, results from 12 CARA telephone surveys and seven CARA-Knowledge networks surveys (using self-administered methods), produce no statistically significant changes in weekly Mass attendance between 2000 and 2008 by either method of polling.  All variations are within the sampling margin of error.  The difference between the two methods of polling is consistently about 8 to 14 percentage points for those who say they attend weekly or more often.  On average, in CARA's self-administered surveys 22% to 23% percent of adult self-identified Catholics say they attend Mass on a weekly basis (i.e. every week).

Some other surveys, such as those conducted by Gallup, ask about religious service attendance in any given week (e.g., the last seven days).  In the table below, we convert the responses from the CARA question and estimate the percentage of Catholics that attend Mass in any given week rather than every week.  This is estimated to be 31.4%.

 

By chance one might expect about 2% of those who say they “rarely or never” attend Mass to have attended Mass in any given seven-day period (odds of 1 in 52).  If one takes the 32% of Catholics responding in this manner and multiplies it by 2%, one can estimate that 0.6% of Catholic Mass attendance in any given week is made up of those who say they "rarely or never" attend Mass.  This same calculation can be done for each category of responses that indicate less than weekly attendance. 

Gallup estimates that 45% of Catholics attend Mass in any given week. CARA estimates this to be about 13 percentage points lower.  This is consistent with expectations as Gallup polls use an interviewer over the phone and are thus influenced by social desirability bias.

Note: Knowledge Networks panel has been shown to be representative to well within 1 percentage point to the U.S. Census Current Population Survey (CPS) demographics for gender, age, race and ethnicity, education, and region.  See: Baker et al. (2003), “Validity of the Survey of Health and Internet and Knowledge Networks Panel and Sampling,” Stanford University and Krosnick and Chiat Chang (2001), “A Comparison of Random Digit Dialing Telephone Survey Methodology with Internet Survey Methodology as Implemented by Knowledge Networks and Harris Interactive,” Ohio State University. The panel is updated on a quarterly basis and those persons who are sampled and asked to join the Knowledge Networks panel receive subsidized Internet access and other incentives.  For those who do not own computers, Knowledge Networks provides a television-based Internet system (MSN TV) for free.  These steps ensure that the Knowledge Networks panel is as reflective as possible of the national population and that it is not biased towards those who have pre-existing access to the Internet.

For references or more information on this topic see: 

Mark Chaves and James C. Cavendish. 1994. "More Evidence on U.S. Catholic Church Attendance." Journal for the Scientific Study of Religion. 33: pp. 376-381.
Stanley Presser and Linda Stinson. 1998. "Data Collection Mode and Social Desirability Bias in Self-Reported Religious Attendance." American Sociological Review. 63: pp. 137-145.
C. Kirk Hadaway and Penny Long Marler. 2005. "How Many Americans Attend Worship Each Week? An Alternative Approach to Measurement." Journal for the Scientific Study of Religion. 44: pp. 307-322.