Nineteen Sixty-four is a research blog for the Center for Applied Research in the Apostolate (CARA) at Georgetown University edited by Mark M. Gray. CARA is a non-profit research center that conducts social scientific studies about the Catholic Church. Founded in 1964, CARA has three major dimensions to its mission: to increase the Catholic Church's self understanding; to serve the applied research needs of Church decision-makers; and to advance scholarly research on religion, particularly Catholicism. Follow CARA on Twitter at: caracatholic.

3.10.2010

Replicate Before You Speculate Too Much…

More social science research findings regarding Catholic colleges and universities are being reported and discussed. The focus has been on an article in the peer-reviewed Journal for the Scientific Study of Religion called, “‘Hooking Up’ at College: Does Religion Make a Difference?

The study concluded that Catholic women attending non-Catholic and Catholic colleges “display roughly a 72 percent increase in the odds of ‘hooking up’ compared to those women with no religious affiliation” (p. 544).  The study also finds that women [Catholic and non-Catholic] at Catholic colleges and universities “are almost four times as likely to have participated in ‘hooking up’ compared to women in secular schools” (p. 544).  Thus, there are results regarding Catholic women at all colleges and for all women at Catholic colleges and universities.

There are some important methodological issues to consider:
  • A “hook up” is very widely defined as “when a girl and a guy get together for a physical encounter and don’t necessarily expect anything further” (p. 540). As the authors caution, “‘Hooking up’ may refer to a broad range of physical acts ranging from kissing to sexual intercourse” (p. 548). It is difficult to know just what respondents are reporting in responding “yes.”
  • The study, published in 2009, is based on a telephone survey conducted in 2001.
  • The sample is national and consists of randomly selected women attending four-year colleges at that time (men are not included in the sample).
  • The original sample includes 1,000 respondents at 195 colleges and universities. After removing Mormon respondents and colleges and accounting for non-response to questions used in the analysis there are 919 respondents studied.
  • A total of 38% of the 919 respondents reported that they had “hooked up” since they began college.
  • A total of 31% of women in the sample are Catholic (287 respondents)—yet most of these respondents attend non-Catholic colleges and universities.
  • Only 6% of all respondents—Catholic and non-Catholic—attended a Catholic college or university (52 respondents).
  • Thus, there are only interviews with 39 Catholic women attending Catholic colleges in the study! A conservative estimate of the number of Catholic women attending Catholic college at the time is 85,000. The margin of sampling error for 39 interviews generalizing to a population of 85,000 is +/- 15.7 percentage points.
  • Furthermore, these large margins of error are compounded by the small number of Catholic colleges these women attended at the institutional level.  The randomly selected respondents ended up only being in attendance at a total of 16 Catholic colleges and universities when more than 240 existed at the time (resulting in a margin of sampling error at the level of the institution of +/- 23.7 percentage points).

Although much of the concluding focus of the article is on Catholic women and Catholic colleges there are simply not enough interviews with women—Catholic and non-Catholic—attending Catholic colleges and universities to generalize to those populations in 2001. Whenever statistical results are based on small sample sizes/sub-groups one must be very careful at interpreting the results and further generalizing these to the populations they are intended to represent. 

The authors have made no mistakes—what they have produced is rather standard practice in academic social science survey research (although I would have strongly recommended controlling for household income which is related to college enrollment and choice). They have identified a compelling statistical association in the data. Rightfully, they note the limitations of the exploratory analysis and welcome additional research. This is what is needed. Replication with a larger sample would tell us if this is an anomaly of small sample size or a real effect (for both religious identity and college affiliation). 

We simply cannot know nor should anyone generalize at this point to tens of thousands of Catholic women attending Catholic colleges today from interviews with 39 such women in 2001.

It is interesting to read the authors hypotheses regarding potential causes for the potential Catholic effects: 
  • “Unlike conservative and mainline Protestant churches, the Catholic Church appears to invest few resources into youth ministry and education. … [T]his lack of spiritual nurturing may lead some young Catholics to rebel against the normative constraints of the Church” (p. 545)
  • “Catholic youth… may not enter college with the same level of religious commitment as their Protestant counterparts” (p. 546).
  • “Simply put, it may be that university investments in religious instruction and education are ‘too little too late’ for some students” (p. 546).
Thus, the authors note that some of the causes may pre-date the arrival of students on a Catholic campus and are thus unrelated to what a Catholic college does (or does not do).

The potential causes related to the institution are noted as such:
  • “Catholic schools may bring together men and women who have much in common, not only religiously but socially as well” (p. 546).
  • “Catholic colleges and universities, like Protestant schools, may place emphasis on the importance of marriage and family. As a result, finding one’s future mate may not only be an individual goal, but an institutional priority as well” (p. 546).
  • “In sharp contrast to conservative Protestant educational institutions, Catholic schools often have loose regulations on alcohol” (p. 546).
It is unlikely that Catholic colleges could be faulted for bringing together students who “have much in common” or in placing emphasis on “marriage and family.” The only potential institutionally related cause of concern noted by authors is related to the regulation of alcohol—an issue that Catholic and non-Catholic colleges have attempted to manage in recent years following significant research on college binge drinking.

The authors are also careful to note some limitations:
  • “The cross-sectional nature of the data makes it impossible to establish causal direction if these empirical associations” (p. 547).
  • “Research of this kind necessarily relies on self-reports of physical encounters, and some skepticism about the reliability of such data may be warranted” (p. 547).
In addition to these issues and the small number of Catholic observations it is also important to note that the regression models overall are fairly weak in explaining who “hooks up” and who does not among the respondents. The largest pseudo R-square is .063 (these can range from 0.0 to 1.0) and this only increased slightly from the baseline model with the addition of religious identity and religious affiliation of the colleges.  In sum, the factors in the model are not explaining a great deal about the likelihood of someone having “hooked up” or not.

The authors note additional research is needed and there is already more available that has been conducted since 2001. This research does not speak to the specifics of “hooking up” but it can narrow the focus a bit by looking at attitudes regarding sex before marriage. This has two positive features to this shift in focus. The first is that the behavior studied is more narrowly defined than “a physical encounter” and second these questions are generally asked in a way that avoids the social desirability biases of self-reported behavior.

The General Social Survey (GSS) provides a look at trends. This nationally representative survey of the U.S. adult population has been administered every few years since 1972. Here is the pertinent question: “There's been a lot of discussion about the way morals and attitudes about sex are changing in this country. If a man and woman have sex relations before marriage, do you think it is always wrong, almost always wrong, wrong only sometimes, or not wrong at all?”

In 1972, 39% of adult Catholics responded “always wrong” to this question. Among Catholics attending Mass at least once a week, 54% responded as such. In the most recent GSS survey, conducted in 2008, only 14% of Catholics responded “always wrong” to this question. Among Catholics attending Mass at least once a week, 30% responded as such. This overall trend for Catholics is shown below.
By comparison, adult Protestants have been more likely to respond “always wrong” to this question historically and still today. There is relative stability among all Protestants and among those attending church at least once a week to this question.
 From these data it is apparent that broad cultural changes are occurring within the Catholic population regarding this issue and these are unlikely to have little to do with the influence of Catholic colleges. More so, the changes are so significant, that Catholic colleges may be greatly challenged in attempting to address it.

CARA most recently asked a similar question in a 2006 CARA Catholic Poll (CCP) of a national random sample adult self-identified Catholics. Respondents were presented with a series of statements and asked about their agreement with each. One of these statements was: “Premarital sex between persons who are committed to each other can be morally acceptable.” Respondents could answer: 5=Strongly disagree, 4=Somewhat disagree, 3=Neither agree nor disagree, 2=Somewhat agree, or 1=Strongly agree.

In this 2006 poll, 26 percent of adult Catholics “strongly” agreed with the statement and another 26 percent “somewhat agreed” resulting in a majority (52 percent) expressing an opinion of agreement that is inconsistent with Catholic Church teachings (19 percent disagreed “somewhat” or “strongly” and the remainder neither agreed nor disagreed).

The table below shows the results of five regression models where the dependent variable is the respondents’ answers to the CARA question about premarital sex (the independent variables are as closely matched as possible to the variables used in the "hook up" study). A positive coefficient represents a positive relationship between an independent variable and the dependent variable. A higher score on the response scale, representing disagreement with the statement, is larger (5) than a response representing agreement (1). Thus, the positive coefficients for age in these models represent that the higher the age of the respondent, on average, the higher their response on the scale—representing greater disagreement. It is the case that older Catholics are more likely than younger Catholics to disagree that “premarital sex between persons who are committed to each other can be morally acceptable.”  Only coefficients noted with an asterisk, such as those for age in all five models, are statistically significant.

This survey included 1,892 respondents (male and female), of which 225 reported that they had attended a Catholic college or university. This results in a margin of sampling error for Catholic college respondents of +/- 6.5 percentage points. It is important to note this survey includes adult Catholics of all ages, thus many reporting enrollment in a Catholic college or university may have been on campus decades ago.

The first model—a baseline estimate—includes no information about the respondents’ education (or income). In this model we learn that among those more likely to disagree with the statement are older, living in the Midwest (compared to the West), married or single and never married (compared to being separated or divorced), attend Mass either weekly or at least once a month (compared to those attending less often), and have higher subjective religiosity (express importance of religion in their life).  The only statistically significant negative coefficient is for those Catholics living in the Northeast where there is more agreement with the statement (compared to the West). This finding is consistent with the results of the “hooking up” study.


The addition of variables about education does not significantly alter the associations noted above.  In model 2, the effect of college enrollment is tested. The excluded reference category is for respondents who have not attended college. Two other variables are included, one representing enrollment at a Catholic college or university and another for enrollment in a college that is not affiliated with the Catholic Church. Both coefficients are negative and statistically significant. However, the coefficient for non-Catholic colleges is larger and more highly significant than the coefficient for enrollment in a Catholic college or university.  These results indicate that Catholics who do not attend college are less likely than those who do to agree that premarital sex can be morally acceptable. There is a college effect, yet it appears this may be slightly more prevalent among Catholics who attended a non-Catholic college.

In model 3, the excluded reference category is for those who attended non-Catholic colleges. Here the effect of not attending college is further illuminated. The coefficient is positive and statistically significant. Also note that now the coefficient for Catholic college enrollment is also positive (but does not achieve statistical significance). In the fourth and fifth models we introduce household income (something lacking in the "hook up" study) and we can see the a clear pattern emerge. There is a statistically significant and negative effect of having attended a non-Catholic college. The statistically significant negative effect from Model 2 is no longer apparent for Catholic colleges in Model 4 now that we have controlled for income (an important factor in the decision to enroll in college and in choice of a private college). Overall, the addition of the education variables did little to boost the adjusted R-square. Thus, these variables do little of the work explaining the variation in agreement and disagreement among Catholics on the issue of premarital sex.

It is also interesting to note that even among the most “conservative” Catholics a significant number agree that, “premarital sex between persons who are committed to each other can be morally acceptable.” For example, 44 percent of those who self-identify their party affiliation as a “strong Republican” agree “somewhat” or “strongly” with the statement as do 43 percent of those who self-identify their ideology as “extremely conservative” or “conservative.” Even one in five adult Catholics (19 percent) who attend Mass once a week or more and who self-identify as “extremely conservative” or “conservative” agree with the statement. 

In CARA’s recent Working Paper # 9 we were able to conduct further analysis with a similar question on a much larger sample of students in college. Some of these results are noted in the appendix of that paper as we did not include the question in the sample of measures that focused on political and social issues and religious behavior in the main text. The Higher Education Research Institute (HERI) question, given to a national sample of college students as freshman in 2004 and again when they were juniors in 2007 asked for their level of agreement with this statement: “If two people really like each other, it’s all right for them to have sex even if they’ve known each other for only a very short time.” Note that this statement encompasses features of all three of the questions analyzed above. It is specific to the sexual nature of the encounter and it includes a time frame that is more consistent with the notion of “hooking up.”

In a similar study using HERI data in 2001, 48 percent of those who had entered a Catholic college self-identifying as Catholic agreed with this statement just before their graduation (some were no longer Catholic at that time). The level of agreement for these students had grown more than 20 percentage points while in college (since their freshman year in 1997). It is important to note that 2001 is the same year in which the “hooking up” study was fielded.

By comparison in 2007, 46 percent of those who had entered a Catholic college self-identifying as Catholic agreed with this statement just before their graduation (the difference is not statistically significant from 2001). The level of agreement for these students had grown only 8 percentage points while in college.  Although both groups of students ended up at similar levels of agreement the key difference is that the juniors in 2007 already came into college with greater agreement with the statement in 2004 than the students entering college in 1997.

CARA analyzed this question using the same regression models used for all other questions in Working Paper #9. Similarly, we found that enrollment in Catholic colleges had no negative nor positive effect on the response to this question. Catholics at non-Catholic colleges and universities, controlling for factors associated with self-selection (i.e., no one is randomly assigned to college), moved similarly toward greater agreement that “it’s all right for them to have sex even if they’ve known each other for only a very short time.” These results are not consistent with the “hook up” study that claims evidence of a specific Catholic college effect among women (the HERI data regressions control for gender, which is also not a statistically significant factor).

It is unlikely that a replication of the 2001 survey in the “hooking up” article with a broader sample of Catholic women attending Catholic colleges would show any statistically significant Catholic institutional effect (i.e., attending a Catholic college compared to a non-Catholic college). A review of other data sources indicates there are undoubtedly changes among the wider Catholic population regarding the issue of premarital sex in general. However, these appear to be much larger than anything specific to what a Catholic college does or does not do. As the GSS and CARA results indicate young Catholics are often sent off to college from homes where the parents do not have attitudes regarding this issue that are consistent with Church teachings. 

Search This Blog

Loading...
© 2009-2013 CARA, Mark M. Gray. Background image courtesy of muohace_dc.