The Surveys and the Analysis

Abstract

This chapter introduces the 2001 and 2013 Church Times surveys and discusses the advantages and disadvantages of using the readers of this newspaper to survey opinions in the Church of England. It then compares the demographic profiles of the laity and clergy in the survey samples at the two points in time. In both groups, there was a marked increase in the age profile between the surveys, which is in line with what we know from other studies and official statistics. On average, the Church of England is getting older and this emerges in the survey data, even given the relatively short span of time between the two surveys. Appendix 2.1 describes in more detail the analytical method applied in Chapters 3–10 and explains how to interpret the graphical reporting of the results.

Appendices

Appendix 2.1

This appendix is aimed at readers who may be relatively unfamiliar with the methods used to assess attitudes and analyse survey data. It also explains in detail how to interpret the graphs used throughout the rest of the book.

Comparing Attitudes Between Groups and Between Surveys

One of the best ways of comparing attitudes or beliefs in large samples such as the one used here is to employ Likert items, named after Rensis Likert, an American social scientist who first developed them in the 1930s. Likert items are statements that project a certain attitude or belief: ‘Hell exists ’, ‘The Church should do more for young people’, ‘Abortion is wrong’ and so on. Respondents are asked to indicate their agreement or disagreement with the statement on a scale which has a number of different responses. The Church Times had a five-point scale: ‘Agree Strongly’, ‘Agree’, ‘Not Certain’, ‘Disagree’ and ‘Disagree Strongly’. People were asked to circle one for each statement. Psychologists often code the answers numerically and combine answers to produce so-called summated rating scales that give a score for a particular general attitude (see, e.g., Coolican, 2014; Oppenheim, 2008; Vogel & Wanke, 2016). In this study, the responses to individual items are used, and to make reporting easier, they have been recoded so that the percentage of people endorsing a statement (i.e. answering either ‘Strongly Agree’ or ‘Agree’) is the main result used throughout.

The variations pointed out in this chapter between the profiles of clergy and laity and between the 2001 and 2013 surveys have consequences for how we go about measuring differences in opinions between various groups or over time. If we want to compare clergy with laity, for example, we could take a simple overall average score for a given item across both surveys. We might then assume that any difference we find reflects the consequence of ordination on attitudes. However, this does not allow for the fact that laity are mainly women and clergy are mainly men, or for the fact that there were relatively more clergy in the second survey than in the first one. There is a danger that what appears to be a difference between clergy and lay people is actually variation related to sex or time, rather than to ordination. In cross-sectional studies of this nature (where we sample a slice of the population at a particular time), this sort of problem is quite common. Even if we had a random sample of lay people and a random sample of clergy, there would be systematic differences in the samples in terms of age, education and gender because in a random sample we would find more men than women (because male clergy outnumber female clergy) and the men would probably be older, on average, than the women (because there are many more retired male clergy than retired female clergy). However, we can overcome these problems mathematically by using multiple regression techniques that test differences between groups while holding all other factors equal. Although the computations are complex when there are many different groups, they are easy with modern computers and the method allows us to compare scores for different groups after controlling for some of the other factors that were measured in the survey.

What is tested is how likely or unlikely it is that the difference observed in a sample represents a difference present in the whole population. For this study, the population being sampled is the whole of the Church Times readership . So the test is whether or not the overall difference in average attitude scores between groups (say men and women) is a real difference or simply a random effect. If the probability of it being just a random effect is less than 1 in 20, we assume it is a genuine difference, and this is what is meant by ‘statistically significant’. We can attach different levels of confidence to the test: if we find a 1 in 100 (1%) probability rather than a 1 in 20 (5%), we can be five times more confident that any difference we observe is a genuine difference. There are two things worth bearing in mind when looking at the figures reporting estimated means for different groups (such as Fig. 3.1). The first is that the scale of horizontal axis, which shows the percentage agreement with the item, varies between items. This is because agreement varied considerably between items and using the same scale each time can obscure what is going on. The downside of not using full 100% scale throughout is that differences can appear large when they are only a few per cent. For this reason, the graphs also have the actual percentage value by the bars. The second thing to bear in mind is that the sample sizes are large, so even small differences are likely to be statistically significant. Statistical significance shows how likely it is that the difference is due to chance, rather than necessarily the size of the difference. Probabilities are shown on the graphs, and those reported as p  < .05 usually refer to differences that are statistically significant but mostly trivial in size.

The differences between groups (men versus women, laity versus clergy, church traditions and so on) were tested on the total returns for both surveys (10,684) using the generalized linear modelling procedures in the statistical software package SPSS (IBM_SPSS, 2013).⁶ The model was set up to test the differences between given groups assuming all other factors were equal. So, for example, there might be a difference in the level of agreement with a test item between men and women overall, but if we allow statistically for the difference between the sexes in average age, education level and whether or not they were ordained, we might find that difference disappears. The different overall scores for men and women in this case could be explained by factors other than simply sex. However, if the difference persists after adding the statistical controls, it might be because men and women have different opinions, irrespective of their background and experience. It could also be that the difference is still due to other factors that were not measured in the survey but which were related to sex (e.g., opinion might be driven by personality, and men and women might have different personality profiles). Measuring as many relevant factors as possible and controlling for their effects in the analysis is the best way of identifying those factors which are most fundamental in shaping opinions.

To examine changes between surveys, it was clear that each tradition needed to be treated separately because patterns of change sometimes varied markedly between them. The underlying question was whether people born in a particular decade were retaining their attitudes between surveys, or whether attitudes were changing within as well as between cohorts. The best way to explore this was to plot average scores for each attitude by church tradition , by birth cohort and by survey. These graphs indicate if attitudes vary between those who were Anglo-catholics , Broad church or Evangelicals , whether younger people differed between older people, and whether attitudes shifted between 2001 and 2013 within people born around the same time.

In longer-term studies, it is possible to plot attitudes of birth cohorts throughout their lives, and this has been done for some attitudes in the British Social Attitudes Survey (Park & Rhead, 2013). In many cases, generations retain attitudes through life, suggesting that long-term attitudinal shifts occur because attitudes change from one generation to the next. However, it could also be possible that both processes are at work: each succeeding generation tends to be more liberal (or conservative), and each generation changes as it ages. In the graphs used in this book, difference between cohorts is indicated by the slopes of the lines, while difference within cohorts over time is indicated by difference between the lines for each survey. However, the separation of points for samples of the same cohort (i.e. for people born in the 1970s, 1960s and so on) could be due to the fact that the cohorts aged a decade between the surveys, and not because opinions changed for other reasons. To ensure that the graphs indicated genuine shifts in cohort opinions, rather than the changes we might expect as the cohort aged, the points for the second survey were corrected to allow for age effects .⁷ In this way, we can say with more confidence that a difference in the average score of the same cohort between surveys represents a genuine shift of opinion.

To illustrate the effects of correction, Fig. 2.3 shows the percentage of Anglo-catholics who agreed that they could influence their church’s decision-making. This is something that seems to vary with age, increasing from early life, peaking in midlife and declining thereafter. This pattern is evident in both the 2001 (solid circles) and the 2013 (open circles) surveys. This means, for example, that people born in the 1970s, who were in their 20s–30s when the first survey took place, would be expected to increase in agreement somewhat in the second survey, when they were in their 40s. What seems to be higher agreement for this cohort in the second survey might simply be the normal effect of ageing. Similarly, the 1930s cohort, who were in their 60s–70s in 2001, might be expected to have lower scores in 2013 because they were then in their 80s.

Fig. 2.3

Example of a cohort graph uncorrected for age-related changes

When we correct the 2013 survey for the average age-related changes across both surveys, we have a better picture of sifts in cohort opinion over the decade. The corrected results are given for the 1920s–1970s cohorts, because these are ones where the same cohort was measured in both surveys. Figure 2.4 suggests opinion was pretty much as expected from the age-related change, with perhaps some increase in the percentage over and above simple ageing for those born in the 1950s and 1960s. This adjustment makes relatively little difference, unless there is a marked age effect, as is the case here.

Fig. 2.4

Example of a cohort graph corrected for age-related changes

The second example (Fig. 2.5) shows the percentage of Anglo-catholics who supported the ordination of practicing homosexuals as priests in 2001 (solid circles) and 2013 (open circles), according to the decade in which they were born and after correcting for age effects . In this case, we see that in both surveys, younger people (born in the 1960s or 1970s) were more supportive of this idea than people born earlier in the century: the slope of both lines is upwards. However, if we look within cohorts born in any particular decade, opinion became more supportive of this idea between the 2001 and 2013 survey, even after allowing for the effects ageing on each cohort. So the shift in opinion in this case is occurring both because of generational change and because of changes within generations over time.

Fig. 2.5

Explanation of a cohort graph corrected for age-related changes

Appendix 2.2

Profiles of the two surveys for laity and clergy

		Laity			Clergy
		2001	2013	Total	2001	2013	Total
	N =	5239	2222	7461	1750	1473	3223
		%	%	%	%	%	%
Sex	Men	43	49	45	82	74	78
	Women	57	51	55	18	26	22
Age group	<50	19	13	17	28	13	21
	50s and 60s	53	42	50	50	53	51
	70+	28	46	33	23	35	28
Marital status	Single	18	20	18	16	13	14
	Married	63	55	61	72	70	71
	Living with partner	1	2	1	1	1	1
	Widowed	11	16	13	4	7	5
	Separated	1	1	1	1	1	1
	Divorced	4	4	4	3	4	3
	Divorced and remarried	2	2	2	3	4	3
	Civil partnership	0	1	0	0	2	0
Location	Rural	37	38	37	37	38	38
	Suburban	26	35	29	29	33	31
	Urban	37	27	34	34	29	32
Education	No degree	36	41	38	11	23	17
	Degree	64	59	62	89	77	83
Work status	In full-time work	25	18	22	54	38	47
	In part-time work	14	12	13	7	9	8
	Homemaker	8	4	7	2	1	1
	Unemployed	1	1	1	1	1	1
	Student	1	3	2	0	1	1
	Retired	51	64	55	37	50	43
Type of work	Manual	4	4	4	1	1	1
	Non-manual	26	22	25	3	5	4
	Professional	70	74	71	96	94	95
Household	<£20k	39	24	34	49	15	33
income	£20–39k	42	43	42	39	58	48
	>£39k	19	33	23	12	27	19
Internet access	None	45	15	36	34	6	21
	Work and/or home	55	85	64	67	94	79
Tradition	Anglo-catholic	44	40	43	48	44	46
	Broad church	40	48	42	32	39	35
	Evangelical	16	12	15	20	17	19