November 2000

The outcomes and impact of Expanded CTF and Community Work

Centre for Operational Research and Evaluation

1. Introduction
   This brief provides the initial findings from the review of the subsidised work appropriation on the outcomes and effectiveness of Community Work and its predecessor, Expanded Community TaskForce (CTF). These findings are still preliminary and subject to review. However, the results are considered robust and, more importantly, are consistent with the findings of previous evaluations of work experience programmes.

2. Background
   In March 1999 the Department of Work and Income (DWI) ¹ was instructed to justify the current level of expenditure on employment programmes within the subsidised work appropriation. In particular, the present review seeks to provide a comprehensive analysis of the impact that the programmes within the appropriation have on the achievement of positive labour market outcomes for participants and to determine whether they are cost-effective in doing so.

3. Methodology
   The question that the review seeks to address is whether the employment programmes within the subsidised work appropriation contributed to the positive outcomes of participants and were cost effective in achieving these outcomes. Determination of effectiveness in both these instances is based on the counter-factual question; that is, what outcomes and cost/benefits might have occurred if the job seeker had not gone onto the programme.

   It is generally recognised that a randomised control group assignment is the most robust method to estimate this counterfactual. However, this method is both expensive and difficult to implement effectively. For these reasons a quasi-experimental design was adopted, which involved the construction of a group of non-participants matched to each participant on the basis of known characteristics. The differences in the outcomes of the participants and the matched group are then used to estimate the impact of the programme. In addition to matching based on specific characteristics, multivariate analysis was also used to control for other factors that could account for any difference in outcomes between the two groups other than participation in the programme. Whilst, it is acknowledged that this method will not completely eliminate the possibility of selection bias² in the observed impacts, nevertheless, it does go a long way to minimise such biases.

   The outcomes of participants and comparison group are based on their benefit and register histories. Using this information, job seekers' labour market status is determined at regular intervals from the start of their placement on the programme or nominal start date for members of the comparison group.³ The possible labour market statuses include:

   1. Left the labour market
   2. Unemployed
   3. Participating in DWI programmes
   4. Training
   5. Part time employment
   6. Subsidised full time employment
   7. Unsubsidised full-time employment

   Of these, 5 through to 7 are considered a positive outcome resulting from participation in either Community Work or CTF.

4. Findings
   Table 1 & 2 below summarised the positive labour market outcomes of Community Work and CTF participants at 3 different time periods from the start of their placement.

Table 1: Proportion of positive labour market outcomes achieved by Community Work after their placement.

Miscellaneous outcomes are those for which the labour market status of the job seeker could not be determined, within the analysis half are counted as a positive outcome.

The indications are that the outcomes of participants in both programmes are relatively similar, with unsubsidised full-time employment the most likely positive outcome achieved. By way of comparison, 12 months after the start of a Job Plus participation, approximately half of participants are in unsubsidised full-time employment.

Table 2: Proportion of positive labour market outcomes achieved by Expanded Community TaskForce participants after the start of their placement.

Miscellaneous outcomes are those for which the labour market status of the job seeker could not be determined, within the analysis half are counted as a positive outcome.

Impact of Community Work and CTF
Whilst labour market outcomes provide part of the picture on the effectiveness of programmes, the key test is to see whether the outcomes of participants are better or worse than that of the comparison group. To a large extent the comparison group represents those job seekers receiving income support only with only a very small proportion participating in any DWI programme or training.⁴

Figure 1: Positive labour market outcomes of Community TaskForce participants and comparison group.

Figure 1 clearly shows that the outcomes between participants and non-participants track each other closely over the study period, and indicates that CTF has had no positive impact on participants' outcomes. This finding is consistent with an earlier outcome evaluation of CTF, which also concluded that the programme had no appreciable positive effect on outcomes (DWI, 1999). Furthermore, the multivariate analysis shows that the estimated effect of the programme on outcomes was significantly negative over the first 12 months, after which there was no significant difference in outcomes between participants and comparison group (Table 3).

Like its predecessor, the outcomes of Community Work participants did not exceed those of the comparison group. However, the lower outcomes of the participants persist for a longer period of time, with the outcomes of the two groups equalising after 22 months rather than 14 for CTF. The multivariate analysis indicates that Community Work had a negative impact over the study period, however, after 12 months the difference is not substantive (impact ratio of 0.99).

Figure 2: Positive labour market outcomes of Community Work participants and comparison group.

Table 3 summarises the findings of the multivariate analysis of the impact of Community Work and CTF, these confirm that the outcomes of participants were indeed lower than that of the comparison group when a range of demographic, socio-economic and labour market variables were controlled for. However, the estimated impact of participation in either programme ceases to be significant after 12 months from the start of the placement.

Table 3: Positive outcomes and impact estimate for Community Work and CTF

		Positive labour market outcomes				Impact Estimate
		<6 mths	7-12 mths	12+ mths		<6 mths	7-12 mths	12+ mths
Community TaskForce		7.0%	19.8%	32.0%		0.66	0.94	1.03
Community Work		6.6%	20.9%	30.7%		0.56	0.89	0.99

The impact estimate represents the ratio in outcomes between participants and comparison, a value of less than 1 indicates that the outcomes of the comparison group exceed those of the participants conversely a value over 1 shows that the participants achieved better outcomes.
Shaded cells show that the estimate is significant at the 95% confidence interval.

Conclusions
The findings reported here are broadly consistent with previous evaluations of work experience programmes both in New Zealand and overseas. The above patterns in outcomes and impact seem to indicate that these programmes have a 'locking in' effect on participants. In other words, it would appear that the probability of people achieving a positive outcome decreases whilst they are on the programme. Previous evaluations (NZES, 1998) have suggested that this is due to the perception by participants of their placement as 'work' and therefore they do not engage in job search activity for the duration of the placement.

It is of concern that this 'locking in effect' appears to be more pronounced for Community Work than for its predecessor CTF. It is suggested that this may be due to the removal of a maximum duration of placements (under CTF placement could last no more than 6 months). Another factor may be the introduction of a base allowance of 20 dollars a week for incidental costs, which can go up to 40 a week for actual and reasonable expenses. The review will undertake further analysis to determine whether these two factors do indeed decrease the probability of participants achieving a positive outcome. However, despite the 'locking in' of participants, this effect appears to be temporary, with the outcomes of participants and the comparison group equalising in the longer term.

1. Also known as Work and Income New Zealand
2. Selection bias refers to the possibility that differences in outcomes between participants and comparison group are due differences in the characteristics between the two rather than their participation in the programme.
3. Each comparison group member is matched to a participant; it is this participant's start date that is the nominal date for the comparison group member.
4. In the context of the current analysis, at any one time approximately 5% of the CTF & Community Work comparison groups were participating in DWI programmes or training (mainly TOPs).