Operated between

Based on intervention documentation in the first instance. If documentation is missing, then we use information on participation starts from 2000 onwards, if an intervention is recorded as starting in 2000, it may have been operating before this date. The operating period excludes trailing years where there is less than 10 participant starts or expenditure was less than $10,000.

Participation starts

Is the average for each financial year the intervention operated or since the year 2000. Participation includes any valid start recorded in the administrative system. Because people can participate in interventions more than once, this figure is not a count of unique participants. For case management services where people can stay on for multiple years, each participation spell is split by financial year. For example if a person remains on General Case Management for three years, then this is recorded as three participant starts, one for each financial year.

Expenditure

Is the average for each financial year the intervention operated. However, expenditure information is only available from the 2006/2007 financial year. Therefore, this value may not be an accurate reflection of the average expenditure for interventions operating before 2006/2007. Expenditure includes all costs to deliver the intervention, including indirect costs. Expenditure is expressed as nominal values (ie it has not been inflation adjusted).

Average cost

Simple average calculated by dividing total expenditure by total starts. Because financial data is available from the 2006/2007 financial year, this average is only for the years since 2006/2007 and is not available for interventions running prior to the 2006/2007 financial year. For case management services where people can participate for multiple years, the average cost is for all starts plus those who were in the service at the start of the year.

Operated between

Based on intervention documentation in the first instance. If documentation is missing, then we use information on participation starts from 2000 onwards, if an intervention is recorded as starting in 2000, it may have been operating before this date. The operating period excludes trailing years where there is less than 10 participant starts or expenditure was less than $10,000.

LLTBR

Likelihood of Long Term Benefit receipt is a statistical risk profiling tool developed to estimate the probability individuals on main benefit will remain on main benefit for the following two years (over 700 out of 730 days). The LLTBR is divided into decile bands that correspond to the probability of remaining on main benefit for two years.

LET score

LET is a statistical risk profiling tool developed to estimate the how much income support a person will receive until they reach the age of 65. LET is scored each week and is estimated for anyone who has been on main benefit. The LET is divided into decile bands from low expected future income support payments (01) to highest (10).

Circumstance

The person's circumstance profile is designed to show the situation a person receiving income support is in. For each person we categorise people into the following groups.

• Independent youth: young people who are no longer supported by their parents or care givers. We identify independent youth through people receiving a youth related benefit (Independent Youth Benefit, Youth Payment).
• Unemployed: people for who do not have child care or other caring responsibilities and are receiving a main benefit. These benefits include: Job Seeker, Student hardship, Emergency, Widow, or Women Alone.
• Sole parent: people with at least one child in their care and are single or receiving a sole parent related benefit.
• Parent : people on main benefit as a couple and have children irrespective of what benefit they are on with the exception of health and disability.
• Caring: people who are caring for someone sick and infirm.
• Ill health/disability: people receiving a benefit for reasons of ill health or disability.
• Retired: people who are no longer working and are receiving a retirement related main benefit (eg New Zealand Superannuation or Veteran's Pension).
• Off benefit: people who are not receiving a main benefit (but can be receiving supplementary assistance).

In cases where a person falls into more than one category, the categories are prioritised as follows.

1. ill health/disability
2. sole parent/ parent
3. caring / independent youth / retired / unemployed/ off benefit.

Participation starts

Is the average for each financial year the intervention operated or since the year 2000. Participation includes any valid start recorded in the administrative system. Because people can participate in interventions more than once, this figure is not a count of unique participants. For case management services where people can stay on for multiple years, each participation spell is split by financial year. For example if a person remains on General Case Management for three years, then this is recorded as three participant starts, one for each financial year.

Main benefit

In the New Zealand income support system main benefits are paid to people who are unable to work. We classify people's main benefit status as into the following categories. The reason we have not used official names is that these have changed over the analysis period, in particular before and after July 2013.

• Caring for Sick and Infirm: people on Supported Living Payment or Invalid's Benefit Caring for Sick and Infirm.
• Emergency maintenance: sole parent related benefits that do not meet the eligibility criteria for sole parent benefits.
• Emergency: people in need of financial assistance but not entitled to any non-sole parent related benefit.
• Invalids: Invalid's Benefit, replaced by the Support Living Payment. Note this excludes people on Caring for Sick and Infirm.
• Job Seeker: Unemployment related benefit and Job Seeker Work Ready from July 2013.
• Off benefit: people not receiving either a main benefit or supplementary assistance.
• Sickness: Sickness Benefit and from July 2013 the Jobseeker Support Benefit with Health Condition or Disability exemption.
• Sole Parent: includes the current Sole Parent Support as well as the previous Domestic Purposes Benefits. Note that sole parents with a youngest child over 14 after July 2013 receives Jobseeker Support.
• Student: people between semesters receive a specific unemployment related benefit (Jobseeker Support Student Hardship)
• Supplementary only: people who are receiving only second tier assistance such as Accommodation Supplement or Disability Allowance.
• Youth Payment: Independent Youth Benefit, replaced by the Youth Payment in 2012.
• Young Parent Payment: Young Parent Payment was introduced in 2012.
• Widow: Widow's Benefit and DPB Woman Alone Benefits were removed in July 2013

Outcome domains

The effectiveness of each EA intervention is assessed against a range of outcome domains. Depending on the intervention, we can measure the effectiveness on one or more of these domains. The impact rating for each outcome domain is based on the impact of the intervention on one outcome measure for each domain. Each outcome measure was selected because we believe it best represents the outcome domain across all interventions.

The outcome domains include:

• Income: intervention is rated as effective if it increases net income from all sources including income support, tax credits and study assistance. Therefore, this measure accounts for both any loss of income from government assistance, as well any earning gains through movement into employment.
• Employment: intervention is rated as effective if it increases the time spent in paid employment. This measure includes increases in employment while still on main benefit.
• Justice: intervention is rated as effective if it reduces the time participants spend in corrections services (ie prison, community service, remand, home detention).
• Qualifications: intervention is rated as effective if it increases the average of the highest qualification held. Highest qualification is based on NZQF levels from 1 through to 9. Therefore, an impact of NZQF 1 would mean participants had increased their highest NZQF level by an average of 1 level (eg from 2.4 to 3.4).
• Study: intervention is rated as effective if it increases the time participants spend enrolled in education or training. Note that being enrolled doesn't necessarily mean a person is attending the course (eg they may have dropped out).
• Welfare: intervention is rated as effective if it reduces the time participants receive a main benefit (eg unemployment, sole parent or health or disability benefits) and receiving employment assistance (eg on a wage subsidy programme).

Effectiveness

By effectiveness, we mean whether an intervention improves participant outcomes relative to the counterfactual (ie the outcomes participants would have had if they had not participated). In the current analysis, we assess effectiveness based on the impact of the intervention on one or more outcome domains. However, we cannot assess the effectiveness of all interventions and the reason for not doing so is also reflected in the ratings.

Our analysis of the impact of employment interventions focuses on five outcome domains. These domains broadly reflect the intended medium and long-term objectives of employment interventions. The rating shows the desired direction of impact. The ratings are described as follows:

• Effective: the intervention has a statistically significant positive impact on one or more primary outcome (eg income, employment, justice, qualifications or independence from welfare) and no evidence of a negative impact on any primary outcome.
• Promising: the trend in impacts indicates the intervention is expected to have a significant positive overall impact in the medium-to-long term. Also, we rate interventions as promising if we cannot evaluate the intervention directly, but where we have a similar intervention rated as effective.
• Mixed: the intervention has both positive and negative impacts on primary outcomes. The most common case is where an intervention increases employment but has a negative impact on welfare.
• No difference: the intervention makes no statistically significant difference for any of the primary outcomes. For smaller interventions, this may reflect insufficient statistical power to detect a meaningful impact and these impacts may become significant when an intervention has more participants.
• Likely negative: trends indicate the intervention will have a negative impact on one or more primary outcomes and there is no evidence of a positive impact on any other primary outcome in the medium to long-term.
• Negative: the intervention has a statistically significant negative impact on one or more primary outcomes and no evidence of a positive impact on any primary outcome.
• Too soon to rate: there has not been enough time to observe the impact of the intervention. Typically, we do not rate an intervention until we have two years of outcome data available.
• Not rated: we have not rated the effectiveness of the intervention at this time.
• Not feasible: it is not considered feasible to estimate the impact of an intervention based on current data or available methods (ie would require some type of randomised control trial to robustly identify what impact an intervention has).

Impact by outcome

• Outcome: name of outcome measure.
• Observed period: follow up period in years that observed impacts are measured over.
• Observed impact: cumulative change in outcome because of the intervention.
• Projected period: number of years that the projected impact was projected over.
• Projected impact: projected cumulative change in outcomes because of the intervention.

Impact sub groups

We have estimate the effectiveness of EA interventions for a small number of sub groups. These include:

• Age: age of the participant at the start of the intervention.
• Ethnic: ethnic identity of the participant current at the analysis run date (i.e. a person may have identified with a different ethnic group at the participation start date). In instances of more than one ethnicity, we have selected the first listed in the administrative system (the order of ethnicities may not necessary reflect that of the individual and may instead simply reflect administrative processes). The grouping of ethnicity is quite high level to ensure enough participants in each group.
  • Pasifika: people from the Pacific Island's excluding New Zealand Maori.
  • Maori: New Zealand Maori.
  • Pakeha: people who identify themselves as New Zealanders or European
  • Other: all other ethnic identities.
• Gender: identified gender at analysis run date, a person may have identified with a different gender at the participation start date. However, changes to gender are rare in the administrative data. Administrative systems record only two genders at the present time.
• Benefit: main benefit status just before people started the intervention. Main benefit is the first tier of income support payments in the New Zealand income support system. The reason for an offset is to avoid confounding of the main benefit status with participation. For example, for wage subsidy programmes participants will exit benefit as part of going on the programme.
  • No Benefit: Not on any main benefit
  • JS All: any unemployment related benefit.
  • JS FT: unemployment related benefit with full time work obligations.
  • JS HCD: unemployment related benefit with work obligation exemption for reason of health condition or disability. This category includes the old Sickness Benefit that are combined with the Unemployment Benefit under Jobseeker Support from July 2013 onward.
  • JS HCD PT: unemployment related benefit with work obligation exemption for reason of health condition or disability with part time work obligations. This category includes the old Sickness Benefit that are combined with the Unemployment Benefit under Jobseeker Support from July 2013 onward.
  • JS WR/YP: unemployment related benefits (Jobseeker Work ready, Unemployment Benefit, Unemployment Benefit Student) and youth related benefits (Independent Youth Benefit, Youth Payment). From July 2013 all sole parents with a youngest child 14 and over are on Jobseeker Support related benefit.
  • SPS: Sole Parent Support main benefit. This category also includes the previous Domestic Purposes related benefits that ended in July 2013. From July 2013 all sole parents with a youngest child 14 and over are on Jobseeker Support related benefit. Before July 2013, sole parents did not transfer to non-sole parent related benefit until their youngest child turned 18 and finished school.
  • SP All: All people on sole parent related benefits.
  • SP FT: Sole parents with full time work obligations (youngest child over 14). For consistency over time, this can include Domestic Purposes as well as Jobseeker Support Work Ready (sole parent).
  • SP PT: Sole parents with part time work obligations tied to the age of their youngest child. The cut off for part time obligations have changed over the last 20 years.
  • SP No: Sole parents with no work obligations tied to the age of their youngest child. The cut off for part time obligations have changed over the last 20 years.
  • SLP: Supported Living Payment is for people with long term health conditions or disabilities that prevent them from working more than 15 hours a week. This includes people on Invalid's Benefit before July 2013.
• LLTBR: Likelihood of Long Term Benefit Receipt (LLTBR) is a statistical risk prediction at just before participation start that predicts the likelihood a person will remain on benefit for the following two years (more than 700 out of 730 days). Low: is under 14%, Medium is 15 to 39% and High is 40% and over.