Evidence Centre seminar: September 2018

Unpacking the higher NEET rate for Māori and Pacific peoples - video transcript

Eyal Apatov - a Senior Analyst within the Oranga Tamariki Evidence Centre:

Thanks everybody for coming.

So this is work I've done while I still at MBIE, as you can recognise the background.

As Kelly said, I used data from the IDI and the results have been tested to pass all the confidentiality tests and another disclaimer, this was work that was mainly done for economists and people who work in the economics sector so my guess is that if you've been involved in the social sector and programmes that influence need(?), that look at youth outcomes will probably more confirm things you already knew as opposed to telling you something new, but, yeah it would be good to get that sort of feedback as well.  So it is mostly for a wide audience and looking at a range of factors as opposed to focusing on specific things like education or health, etc.

So shortly, the studies about that gap.  Over here we see the rate of youth, 15 to 24 that are not in employment, education or training.  This is a break down by ethnicity and you can see that there is a persistent gap between the NEET rate of Māori and Pacifica or pacific peoples compared to other ethnic groups.  Roughly about twice.  And that gap, even though it kind of closes down it is still constant and persistent.

So that is one fact.  We know that the NEET population is diverse but there are some characteristics that are more common, for example, early school leavers, early parenting, residing in certain areas, having family characteristics.  We know that these factors that generally predict NEET, NEET status, are also more common amongst Māori and Pacific youth.

So the next question is what's going to happen to that NEET rate gap if Māori and Pacifica had on average the same distribution across these characteristics if those gaps between risk factors were eliminated would we expect that NEET rate gap to be eliminated as well.

Another thing I looked at is are certain risk factors more important to explain that gap for some sub groups than for others.  To go into a bit more detail about what I was talking about, what happens if we eliminate the gap in risk factors and how that effects the NEET rate gap, we have here an example with two groups: group A and group B.  The horizontal axis shows the average hypothetical education level for both groups.  So the average for group A is level 6, the average for group B is level 2.  On the vertical axis we see the corresponding weekly pay.  For both groups the lines show the relationship between education level and learning.  So as I say, group A has on average level 6 qualifications and earn about $700, group A has level 2 qualification earns an average $250.  So there is a gap in earning and there is a gap in qualification level.

If we had a programme that pushed the qualification level of group B, because the relationship is more or less the same, the wage gap will be eliminated, or largely eliminated. 

But there could be a situation where it's not just the average qualification level is different, but their return to qualification.  How much extra do you earn when you increase your qualification level?  So in this sort of story when we increase the qualification level of group B from two to six, we do close that earning gap, but not all the way.  So it explains some of the gap, but not all of it. 

So we in MBIE wanted to know how much of the gap -- how much of the need gap can be closed if those differences and risk factors are closed.  What story do we have?  Is this just a story about differences in characteristics or are there other things?  So for example, the chart on the right is more of a story of the gender wage gap.  You take males, you take females, you equalise education and other background factors you explain about a fifth of the gap. 

If you like equations more than pictures this is the Greek letter version.  Basically what is says, it repeats the pictures.  Your likelihood of being a NEET depends on having certain risk factors, which is those Xs and other stuff, which is that (inaudible).  You look at each ethnicity in the study and you calculate that relationship and then if you take any two groups you can break down the gap in NEET rate into three factors, broad factors, highlighted in three different colours.  The first part is the that the two groups -- the gap is explained by the two groups having differences in risk factors.  So there was that horizontal axis.  Group A had level 6 education.  Group B had level 2 education and that is why we see a gap in earning. 

The second part is differences in returns to risk factors.  So there were the differences in slopes. 

The third part is a bit more tricky but the intuition is that -- in this study I'm not just going to look at education, I'm going to look at a lot of factors and what this term(?) asks is are the risk factors that have the biggest gap also have the biggest difference in slopes.  So that could tell, for example, some groups are more likely to reside in poor areas and if they reside in more poor areas they're also more likely to be NEET.  So the effects of things were we see the biggest gap is also more damaging, for example.

So I'm keeping with the official definition of NEET and looking at youth age 15 to 24, focusing on 2016 and I look at long term spells of NEET and that's because past studies found that this is a certain -- like the risk of experiencing adverse outcomes in future years is greater if you experience long terms spells of NEET as opposed to one week of being NEET.  So I'm comparing outcomes and NEET rates between ethnicities and I got four broad ethnic groups.  One is youth that identifies solely as Māori and it's termed Māori only.  Youth that identify as Māori and other ethnicities, termed as Māori and Pacific people, their label will be also Pacifica, I didn't have the time to change that.  

The control group, the group I'm comparing is everybody in my sample that didn't identify as Māori or Pacifica, some (inaudible) Māori Pacific people and (inaudible) call other.  And rather than looking at the entire ethnic group I'm breaking it down by sub age group and gender.  So this is based on administrative record from the IDI.  So I'm following the approach that McLauren Toman took from the Treasury to define it.  It is basically assigning activities to an individual in each month.  So I'm asking for each individual, have you been overseas in a month for 15 days or more, if not have you been in custody for 15 days or more, if not have you been enrolled to an educational course for a day or more in that month, if not, did you earn at least $10 from wages, salaries or self-employment, if the answer to all of these is no I'm saying you are a NEET in that month. 

To be long term NEET you need to have six of those in a row, consecutive months.  So my sample has almost 600,000 observations.  This is about 90% of the overall estimated population.  The biggest difference is I exclude temporary migrants. 

I collect information about the usual suspects, educational achievements, other personal details, family characteristics and area factors that have all in past studies been found to predict youth outcomes. 

Looking at the data, this chart will kind of look familiar to the first chart.  This is the long term NEET rate by ethnic group and just like the official rate it has been falling since 2012 and at a slightly faster rate for Māori and Pacifica but still there's a quite large gap between that and that of other, non-Māori and Pacifica. 

If I look at the break down by the 15 to 19 year old, just like in the official stats, there's not much difference across gender at this age group and as the overall group the highest rate is for Māori only followed by Māori Pacifica and other.  When I look at all the age groups you can see the NEET rate increase and the gap in terms of percentage point difference also increase, and that is similar to the official stats.  What is also similar to the official stats is the higher rate for females at this age group. 

The official stats allow you to look at types of NEETS and a lot of the difference between -- most of the difference between male and female is to do with caregiving duties.  Now I don't have access to caregiving duties, but what I've done is broken down the NEET rates by whether you have one or more children.  So on the left, this is the NEET rate for 20 - 24 year olds without children and on the right what is the NEET rate if you have at least one child. 

The rate for male increases, doubles, but the rate for females is over half.  And at the bottom, in percentages, that's the portion of all NEETs that are from that sub-group.  So 5% of all long term NEET are males with one or more child compared to almost one out of four.  So one out of four long term NEET is mothers.  So that is something that changed my perspective about long term NEET.  I kind of imagined street kids being all, you know, street kids, but actually it's mothers that is a big part of that story.

Other high level findings, so as in past studies I found that Māori and Pacifica have greater prevalences of the factors that predict they're going to be NEET.  So low decile -- attending low decile schools, parents with benefit dependency. 

On the other hand when I compare the slopes, how do these factors -- what is the likelihood of if you have this factor to become NEET.  The relationships weren't that different.  So if you think about the example, the slopes weren't that far apart.  There were some differences but generally if you're an early school leaver you're probably going to be NEET regardless of your ethnicity or ethnicity doesn't matter as much at this point. 

Just a quick point about (inaudible) basically I could see from the returns that having a driving licence of a bachelor degree reduces your likelihood of being a NEET.  If you're a mother as well there's a compound effect.  So more than the sum of the parts. 

So looking at the results of the composition now.  So I am going to go slowly because there is quite a lot in these tables.  So these tables look at the composition results between Māori, Māori only Pacifica and other.  My control group for those ages 15 to 19.  And there's a break down by male and female labelled M and F. 

The first row shows the difference in long term NEET rates.  So if you remember the bar charts, that is the gap between any two groups.  So the NEET rate for Māori only was around 13% males.  The one for other was 9.5 percentage points lower, so that 0.095. 

Then the total gap, the total NEET rate gap is broken down in to three factors.  Risk factors, observable risk factors, that's that horizontal axis, how much of it relates to that.  Returns, how much of it has to do with slopes, differences in slopes and the interaction affect. 

In brackets I'm showing how much of the gap is attributed to each factor.  So for Māori only male 15 to 19 93% of the gap is explained by differences in risk factors. 

If you just follow that first row you see it varies from 77 to 126%.  What it means when more of the 100% of the gap is explained, is that if, in this example, Pacifica had on average the same characteristics of non-Māori and Pacifica their NEET rate is going to be relatively lower. 

Returns too is important, but slightly less, between 23% and 32%.  And if we look at the 24 year old group this pattern holds. 

Risk, differences in risk factors, explain between 80% and 112% of the gap. 

Now I wouldn't necessarily that it's 80 and not 79 and not 82 but I would believe the relative importance, and as you can see the returns, differences in the slopes are even less significant for the older age group and even more so for interactions. 

If I look at specific factors I want to know which factors are important for each group.  Here I'm focusing on Māori only, on the NEET rate for Māori only males aged 20 to 24. 

The vertical line is the actual gap, almost 13 percentage points.  And now I'm asking if everything was the same but Māori only had the same share of youth that didn't receive any suspension and warning by age 16, what is going to be my new gap? 

So this chart shows that the gap would fall by about 1 percentage point.  There would be a slightly larger effect if they are the same share of parents that receive benefit.  Slightly bigger effect for holding a current driving licence and a bigger effect if the distribution across highest qualification was the same between Māori only and other. 

If you add all the factors you can see the gap almost eliminated, just over one percentage point.

If we look at Pacifica females, we look down at the 1, 2, 3 -- the first four factors are the same as from the previous slide with relatively similar effects.  But the effect of having children, differences in the proportion of Pacifica females with children explains about half the gap.  Much bigger than qualification or driving licence provision.  If we add all the factors we see that we are just left the zero meaning that the gap will reverse because Pacifica are expected to have a lower long term NEET rate compared to non-Māori and Pacifica.

So what do we do with all this?  This is good news, I mean, I the rate is falling and what explains the gap is things that we can see and if we can see them we can target them.  It is a better story than only 10% of the gap is explained by things we can see. 

Even though there is repetition across risk factors there is some variation across sub-groups.  So the biggest difference was parenting, much larger effect on females compared to males.  Other effects were area level effects.  Area deprivation was much stronger for 20 to 24 year olds, was stronger for Māori.  Qualifications were important for the older age group and driving licence seems to have a stronger -- explaining a stronger portion of the gap for males.

But overall it says it's supporting interventions that promotes school performance and provision of driving licence have the potential to substantially reduce the NEET rate gap. 

What is less clear, there is some area level and parental level -- family level effects that were as strong as qualification or driving licence.  What does that mean in terms of policy intervention?  I can't think of any direct response, but it does highlight risk groups.  Groups we can look at further and better understand.  So, for example, if we think about the -- well, there's a lot of Māori residing in highly deprived areas, but Māori that reside in these areas are also more likely to be long term NEET.  Why?  Is the experience of being a NEET in a highly deprived area the same as a non-deprived area?  What is the difference between being NEET or non-NEET in a highly deprived area.  Why having parents that are currently on benefit explain your likelihood of being NEET as much as your highest qualification?  

And just a final thought is that a lot of the stories about motherhood and my thought was the IDI is usually based on individual level unit of analysis and I was thinking, "Is this the right level?"  So if we had a policy and it would cut the rate for mothers by half, is this a success story necessarily?  What would be the outcomes for dependant own children, females doing caregiving duties not for their children, for other dependants?  So for me it was like a wake up call to think about beyond the individual level, to also consider family level analysis. 

And I'll just leave you with that. 

Thank you very much.

End of transcript.

Estimating the impact of Social Workers in Schools - video transcript

Moira Wilson - Principal Analyst, Resarch and Evaluation, Ministry of Social Development:

Great to be here today to talk about some of the work that's been going on and is coming up around the SWiS service.  The presentation Min and I have for you nests quite well with AL's and that SWiS is one of the preventive services for children that's intended to help support school engagement and retention.

I need to open with the same disclaimer as AL in broad terms. So the results I'm going to talk about today are not official statistics, access to the de-identified data used was provided by Stats New Zealand in accordance with security and confidentiality provisions in Stats Act and results have been confidentialised. And I also need to acknowledge our co-authors. The work was a lovely collaboration with Dean Heslop from (inaudible) Professor Michael Belgrave from Massey University and Pete McMillan. We also had a lot of help along the way and advice from others and I'd like to gratefully acknowledge that too. It's useful to briefly start with the context for our study. When we started in 2016 government agencies were facing increased demand for evidence that investment in social services was making a positive difference for the families that were being served and the children and the communities and at the same time the Stats New Zealand IDI was opening up new opportunities for impact evaluation that could potentially provide some of that evidence.  The data available relating to children in particular was being greatly expanded at that time. Those data were opening up new opportunities but also posing some challenges. The admin data and the IDI missed many of the important outcomes that are sought by social programmes, where measures of that are available they might provide imperfect proxies for the outcomes that are sought or changes over time might have more to do with changes in administrative processes or service access than real change in outcomes for children and their families.  And they've got the potential for telling a biased story about the prevalence of the outcome we're concerned with, depending on who has service access or who's targeted for a service.  They have important sources of error(?) and even when good measures are available there's only some cases in which robust impact evaluation as possible.  So it's important to view these data as just one possible part of the evaluative story and to take quite a lot of care in interpreting what we see happening in the data.

It was what that context that we went into the IDI to examine the impact of SWiS on children's outcomes.   SWiS is a government funded community social work service available in low decile primary and intermediate schools and kura kaupapa Māori services are provided by social workers employed by contracted social service providers and what the social workers provide as individual casework for children and their families and whanau, group programs.  There's been a reduction in funding for broke group programs over time but those are still part of the mix and community liaison and service coordination for the children and the whanau.  Participation in the service by the children and their families and whānau is voluntary. The evolution of the service is really interesting and we were very fortunate to have Michael Belgrave as part of the team. He was closely involved in the early development of SWiS and in our report, which is available through the Oranga Tamariki and MSD websites, Michael provides a historical overview which traces the origins of the service back to the calls that were being made by iwi in the 1980s for government to resource Māori to provide services to Māori and efforts in the late 1990s to improve the coordination of services for families and whānau under Strengthening Families. So SWiS was part of a new set of interventions contributing to strengthening families which included Family Start and tangentially resource teachers of learning and behaviour.

Schools welcomed the service as a result of the upheavals and the economic circumstances of families that were happening around that time. They felt they had few other options to address what they saw as ever more complex needs of some of their students. After its introduction the service was rapidly expanded and there were a few periods of expansion. The latest expansion was in 2012/13 which took the service into all schools and kura that were decile 1 to 3 at the time and took the number of schools and kura served to 700.  

Most of the children in these schools are Māori and Pacific or Māori or Pacific. While the service is a high intensity one at the individual level its low intensity at the school level so each social worker works with around 16 children at any time. But it's working in a cluster of schools or a school with a large role that might be around 400 to 700 students.  So not a lot of resource is going into schools relative to the needs that might exist in some cases.

Just to quickly take you through some of the past studies that had been done at the time that we started our study. There was some really rich and indepth evaluations done in the early 2000s which found strong support for the service and were really important in shaping its early development.  Kaupapa Māori studies by Rachel Selby, Athena Hollis English and Hayley Bell undertaken some years ago now have affirmed the value of SWIS as a part of a package of programmes delivered by Māori providers.  Māori providers and Māori social workers were able to deliver the service with in a kaupapa Māori framework and that kind of spread to mainstream provision of services as well through a very strong network of Māori social workers and social work conferences.

So SWiS was an important service in broadening out the role of social work in New Zealand.  All of the earlier studies found that many clients and families had deeply held negative views about social workers and that these attitudes could be turned around by good quality and responsive engagement by the school social workers in a voluntary context.  In terms of quantitative estimates of the difference the programme makes there was a study in 2015 which provided some encouraging results. But that study had some limitations and further work was recommended.

So what did our study do?  It estimated the impact of that very large expansion in 2012 on selected outcomes using the IDI. We examined whether there were indications of improvements and outcomes for students by looking at whether they were less likely to have stand downs or suspensions from school, care and protection notifications to what was then Child, Youth and Family and police apprehensions for alleged offending. We used a difference and differences estimation approach and here's the basic outline of that in words. We're looking at differences over time for students in schools that were affected and looking to see if those differences were different to the differences for students in comparison schools over the same period.

And if you like pictures more than equations this one's for you. What we did was compare students in three groups of schools. Schools that were already served by SWIS before the expansion, these were mainly decile 1 3 schools and interestingly they were more likely to be decile 1 to 2 schools than the newly served schools. Our main group of interest were the decile1 to 3 schools that were newly served and we also used as a comparison group decile 4 to 5 schools that had never received SWIS. We were looking to see if there was a reduction in the rates of the adverse outcomes that we were looking at following the introduction of SWIS that occurred in the newly served schools but not the comparison schools.  We would expect students who were the direct recipients of individual casework to have shown the largest difference in outcomes if there was an effect on outcomes. And data from the early implementation of the service had suggested that boys and Māori students, for example, were more likely than average to receive the individual casework.  

So we examined the possibility of these differential effects by interacting the variable that was intended to capture the effects of this SWiS expansion, if it was there, with observable characteristics like being Māori, like being a boy and other characteristics that might proxy for a greater likelihood of being a direct recipient.

We also use that same interaction approach to look to see if the effect was greater in schools that were the base schools for the social workers, where they might have had a greater presence in the daily lives of the students and in kura kaupapa Māori compared with mainstream schools.

We spent a bit of time trying to understand what was going on in terms of services and schools that might have also influenced student outcomes at about around the same time as the expansion. If another programme with similar aims was expanding with a similar patterning across schools then this would be important to know.

We concluded that the expansion of SWiS was the biggest shift in services and schools at that time, but you'll see here that a lot is happening in the school environment over this period and we were only able to capture data at the school level for a subset of the programmes that were expanding.  So we had health promoting schools, milk for schools and a number of other initiatives also ramping up over this time.  These ones on this graphic other ones that we were able to take account of in the modelling.

So what where are estimated impacts for subgroups of students who would be the most likely to be the direct recipients of individual casework provided by the social workers. We found a general pattern of improvements in outcomes in the schools including newly served relative to outcomes for students and the comparison schools. Sixteen of the twenty one interaction effects that we looked at were negative so that's suggesting an improvement in outcomes and five of these were significant at the 1 or 5 per cent level.

So if that the expansion had had no effect we would might have expected ten or eleven of the effects to be negative and we might have expected one of the effects to be statistically significant by chance.

Relative to trends the similar students in comparison schools and kura for Māori boys enrolled in the base schools with the social workers the SWIS expansion was associated with lower rates of police apprehensions for alleged offending and for Pacific students there were lower rates of care or protection notifications to CYF.  So although these results weren't strong statistically we saw them as really encouraging.

So those were results for the subgroups we thought would be the most likely to be the direct recipients of SWIS.  When we looked at all the children on the role without considering the subgroup effects we found no statistically significant evidence that the SWIS expansion improved outcomes for students overall.

We did see a tiny shift immediately the year that the expansion happened and notifications to Child, Youth and Family but that was very short lived.  So there may have been a sort of an anticipatory effect, "Here comes the social worker. We won't need to refer to CYF" then but that was quite short lived.

Before wrapping up I just want to draw your attention to some really interesting results for students enrolled in kura kaupapa Māori. We found no difference in the impact of the SWIS expansion even with those subgroup effects, comparing kura and mainstream schools, but we found associations between enrolling in a kura and large improvements in each of the outcomes we examined and these were all highly significant.  After controlling for other observed differences between students enrolled in the two groups of schools, including some measures that we thought might pick up early adversities and early socio economic disadvantage in the lives of the students.  So these results really clearly invite some further study if you know anyone who's interested in a project, we're really willing to share where we got to as the basis for further study.

It's useful just to circle back round to some of the limitations of this kind of study. Although the large scale expansion of SWIS provided a good opportunity for quasi experimental impact evaluation in this case, the complexity of the school service environment and the low intensity of SWIS at the school level made in estimating impacts difficult, given the wide variation in student needs and in the services that would have been provided in response.  We probably wouldn't have expected large effects on single outcomes in any case.  And again the administrative data has limitations. It did not enable us to look at a range of outcomes for which strong perceptions of positive change in the lives of students and their whānau have been reported in other studies. So when the research that Athena Hollis English and colleagues did, among the indicators that Māori social workers they interviewed were looking for when they were thinking about whether the whānau they were working with were seeing improvements where things like living more positively as a cohesive unit and a willingness to engage in activities both within the school and after school.

Our study could never have got at any of that sort of granular improvement. The other the other thing to say is that in some of the limited overseas literature its impacts on these sorts of outcomes, these kind of less tangible, less academic, less traditional school outcomes that seem to have been the ones that have been moved by these sorts of caseworkers in schools.

So against these limitations - I probably should have called this "strengths and limitations" - against these limitations the strengths of linked data, linked administrative data, is it offers the ability to examine outcomes across domains that we've traditionally looked at in silo.  In our case education, child welfare and justice.  It also offers a long and growing longitudinal data source unaffected by non-response bias.  A large, and in our case, comprehensive sample of the populations of interest allowing us to look at some relatively rare outcomes, so stand downs and suspensions and police apprehensions for offending, for example, for the age group of children that we were looking at are extremely rare.  However, it can only tell us part of the evaluative story and it's important to maintain the flow of qualitative insights as the context in which SWIS operates changes.  For example, all of the qualitative studies to date predate the emergence of Whānau Ora as a programme.

So to conclude, based on the past studies SWiS offers a preventive social works service that's acceptable to families and whānau and experienced as helpful by schools and kura and is seen as having a wide range of important benefits.  In this study there were indications of some encouraging impacts on outcomes that could be measured using linked administrative data for the students most likely to have been the direct recipients of the service.  I'll hand over to Min.

(Min Vette - Team Leader, Services in Schools, Oranga Tamariki)

Ā tēnā tātou.  Tēnā koe, Moia, mo tō kōrero.  So thank you very much for getting us to this point.  I've got five minutes to finish us off.  So, with all those linked data, you know, we still ask the question, so there are some encouraging results there.  And so the question is so how does this happen and what's missing in this data that tells us the story.  Something that we can do something about.  

So what we want to do is to be able to do another study.  We've been asked by prime ministers to add another study to this.  So we're going to go ahead and do a qualitative study.  So there's a new SWIS evaluation that we're starting to get underway and this one is going to look at implementing (inaudible) methodology right from its beginning. The reason for this is because of the encouraging data that we have through the IDI and the majority of Māori clients who are accessing the service.  And wanting to ensure that we tell those stories in the way that they are, in terms of Māori stories of them as Māori. And we want to ensure that the methodology used for that can ensure that we get those stories through as best as we can.

So (inaudible) methodology is something that we've used before.  And it did show some slightly different findings.  And that's the Family Start quasi if you're willing to have a look at that.  It's a method that was developed by Angus McFarland in 2011 as a model for reconciling the western science and teo Māori research and evaluation perspectives.

So I wanted to just for those who are unfamiliar with this method, the analogy used or the metaphor that's used is the braided rivers of the South Island.  Really important rivers for South Islanders. You only get these kind of rivers down the south, we don't have up in the north here.

But I was told by a Ngai Tahu elder about these braided rivers and that they have different streams of water coming through them. One stream comes from the alpines as the snow melts, and so it was once a solid and the other's from the rain it hits the land and then soaks down through into the tributaries and those two streams of water come together.  And chemically they don't merge very well but they do merge when they come into to the awa and then as they flow on they will naturally separate themselves. And then as they hit more tributaries it will merge again. So that's the kind of metaphor or analogy that we'll be using for our (inaudible) methodology is to bring the teo Māori and the Western theory through together, at stages they will merge and then they'll separate and be their own.

So we feel that we will get a perspective that is a joint perspective that will ensure that both of those views can be represented. So in that as well we'll be drawing on other Māori programmes or services that have an effect for families and children as well. And one of those Whānau Ora.  So we'll be looking at how that connects with SWIS. And also looking at international indigenous social workers in schools, they're all over the world.  So these indigenous ones in Toronto, as an example, and Australia. And so bringing those international indigenous services and having a look at some of that alongside what we have here.

This methodology will allow us to bring that through in another way as well.  So this research that we've got underway, it hasn't quite got underway, we're still in the stage of confirming who will be doing it, but it is to follow this quasi IDI evaluation that we've just done and it will align that with a qualitative, so you'll have the quantitative and qualitative sitting together.

So to understand the perspectives of whānau and tamariki. So this is specifically going to get the voices of the children and the families that use the service, that have access to, what it was for them, how they felt about it through what they did and also through their social workers.

So really excited about that, about hearing from the tamariki.  We have heard their voice once before in the research that Athena and Rachel Selby did.  I want to revisit that SWiS model, what it looks like and how it works for schools and kura and ensuring that we capture the view of the Māori tamariki whānau using that kaupapa Maori research. We will be doing quite a lot of work and interviews within kura kaupapa so we want to be able to ensure that that approach is also appropriate for that environment.

And using cultural appropriate responsive methods that we do.  So that's the evaluation going forward, a new one which we're really excited about and we have some good support there for that.  He anoi ra tātou.  That was my little bit.  I'm going to hand over to Davina.

(Davina Jones - Evidence Centre, Oranga Tamariki)  

Well I don't really have a lot to add to what Min said because I think the case has already been made for why we need to continue and move forward. The qualitative studies that we have had, while valuable, are a little out of date and when we had the quasi experimental design ministers found that really valuable. But they also wanted to unpack the black box to know how and why, as Min has said, those results were achieved and to understand how people were experiencing the social workers in schools programme on the ground in both by Māori for Māori settings and mainstream settings.

So really I have one further slide that just talks about all the different sort of methods that we're considering at the moment. We haven't fully landed on exactly what the whole study will look like, but as Min said they'll be both by Māori for Māori streams that we look to dig into and unpack and examine as well as looking at how the model is working generally. So we're going to have a literature review there's going to be a survey and of course we want to carry out interviews to actually hear from people themselves about their experiences and look at SDQ scores as well which is a validated instrument. The strengths and difficulties questionnaire that's also used in the before school check.

So that's a very brief overview, but as I said, we're just shaping that study at the moment with potential partners.

End of transcript.