Evidence Centre Seminar: June 2024

VALMAI COPELAND: Tēnā koutou katoa. Welcome to this Evidence Centre seminar. I will start with the introductions and then we will move to the main body of our talk. This seminar – as you are all aware, I'm sure – is on the evaluation of the Transition Support Service, which research evaluation provider, Malatest, and myself have been working on for some years now. Ko Valmai Copeland toku ingoa. I was the evaluation lead from the Oranga Tamariki side for the evaluation, and I will be chairing this session. We have got 2 speakers lined up: Debbie who will be doing the major talk, and Sarah Ashton from Oranga Tamariki. I will now open with a karakia.

Kia hora te marino

Kia whakapapa pounamu te moana

Hei huarahi mā tatou i te rangi nei

Aroha atu

Aroha mai

Tātou i ā tatou katoa

Hui ē!

Tāiki ē!

This is an exciting moment for me because it is the culmination of over 5 years of work on the evaluation and the completion of a project that started for some of us in 2018 with developing an understanding of the population of young people leaving care from the ages of 15 onwards. It builds on work that was done for the Expert Advisory Panel into care, which resulted in the Transition Support Service.

Debbie McLeod, Director of Malatest International – our evaluation partner for TSS – will speak first. Debbie is well known in evaluation circles, having led a number of substantial evaluation and research projects, particularly in the health and social sectors area. Sarah Ashton, who is General Manager of our Transition Support Service, will follow from Debbie with an overview of how Oranga Tamariki has responded to and used the evaluation findings. Sarah will also supply us with the details of what actually happens in delivering the service and will pop in to enlarge on that at some points. Sarah worked with the Expert Advisory Panel in 2015 to identify the need for a better support for young people leaving care. She joined CYFS, soon to be Oranga Tamariki, in 2016 to work on the design and development of the service and, as I said earlier, she is now the General Manager in that area.

I will now hand over to Debbie McLeod to do our presentation. Thank you, Debbie.

DR DEBBIE MCLEOD: Thank you, kia ora. Thank you for the introduction and thank you for the opportunity to be part of this evaluation. I also just want to acknowledge everyone that has come along. I think this has very much been a partnership evaluation and I feel honoured to be able to share the voices of the young people and the kaupapa partners for Oranga Tamariki in presenting back the findings of the evaluation. This presentation is a synthesis of the final evaluation findings. It is a very much a synthesis and it sits alongside all of the separate evaluation reports, which can be found on the Oranga Tamariki website.

What I wanted to do is firstly just talk about what is the Transition Support Service because I'm aware that in the group here the details of what is provided might not be fully known, and that is where I will draw on Sarah's expert help. How is transition support delivered, what difference it is making and how well transition support is working. Then to talk a little bit about what we have learnt for the future delivery of transition support.

So firstly, what are the transition support services? They're a range of services for young people aged between 15 and 25 to support their transition from care and custody into adult life. It is now established as a core Oranga Tamariki and has grown year on year since its implementation on 1 July 2019. As Valmai said, the design of the service was based on feedback from young people and others, and a lot of work went into thinking about what was most needed. Young people are eligible if they have been in care or custody for 3 months continuously. That is quite simple criteria now. When the service first started, there were some more complex criteria but now it is very clear cut. To the year ended June 2023 – which is the timeframe that we will be talking about the evaluation over – 61% of eligible young people were referred to a transition worker. That is quite a sizeable number of young people: 1732.

The aims set out in the service design were that the Transition Support Service aims to support more young people basically to have safe and stable living arrangements, to be healthy and recovering from trauma, to have a trusted adult in their lives and to be engaged with family, cultural and community groups, to have the life skills they need to thrive as adults, and to be in education, employment, training and volunteering. So just bear those aims in mind as they were the objectives that we evaluated the service against.

The foundation of transition support are the Transition Support Service partners. To the end of the evaluation period there are 70 partners and 137 – nearly 138 – full-time equivalent transition workers who were the on-the-ground, major face of support for young people. On this map, the black dots show the scattering of the location of the partners. So the location of the head office addresses, so there are outreach services from some of those partners, for example, the west coast of the South Island is covered even though there's no black dots there.

Sarah, would you like to talk through the process?

SARAH ASHTON: Sure. As Debbie was saying, our transition partners are the foundation of the service so although there are a range of different supports that young people can access the transition workers based in the community are the largest part of the service. It was really clear from young people during the service design that they wanted that ongoing support to not be from Oranga Tamariki. So we have a job in facilitating that but it's really important, and it's proven through the evaluation, that the value of that support is it being based in their community.

It starts with young people being offered a referral to a transition worker. This usually happens around 16 years of age. It can happen a bit earlier if young people will leave care earlier. So because this is a relationship based service the intent is to refer young people earlier so they have a chance to build a relationship and time to build trust with their transition worker prior to leaving care. So whilst they're still in care, the transition worker is there to support and build that relationship. The Oranga Tamariki social worker still has the key responsibilities for the assessment and the planning and the preparation, but the transition worker is there to support with implementing that transition plan.

Once the young person leaves care then the transition worker becomes the lead support for that young person, so that's helping them to set and achieve goals, helping them with the connections that they need, helping them to access the adult supports that they need. That can continue through to the age of 21 years, so it's quite long-term support. It can be up to 5 years of support that young people can receive. Some young people do stay on a big longer – they have complex needs. After that they are able to access the transition assistance helpline that we have set up, which is a dedicated 0800 number with youth workers that are there to provide support, assistance, and some financial assistance, to young people that request support up to the age of 25.

Obviously it's a voluntary service, not all young people will want to access it, so for young people that decline a referral we will continue to maintain contact with them through our transition assistance helpline, so we'll still continue to proactively offer that support up until 21 and they can continue to request that support to the age of 25.

DR DEBBIE MCLEOD: So the evaluation was also a partnership evaluation; it was a partnership with the Evidence Centre at Oranga Tamariki, but most of all – and I think most importantly – it was a partnership with Sarah and her team, the Transition Support Service. Evaluation is only worth funding if it's used by the team, if it actually makes a difference to inform ongoing delivery or ongoing policy development. So this was one of the best evaluations I've worked on from the point of view of the findings being used by our clients.

There were 3 phases in the evaluation. The first phase was very much formative, just looking at what was happening, how the service was being set up. The second part was more focused on processes like tracking the numbers who were referred, finding out, getting early feedback from rangatahi who had been referred for transition support, and then we moved on to an outcomes evaluation, which was after the service had actually had time to be properly embedded, to have a look at what kinds of results were being achieved for the young people as a result of transition support. Central to the evaluation were the voices of young people, their caregivers and their transition workers, which was really important.

So the main evaluation tools: we had some focused evaluation topics where we looked at main parts of the policies, so entitlement to remain and return, supported accommodation, some of the systemic issues that were there, housing. We talked to young people in youth justice settings and we also looked at the education and training context. We heard young people's voices through some annual surveys, which were called the Just Sayin' surveys. Now, surveys have limitations, we all know that, but they're also a good way of getting a breadth of viewpoint, which can be used to complement the in-depth perspectives that we got from a range of interviews with young people and their caregivers.

The Just Sayin' surveys went out; there were 4 lots of them from 2020 through to 2023. The final one was answered by 376 young people between the ages of 16 and 21, and 66 young people who were older than 21. By 2023, it was worth trying to talk to some of the older young people. The surveys were initially online but my team phoned up many, many, many phone calls and many texts to young people to try and make sure that we reached as many as possible. The transition workers – we also appreciate their help in encouraging young people to complete the survey and saying that this was a useful thing to do and giving them a way to do that.

As with any evaluation, there are some limitations and some strengths. I think this strength was that we had the opportunity to work in partnership, to combine a whole lot of different information sources. But one of the challenges is young people who transition from Oranga Tamariki care and did not receive transition worker support were very difficult to connect with. We spoke to a few through their Transition Support Service, but some who just disappeared – of that group, some may have been well supported by the families they were in, by their birth whānau. Some we know were not well supported and were potentially homeless. We don't have their voices to the same extent as we have the voices of others, so in looking at the results, it's important just to bear that in the back of your minds.

Then of course Oranga Tamariki, a major piece of work that was done, and I saw Eyal Apatov appear briefly on my screen and then vanish. He has completed some quantitative analysis of data in the IDI, the Integrated Data Infrastructure, which has a flag or a marker in it of whether young people have been referred to transition support or not. It doesn't say how much support they've received or whether in fact they received any support, but we can look at the difference between those who were referred with care experience and those who weren't. That's been really valuable in providing some early insights; but it's still early on the pathway of change for young people.

So based on Just Sayin', the survey I mentioned, transition support has made a positive difference for many young people. When we ask about different aspects of their lives they may have received transition support for, most young people said they had received transition support with at least one aspect of their lives that aligned with the intended outcomes of the Transition Support Service. You can see on that table that about a third had support to connect with birth whānau, nearly a half to connect with a doctor or nurse, a third with a counsellor, about a quarter with a dentist, nearly a half connection with education and training, and about one in five connection with employment.

It doesn't mean that the others were not connected. It may mean that on their pathway or journey they weren't quite ready to, for example, have support with employment. They may still be trying to resolve health issues. Most young people, both in care and who had left care, when we asked them a number of questions about their relationship with their transition worker were really, really positive about that relationship and the support they were receiving. 

This slide just has some examples of quotes that we had from young people about the kinds of support they'd received under those objectives of the Transition Support Service. I will read through them quickly – some of you may have a small screen that makes it hard to see.

In terms of living arrangements: "The accommodation I was in wasn't good. They wouldn't listen when I said where I was staying wasn't good. They would say it's fine, I got to stay there… The transition worker has helped me a lot, with housing stuff."

More young people are healthy and recovering from trauma: "My transition worker helped with everything that I needed. I needed to go to the doctor, got that sorted. Got a new doctor who's a lot better. Just made things so smooth compared with Oranga Tamariki and the lots of paperwork and lots of processes that they have to do."

More young people have a trusted adult in their lives and are engaged with family, cultural and community groups: "We are now having to go through all the emotions, the ups and downs in our relationship. I have to work out how to be a child to my mum, and my mum to be a mum to me in this short amount of time. We're having to go through all of this now because we lost all those years that we would naturally have gone through it."

More young people in education, employment, training or volunteering: "Anything to do with that other stuff, you know, jobs or all that, I just go to my transition worker. She makes it real easy though, she won't make it difficult. We're going to do this and then we just do it. She makes it simple."

More young people have the life skills they need to thrive as adults: "My transition worker takes me to all these courses, relationship courses, just to be a better person. Managing money. Got my forklift licence, my full licence."

So hopefully those quotes just give you a little bit of a feeling of the way and the kind of support that young people are receiving. So Eyal's work looking at the analysis of data from the Integrated Data Infrastructure, so complementing, because we all know that funders like to see numbers and they like to be able to look at numbers in terms of dollars et cetera, and benefits. So the analysis from the Integrated Data Infrastructure complements the qualitative feedback with some crunchier data. And already, even though it's early in the process, the main changes demonstrated are increases in the numbers of young people gaining a restricted licence – that's quite an important pathway in terms of skills development and employment – lower likelihood to record prison/remand correction sentences or community service correction sentences, a lower likelihood to record emergency department admissions, to receive benefit income, and to record a vulnerably transient status. Young people who had transition support are more likely to, on average, earn wages and salary income for an additional 2.5 months.

Those are very quantifiable kinds of measures, they align with the objectives, but of course we are limited by the information that is collected in the Integrated Data Infrastructure. It's very hard to look at the qualitative measures, the wellbeing kinds of measures, the reconnection with birth whānau, et cetera. But by looking at some of these quantitative outcomes it's very encouraging to see that even at this early stage there are some measurable differences.

So having looked at the fact that we are seeing some positive outcomes, we then ask how well is transition support working and what do we need to think about in terms of thinking about how to strengthen the service and to make it even better? There are a few things that are really important to take note of. The profile of young people leaving care today, so the 16 to 18 year olds, they reflect policies and practices of earlier years to a large extent. Rangatahi Māori are overrepresented in the cohort so obviously the kinds of support that are provided has to be appropriate for rangatahi Māori.

Young people who are transitioning from care have a wide range of different care experiences. We heard from some about stable foster parent scenarios that have worked well, they were part of the family, they were not treated any differently to the birth children in their foster family. Then in the reverse, we heard about others who were living rough, who had experienced abuse and neglect. One particularly sad story that stuck to my mind was a young person who would talk about how the family would go away on holiday but they were never taken along as well, so they were treated very, very differently from the other young people in the family. We did a big survey of young people who were college age, so 12 or 13 through to 18, and in that survey we looked at young people, it's called What About Me. It was slightly over 7000 young people and we looked at those who had care experience – either themselves or in their immediate family – and we compared them with young people who didn't have care experience. There was some really quite big differences just in the cohort, so young people who were in education, at school, on average were way less worried about money, they had much more stable environments. The young people with care experience had substantially more negative responses to health and wellbeing questions, and really scary proportions had either attempted suicide or had suicidal ideation, and they had also experienced more challenges in education settings. So this is the group of young people who our transition workers are supporting. They're not the same as other young people. They come with a history and background of trauma and, for many of them, disadvantage. Within that group, young people who are identified as rainbow, and disabled young people, faced additional challenges and they had further needs. So I just wanted to set the scene for the cohort of young people that are transitioning from care.

The first step in the process is that referral that Sarah talked about. The young people who are receiving transition support reflect the demographic profile of the eligible cohort, so there's no sort of particular bias there or difference between young people from different demographic profiles. The referral rates, so that first point of connection, has plateaued at about 61% of the eligible cohort. We don't know a lot about the other 40-odd percent. As Sarah said, it is a voluntary service. Some young people decline support, some young people are probably never offered support, that interface between Oranga Tamariki social workers and transition workers is still developing. Some may be referred and just never take up the support. Most of the referrals, or the majority, are at age 18 – although some younger age groups are referred, the peak number at age 18. The age of referral to some extent is affected by workloads and waiting lists and how many young people a transition service is able to support. Some do have waiting lists, and the regional full-time equivalent of transition workers to eligible young people does differ slightly, so in some areas there are a higher number of young people that each transition worker would have potentially had to support.

We have concluded that more work is needed at the interface between Oranga Tamariki and the transition support partners. There's more work needed to understand how to improve that, how to ensure that all young people have the opportunity for transition support.

As well as just the opportunity, the choice of transition worker is really important for young people, but also you've got to be realistic and in some regions where there's one transition worker, then choice is going to be more limited. Regionally, there are different partnerships, and there's some support being provided by video or a more remote system, so that enables choice. When asked in the Just Sayin' survey what was important in a transition worker, 81% of young people talked about personality. It was, basically, did they get on, did they feel good, as one of those quotes says, did they match the young person's vibe.

For rangatahi Māori, 61% were supported by a Māori Transition Support Service partner or a Māori transition worker, and that was really important to rangatahi Māori. The number for Pacific young people was 50%.

Once young people are referred – step one is the referral – step two is building trust. I have given you a feel for who the cohort are. Some of them have experienced a lot of trauma, a lot of disadvantage, so that building trust step, it happens in parallel with assessment, planning and support. But it's a really important step, and we heard from the Transition Support Service partners that building trust, it's not a quick thing. It can take one or two years or longer with some young people, depending on what their level of need is. So the service is very much centred around the young person. It's not based on a prescribed process of you see them 3 times and then they'll be ready to do some education or then you'll do that. It's very much around a relationship-based service, building trust, developing the relationship as a platform for then looking at assessment, planning alongside the young person, setting goals, et cetera, and supporting them.

So just a few quotes there that indicate, I guess, the percentage of young people who said that their transition worker understands the support they need, is there when they need them, and does what they said they would do most of the time. So:

“They are still in survival mode. They just want someone to listen to them… they don't have hope and they can't see goals.”

"The bro’s been real mean with, like, easing through this process of transitioning out, with trying to help me explore my hobbies and that because I don't have many hobbies. Transition worker goes out of the way and checking on things I like."

"I've never had this kind of stability in my life before. And this [this was a supported accommodation provider] was definitely the place to be."

"They are meant to disengage at 21 but some still need somebody and the transition assistance line is better than nothing and we try to engage them with it at 21."

So at 21 to 25 they move to transition assistance line support if they are ready to. There is a lot of flexibility around that age. The transition assistance line complements the transition worker support. As the cohort ages, that first cohort have been aeging out for a year now but that volume of young people is going to start moving towards more transition assistance line support, away from the transition worker. We talked to young people and evaluated the transition assistance line service and it was incredibly positive. Young people were super positive about the service. Interestingly, and we had not really taken that into account before we started the interviews, was the high proportion of transition workers who also use the transition assistance line, and that has developed now into almost like a group of people with really good regional knowledge, which Sarah and her team have now sort of brought their insights into some of the policy and delivery.

So what have we learnt for the future? There are some systemic challenges that are sitting around supporting young people as they transition from Oranga Tamariki care. A big one is housing shortages. If young people don't have anywhere stable to live, it's very hard for the transition workers to move along that pathway of skills development, education, et cetera. Having somewhere to live is very, very important and it doesn't sit just with Oranga Tamariki. The other area that there's a real gap in is specialised support for mental health and trauma, particularly trauma for the young people in that transition age group. It's very hard to access external services and many young people may be quite challenged by their mental health but still not reach a threshold for services specifically related for adolescent mental health. Specialist support of advice for young with intellectual disability may also be difficult to access, particularly in some parts of the service.

So an overview of the recommendations and opportunities arising from the evaluation. Number 1, I think the most important one is to continue to provide transition support for young people leaving Oranga Tamariki care as there is evidence that it is making a positive difference for young people. Those differences are going to flow through to the young person's life but also flow through to intergenerational impacts. To ensure there is regional equity and transition worker funding and that young people do have some regional choices. It's very difficult – young people are mobile, budgets are set sort of annually, and then we heard from transition workers how young people suddenly decide – there's a big flow of them from one region to another. Maybe there are employment opportunities. It was very unclear what might sit under that but young people are very mobile.

Supported accommodation was doing very well in terms of meeting some of the housing challenges and preparing young people for living in private rentals or other kinds of accommodation but there's not enough of it. Continuing to ensure the transition assistance line is adequately resourced as the numbers of young people as the numbers of young people reaching 21 increase the demand on the transition assistance line. To improve Oranga Tamariki planning processes and referrals for transition support, so improving that interface, that's the gateway to young people getting from their social workers into support from a transition worker. Enabling equity of access to transition support is aligned with making sure that there's regional access and that the regional referral numbers are similar. Enabling support for young people with multi-layered support needs, so looking at what can be done to support young people with mental health issues, with trauma, and who have multi-layered support needs.

Improving reporting systems so good-quality data are available to inform service development and delivery. Another limitation of the evaluation was that what is happening at the provider level is a little bit of a black box in that we have qualitative data, but we don't have any quantitative data about the intensity of support and how that might align with young people's needs. Then finally to continue to work with other central and regional Government agencies to address the systemic barriers to effective transition.