Keywords

1 Introduction

There are nearly 2 million homeless youth in the United States each year. These are young people between the age of 13 and 24 who are homeless, unaccompanied by family, living outdoors, in places not fit for human habitation, and in emergency shelters [9]. The consequences of youth homelessness are many, including many preventable problems such as exposure to violence, trauma, substance use, and sexually transmitted disease [9]. A critical solution to improve long term outcomes for homeless youth is to quickly and efficiently help the homeless youth find safe and stable housing situations. Indeed, there are many non-profit organizations and public sector programs designed to do this. In almost all communities in the United States, the number of youth experiencing homelessness exceeds the capacity of the housing resources available to youth [3]. This situation leaves communities with the terrible predicament of trying to decide who to prioritize for the precious few spots in housing programs which are available at any given time. Most communities have moved to what is referred to as a Coordinated Entry System. In such systems, most agencies within a community pool their housing resources in a centralized system. Persons who are seeking housing are first assessed for eligibility for housing, which usually includes HUD-defined chronic homelessness, other criteria such as veteran status, and “vulnerability”. Based on these assessments, individual youth are prioritized for housing and placed on waiting lists until appropriate housing becomes available in the community [3]. Despite these efforts, most of the prioritization decisions are made by humans manually working in the housing communities using some (possibly unknown) rubric. Could we understand how humans make these prioritization decisions? Could we provide important insights to the local communities using the housing assistance assignment data from the past? In this paper, we provide simple machine learning analyses and tools that could be of use to communities. Our results are the most comprehensive, non-experimental evaluation of youth coordinated entry systems of which we are aware. We view this paper as a gateway for providing policy recommendations to improve the housing systems.

1.1 Our Goal

HUD wants community housing systems to be systematic, evidence-based and grounded in research [3, 4]. Despite of this, save for a few exceptions (e.g. [2]), the current housing allocation system for youth has not been evaluated for its success. As a result, the goal of this paper is to see if we can evaluate the success of the current system using the data from the HUD’s Homelessness Management Information System (HMIS), the primary repository for data on homeless services delivery in the U.S. If we can uncover (which we have) new insights in the current system, there is a potential to make a major impact in policy and societal outcomes. We present the first study on evaluating such system.

Our Contribution. In additional to providing an overview of the current housing system, we provide insights that would help the communities to understand and evaluate the current housing assignment process. In particular, using the past housing assistance assignment data of homeless youth, we:

  1. (a)

    Infer the decision rules of the local communities for providing youth with housing assistance;

  2. (b)

    Evaluate the outcome of the current assignment process;

  3. (c)

    Build and learn an interpretable classifier to predict homelessness outcome of each youth to a homelessness exitFootnote 1; by leveraging vulnerability assessment tools;

  4. (d)

    Provide public policy recommendations to improve the housing system.

Since our tools need to be understood by the housing communities, it is important for the tools to be explainable and easy to use. As such, we focus on learning interpretable classifiers such as logistic regressions and decision trees. While our analyses and the tools are simple, they are extremely impactful as evident by the fact that we are requested by the HUD to make policy recommendations based on our findings. The remainder of this paper is organized as follows. In Sect. 2, we provide an overview of the current housing systems for homeless youth. In Sect. 3, we discuss the dataset obtained from Ian De Jong (Orgcode), as part of a working group called “Youth Homelessness Data, Policy, Research” led by Megan Gibbard (A Way Home America) and Megan Blondin (MANY), which includes members of HUD, USICH, and ACF, as well as researchers from USC (Rice) and Chapin Hall at the University of Chicago (Morton). In Sect. 4, we discuss our methodology for learning the classifiers. We then infer the communities’ decision rules for assigning youth to various housing programs in Sect. 5 and show the outcome of such assignment in Sect. 6. In Sect. 7, we show how we can use the vulnerability assessment tools/features to predict homelessness outcome of homeless youth. In Sect. 8, we present our policy recommendations to the HUD and housing communities based on our analysis and the summary of our results. In Sect. 9, we conclude this paper by highlighting the values of our study to the communities and HUD.

2 Current Approach for Housing Prioritization

HUD offers many mandates, guidelines, and best practice recommendations to communities on housing youth [3, 4]. In most Coordinated Entry Systems for homeless youth, housing agencies within a community pool their housing resources in a centralized system. First, a homeless youth enters a centralized intake location (e.g. emergency shelters, street outreach workers, or drop-in centers) to sign up for housing support. There, they are assessed for housing eligibility and vulnerability/risk. All this information is then entered into the HMIS. Then, based on these assessments, a case manager or a team of housing navigators decide how that youth is to be prioritized for housing. The youth is then placed on a waiting list until appropriate housing becomes available.

Although communities may decide for themselves what risk/vulnerability assessment tool to use, the most frequently used tool for assessing the risk levels of youth is the Next Step Tool (NST) for Homeless Youth developed by OrgCode Consulting Inc. and Community Solutions and thus we focus our analyses on this tool.Footnote 2 Roughly speaking, the NST is a set of multiple-choice, dichotomous, and frequency-type questions to measure a youth’s vulnerability based on his/her history of housing and homelessness, risks, socialization and daily functions, and wellness. Based on the results of NST, a youth is scored from 0 to 17.

Based on the recommendations provided in the NST documentation, youth who score 8 to 17 are designated “high risk” youth and should be prioritized for Permanent Supportive Housing (PSH), a resource-intensive housing program which includes “wrap-around” social services for youth to assist them in remaining stably housed. Youth who score lower (the 4–7 range) are typically referred to Rapid Rehousing (RRH) which is a short-term rental subsidy program that infrequently has many social services attached. Some youth who score low (less that 4) may not ever receive housing resources. For many providers and communities, this step is often painful as the desire to help all homeless youth is foremost in the minds of every provider. The NST scoring recommendations are not a hard and fast set of rules, but as we show in our analyses, most communities follow these cut points when assigning housing to youth.

However, the NST is a general vulnerability measure, not tied to a particular outcome, and no research has been conducted to date which links this tool to particular outcomes, particularly long-term housing stability. As noted by many communities, the housing stability of a youth as they exit a program is often the most robust measure of success [7]. That is, they want to assign youth to the appropriate housing programs in order to maximize the youth’s chances of being stably housed in the future. For instance, if a youth is placed in PSH, a successful outcome would be continuation of stay unless they transition to stable unsubsidized housing. For those receiving RRH, remaining a stable renter without further government assistance is a positive outcome. Such outcomes, however, might not have any positive correlation with the youth’s risk levels.

3 Description of the Data

The dataset consists of 10,922 homeless youth registered for housing services from the HMIS database from different communities in the U.S.Footnote 3 In the context of social science and our domain, this dataset is considered to be large and valuable by many researchers and industrial partners. These records were anonymized and provided by Iain De Jong of Orgcode. Some youth have already been assigned to some housing programs while others are still waiting for housing assignments. Each record has the youth’s age, gender, LGBT status, ethnicity, type of communities, and a list of responses to the NST questions (including NST score) assessing a youth’s vulnerability.

Table 1. Different subsets of features.
Full size table

3.1 Features from the Youth

The features of the youth are divided into the following subsets as in Table 1. The DOM features are basic demographic characteristics of the youth. The COM features are the type of community in which a youth lives. The NST evaluates the vulnerability of a youth based on his/her responses to the forty questions (NSTQ) about youth’s history of housing and homelessness, risks, socialization, daily functioning, and wellness components. Each component scores a youth based on the responses to the questions within the component (NSTT). The NST score (NSTS) is the sum of the scores from the components.

Table 2. Basic statistics of the data. #Y = Number of Youth, TofE = Type of Exits, #SH = Number of Youth Still Housed, and AvgNSTS = Avg. NST Scores.
Full size table

3.2 Types of Exits from Homelessness

For each homeless youth in the data, there are fields specifying his/her type of exit from homelessness and whether s/he is still living in the same type of exit after a fixed time period (a.k.a. Still Housed). The still-housed responses indicate whether a housing program was successful for the youth; “Yes” answers indicate a youth is still stably housed, a positive outcome; and “No” indicates a youth has exited housing assistance and returned to homelessness, a negative outcome. Table 2 lists the number of youth in each type of exit in the data.

A large number of youth is still waiting for housing assignments and/or have been lost to the housing system. In many cases, some homeless youth went to live with their family members (Family) or were able to find housing themselves (Self Resolve). There are three main types of housing programs in the dataset: supportive services for veteran families (SSVF), permanent supportive housing (PSH), and rapid re-housing (RRH).

Fig. 1.
figure 1

Histograms of housing assignment/placement by NST scores.

Full size image

Figure 1 shows the housing assignment by the NST scores. Most communities are not assigning youth to PSH with NST scores lower than 8 while most communities assign youth with NST scores between 3 and 9 to RRH.

Given the data, our prediction tasks are to (1) infer the decision rules of the local communities and (2) understand how the features of a youth affect the probability that a youth will have positive outcomes (i.e. still-housed) given different types of exits from homelessness. Due to the small sample sizes of SSVF, we focus on Family, PSH, RRH, and Self Resolve exits.

4 Methodology and Evaluation Metrics

To infer the decision rules of the local communities for assigning youth to PSH and RRH, we consider multiclass classification problem of classifying youth into Family, PSH, RRH, or Self Resolve exits. This allows us to infer the most likely exit of the youth and the decision rules. We use the one vs all strategies where we train a classifier per class and view the same-class examples as positive examples and the other classes’ examples as negative examples. To infer a youth’s probabilities of success for different types of exits, we can naturally cast the problem as binary classification and learn a binary classifier for each type of exit. In this binary classification, for each type of exit, youth assigned to the exit with “Yes” and “No” still-housed responses are positive and negative examples, respectively. In both cases, we consider the following classifiers and performance measure.

4.1 Classifiers

Due to the explainability and ease of interpretation for end users, we focused on learning logistic regression and decision tree classifiers for each type of exit [5, 6, 8]. Moreover, we require the classifier to output class posterior probabilities for each of our classifiers.Footnote 4 To learn our classifiers, we use 80% and 20% of the data, pertaining to the type of exit, for training and testing, respectively. We use 10-fold cross validation in the training set to find the best hyperparameters to regularize the classifiers (L1-norm for logistic regression and depth for the decision tree). For constructing the decision (classification) trees, we consider the standard CART model to build a binary tree and select nodes/features and values to split based on Gini’s diversity index [1]. For each split, we consider all the possible pairs of features and values. To control the depth of the tree, we use cross validation to select the best minimum number of nodes at the leaves. Finally, we prune the learned tree to obtain a simpler decision tree classifier.

Using different feature combinations, we learn logistic regression and decision tree classifiers and measure the performance using AUROC (defined below).

4.2 Performance Measure

We measure the predictive performance using the area under the receiver operating characteristic curve (AUROC). The ROC is constructed for the learned classifiers based on the true positive rate (true positive divided by true positive plus false negative) and the false positive rate (false positive divided by false positive plus true negative) points for each possible cutoff posterior probabilities in the test data. We then compute the area under the ROC. Roughly speaking, the AUROC is equal to the probability that a randomly chosen youth with a positive class/still-housed outcome ranks above (i.e, has a higher probability of success) than a randomly chosen youth with a negative class/still-housed outcome. Thus, higher AUROC indicates that the classifier is able to distinguish the classes effectively. AUROC is particularly useful in our setting because the unbalanced nature of our data (\({\approx }\)76–82\(\%\) positive outcomes) as the standard 50\(\%\) cutoffs for computing accuracy could provide us with a representative model rather than a discriminative model. We report the average AUROC over 100 different 80\(\%\) and 20\(\%\) splits of our data into training and testing for the learned classifiers. We omit reporting the small standard deviations for brevity.

4.3 Building the Final Model

To build the final model, we trained the classifiers using all of the available data for each exit. We highlight the important coefficients and decision nodes of the learned logistic regressions and decision trees. We can interpret the exponentiated coefficient of a predictor as the odds ratio when holding other predictors constant (i.e, a one-unit increase in the predictor value corresponds to some percentage of (multiplicity) increase in the odds of being successful).

5 Decision Rules and Youths’ Most Likely Exits

In this section, we are interested in inferring the decision rules used by the communities to assign and prioritize youth for housing assistance (e.g., PSH and RRH) as well as the youths’ likelihood of returning to family (e.g., Family) and finding their own housing or rental subsides (e.g., Self Resolve) without any housing assistance. Table 3 shows the AUROC of the logistic regression and decision tree classifiers for the youths’ exits from homelessness using different combination of features. The learned logistic regressions and decision trees for PSH and RRH can be viewed as decision rules for assigning youth.

Table 3. AUROC of logistic regression and decision tree for each type of exits. F = Family Exit, P = PSH Exit, R = RRH Exit, S = Self-resolve Exit.
Full size table

5.1 Decision Rules for Assigning Youth to PSH and RRH

From Table 3, we observe that a youth’s NST score (NSTS) alone is a very good predictor for predicting whether the youth will exit homelessness to PSH. The learned logistic regression with NSTS feature yields the highest AUROC than the learned decision tree with any combination of features. Moreover, its decision boundary is similar to the NST recommendation (see left of Fig. 2).

Fig. 2.
figure 2

Logistic regressions of the probabilities of (left) PSH exits and self resolve (right) exits. The coefficient and constant of the left (right) logistic regression are 1.58 (−0.83) and −14.74 (2.88), respectively.

Full size image
Fig. 3.
figure 3

Decision trees for RRH exits. The probabilities of assigning to RRH are displayed at the leave nodes (i.e., Pr(RHH)).

Full size image

On the other hand, it seems that the decision rule of RRH is more complicated – the NSTS alone is no longer the best predictor for predicting RRH exit. Indeed, the RRH decision rule can be better captured using a non-linear decision tree classifier (see Table 3). In general, the learned decision trees with high AUROC have a very similar structure – those with NST scores less than 5 or greater than 9 have almost no chance of getting RRH while those with NST scores between 5 and 9 have high chance of getting RRH subject to various additional conditions such as ages, lengths since last lived in stable housing, and violence at home between family members (see Fig. 3 as an example learned decision tree with feature set NSTA + DOM + COM). The learned decision rule for RRH is similar to the NST recommendation. However, youth seem to be selected based on additional criteria other than the NST score alone (Table 4).

Table 4. Family exit: top-6 (ordered) important features of the learned logistic regression.
Full size table

5.2 Youths’ Exits to Family and Self Resolve

Similarly, Table 3 shows the set of features that can help us to predict youths’ chances of existing homelessness to Family and Self Resolve. To predict Self Resolve exit, the learned logistic regression that uses NSTS as the only feature is a good predictive model relative to different combination of features and decision tree classifiers. Its decision boundary is shown in (the right of) Fig. 2. Interestingly, the youth with low NST scores have high chances of self resolve. However, youth with NST scores between 6 and 8 seem to have low chances of going to PSH and being able to Self Resolve.

While the logistic regression that uses NSTS as the only feature is a good predictive model for predicting Family exit, other combinations of features improve the AUROC. In particular, the logistic regressions learned using NSTA + COM and NSTA + DOM + COM yield the highest AUROC. We select the model with the lowest number of features and report their coefficients. Figure 4 displays the top-6 important features of the learned logistic regression with NSTA + COM. The important features are related to the communities, ages, and NST scores of the youth. Surprisingly, the youths’ communities have some impact on the youths’ chances of going back to Family. As a result, we list and separate the communities with positive and negative weights in Table 5. Youth in the positive communities seem to be more likely go to back to home while those in the negative communities are less likely.

Table 5. Family exit: positive and negative weighted communities from the learned logistic regression.
Full size table

6 Outcome of the Current Housing Assignment Process

Now that we have have a better understanding of how the assignments are performed in practice, we discuss the outcome of such practice by looking at the number of youth still-housed for each type of exits based on the NST scores.

Fig. 4.
figure 4

Percentages of youth still-housed vs NST scores.

Full size image

Figure 4 shows the percent of youth remaining housed by the housing placement based on their NST score. While Table 2 indicates that 75%–81% of youth are still-housed under the assignment, the above analysis provides us some comparative statistics when the percentages are breakdown by the NST scores.

In particular, as the NST score increases, the number of youth who is still-housed decreases. PSH seems effective for youth with high NST scores: 70% of youth with NST score of 14 and 100% of the 16 youth with NST score 7 or lower are still-housed. For RRH, 80%–70% of youth with score 4–9 remain still-housed while 57% of he 19 youth with score 10 is still-housed. For the youth that went back to their family, 90% of those with NST score of 4 or less and 80% of those with NST score of 5–6 are still-housed. However, the percentage drops significantly for higher NST score youth. For the self-resolve youth, roughly 90% of low NST score (1–4) youth is still-housed while the percentage drops below 50% for those with high NST scores. As we will show in the next section, there are additional factors beyond NST score that could be used to predict the still-housed outcome of the youth.

7 Predicting the Outcomes of Housing Assignments

In this section, we want to learn an interpretable classifier that would tell us a youth’s probability of success (i.e., positive still-housed outcome) for a particular type of exit. Our plan is that communities can use our classifier as a decision aid to assist in determining the probability of success for a particular youth in a specific housing program. Such human-machine interaction will provide improved decision-making in the allocation of housing programs to youth.

Table 6. AUROC of logistic regression and decision tree for each type of exits. F = Family Exit, P = PSH Exit, R = RRH Exit, S = Self-resolve Exit.
Full size table

7.1 Logistic Regressions and Decision Trees

Table 6 shows the average AUROC of the classifiers for different subsets of features for each type of exit. First, the performance of the logistic regressions and decision trees is similar for most feature combinations. The COM features are not very useful and the learned models have AUROC only slightly better than random. The NST score (NSTS) alone is a reasonable predictor of the outcome. We tried different combinations of NSTQ, NSTT, and NSTS and found that we are able to build a better model with a higher/similar AUROC for all types of exits by combining all three subsets (i.e., NSTA). We also added COM and DOM to NSTA to see if we could improve the model. Unfortunately, the AUROC did not improve much. For consistency, we report the learned logistic regression and decision tree classifiers using the feature set NSTA. For all of the exit types except PSH exit, our learned models are reasonable (with good AUROC scores). For RRH (R) exit and Self Resolve (S) exit, the AUROC increased by 15% to 30% over the AUROC of the learned logistic regressions using only the NST score. This provides some important indication that other parts of NST are useful for building better predictive models. On the other hand, our models for PSH seem to be much weaker than the other models for different types of exits, and the performance does not improve with more features.

7.2 Significant Features of the Learned Models

Now that we have discussed the performance of the classifiers, let us look at important features in the models for RRH and Self-resolve exits as they benefit from more features. We select the model with the lowest number of features and train the classifiers using all of the available data for the exits. We highlight the important coefficients of the logistic regressions.

Learned Logistic Regressions. Tables 7 and 8 show the top-6 most important features (in terms of weight) of the learned logistic regressions for RRH and Self Resolve exits. In many of these classifiers, the locations where youth most frequently sleep and the NST scores are important features.

Table 7. RRH exit: top-6 (ordered) important features of the learned logistic regression.
Full size table

For RRH exit (Table 7), having physical, mental, and substance abuse issues and sleeping in shelters most frequently are positive factors for being successful in RRH. Youth with age of 17 or younger have lowered chances for success. For Self Resolve exit (Table 8), youth that are 17 or younger or have some physical, mental, substance abuse, and medication issues have decreased chances of being able to successfully exit homelessness on their own. If a youth has used marijuana at 12 or younger, then the youth has an increased probability of being successful.

Table 8. Self resolve exit: top-6 (ordered) important features of the learned logistic regression.
Full size table

8 Policy Recommendations from Our Empirical Analysis

To obtain feedback on the findings, we shared and discussed our initial results with some members of the local communities and HUD on August and November of 2017. Our audiences (even for those that do not have any technical background) found the logistic regression and decision tree classifiers interpretable and understandable after some brief explanation. They found the results helpful for justifying the existing method for prioritizing youth for housing program based on the NST score. Moreover, they are intrigued and excited to see the important features for determining youths’ chances of successes in different types of exits. Due to the significant of our results and findings, the housing communities and HUD asked us to make policy recommendations based on our analysis. As a result, below is our recommendations from the analysis:

  1. 1.

    Use the TAY-VI-SPDAT: Next Step Tool for Homeless Youth (NST) to assess vulnerability.

  2. 2.

    PSH is effective for almost any youth, regardless of NST score.

  3. 3.

    Rapid rehousing (RRH) can be an effective solution for youth who score up to 10.

  4. 4.

    66% of youth who score less than 4 successfully self-resolve or return home. Family reunification and other case management services appear sufficient for many.

  5. 5.

    More experimentation with rapid rehousing for higher scoring youth is needed.

Given that most of the local communities is assigning homeless youth to PSH and RRH using various components of the NST tool according to our analysis in Sect. 5, the policy on using the NST tool is a natural step and is easy to incorporate into any existing local housing systems. Policy recommendations of (2) and (3) are also sensible since some of the existing assignments are being done based on these cutoffs. Our observation of (4) will allow the communities to better allocate housing resources to those of higher NST scores since lower NST score homeless youth can find successful outcome by going back to family or self-resolve. The last item in the policy recommendations aims to solicit more data from the communities so that we can provide a better evaluation of RRH.

Table 9. Observation summary. F = Family Exit, P = PSH Exit, R = RRH Exit, S = Self-resolve Exit, NSTS = NST Score, Beyond NSTS = Features Beyond NSTS
Full size table

In addition to these recommendations, Table 9 provides a summary of our results. We observe that in many situations, the assignment decisions and outcomes are not strictly depending on just the NSTS score for other types of exits except PSH. For instances, those that were placed into RRH and their assignment outcome are based on their responses to the NST questionaries, demographic information, and communities.

9 Conclusion, Discussion, and Future Work

This initial study has generated a great deal of interest from the housing communities and government agencies and a set of implementable public policy recommendations. Moving forward, there is much potential for creating decision aids to improve housing systems. There is room for effective human-machine collaboration in the context of housing allocation. Our analyses show that the local communities follow the current housing system guidelines closely for assigning youth to PSH. On the other hand, local communities have used additional criteria to assign youth to RRH. Our analyses also show that NST scores have a small negative correlation when youth are given housing interventions but a profound negative trajectory as NST score increases without intervention. This suggests that assigning youth based on NST score is an effective intervention for assisting high risk youth. As such, the current housing assignment systems are providing a much needed housing resource to youth who would otherwise not achieve stable living on their own.

Moving Forward. Assignment decisions based on NST scores, however, can be greatly augmented by additional predictive analytics. As shown in Sect. 7, we can potentially improve the current housing systems by providing better interpretable and explainable tools/classifiers to estimate a youth’s probability of success for each possible type of exit (i.e. PSH, RRH, Family, Self Resolve). Given these probabilities, social workers in each community can decide more precisely which housing intervention (PSH or RRH) is best or whether a given youth is likely to successfully achieve stability without help from the system (family or self-resolution). Such information could do much to aid housing providers in making more informed decisions as to where to place a youth such that he/she is most likely to succeed. Moreover, providers may feel less anxiety about providing limited resources to some youth if they have information that suggests that a youth has a high probability of self-resolution or return to family. Thus, social service efforts can be focused more comfortably on those youth who are unlikely to succeed unless given more intensive resources such as PSH or RRH.

Our decision aids can further complement a human user. The ordered important features identified by the logistic regressions can serve as “red flags” for providers. For example, youth who have been abused or traumatized are less likely to be successful in PSH. This does not mean that providers should not place youth with such histories into PSH. Rather, additional supports, perhaps mental health treatment for trauma, are needed for youth with such a history. Likewise, youth who are 17 or younger are much less likely to be able to succeed in RRH, suggesting that youth under 17 if given rental subsidies may need added attention beyond just the basic rental subsidy in order to improve their chances of remaining stably housed.

Based on these preliminary findings, youth housing systems are an ideal setting in which to further explore the potential for decision aid systems for social service providers. These two basic additions which we have outlined here, do much to enhance the vulnerability screening currently in place and could greatly aid humans in making difficult decisions about which youth to place in which housing programs, and which youth within those programs may need additional attention in order to thrive. In the future, by continuing to work with HUD and local communities we hope to build a decision aid system that will provide humans with enhanced predictive criteria for outcomes of housing placements for particular youth. Finally, we plan to provide useful interactive assistants, such as graphical user interfaces, to facilitate and encourage the collaboration between machine (i.e., our system) and human users in the community.

Lesson Learned. Many housing providers are resistant to using tools whether based on an index or machine learning that will decide on housing placements in an automated fashion. Housing is a critical resource that profoundly impacts the well-being of youth. People working in the communities that provide housing assistance to youth feel that humans must remain a part of the decision-making process [7, 9]. Many current systems are often perceived as too rigid, and future systems must make room for human-machine interaction in decision-making.