Evaluation of machine learning methods for predicting eradication of aquatic invasive species

Abstract

In the work, we evaluate the performance of machine learning approaches for predicting successful eradication of aquatic invasive species (AIS) and assess the extent to which eradication of an invasive species depends on the certain specified ecological features of the target ecosystem and/or features that characterize the planned intervention. We studied the outcomes of 143 planned attempts for eradicating AIS, where each attempt was described by ecological and eradication-strategy-related features of the target ecosystem. We considered several machine learning approaches to determine whether one could produce a classifier that accurately predicts weather an invasive species will be eradicated. To assess each learner’s performance, we examined its tenfold cross-validated prediction accuracy as well as the false positive rate, the F-measure, and the Area Under the ROC Curve. We also used Kaplan–Meier survival analysis to determine which features are relevant to predicting the time required for each eradication program. Across the five typical machine learning approaches, our analysis suggests that learners trained by the decision tree work well, and have the best performance. In particular, by examining the trained decision tree model, we found that if an occupied area was not large and/or containments of AIS dispersal were employed, the eradication of AIS was likely to be successful. We also trained decision tree models over only the ecological features and found that their performances were comparable with that of models trained using all features. As our trained decision tree models are accurate, decision makers can use them to estimate the result of the proposed actions before they commit to which specific strategy should be applied.

Appendices

Appendix 1: Definitions

In the main text, we used accuracy, AUC, F-measure, precision and recall to compare the performance of different machine learning algorithms (Powers 2011). Here, we will give a precise description and formula computed based on the following confusion matrix:

		Prediction
		Success	Failure
Truth	Success	TP	FN
	Failure	FP	TN

• Accuracy: The ratio of number of correctly predicted trials and the total number of trials, $\frac{TP+TN}{TP+FP+FN+TN}$.

• Precision: the fraction of predicted ‘Success’ trials that are true: $\frac{TP}{TP+FP}$.

• Recall: The fraction of successful trials that are correctly classified, $\frac{TP}{TP+FN}$.

• F-measure: Harmonic mean of Precision and Recall: $\frac{2\times {\text {Precision}}\times {\text {Recall}}}{{\text {Precision}}+{\text {Recall}}}$.

• AUC: The area under the receiving operating curve (ROC) for a model; here, we followed the method presented in Ferri et al. (2002) for our decision tree model and methods in Fawcett (2006) for other models.

Appendix 2: Kaplan–Meier analysis

We viewed (various subsets of our) database as ‘survival data’, where we set ‘eradication time’ to be the duration of eradication attempts and the ‘censor’ bit to uncensored if eradications succeeded, and to censored if the eradications failed. We then use this idea to compute a Kaplan–Meier survival curve, which produces $P({\hbox {Time to eradication}} \ge T)$ as a function of time T (Cox and Oakes 1984; Lawless 2002; Kleinbaum and Klein 2005).

To explain this process, consider the subset of 61 instances with ‘containment = yes’. We first sorted the durations of these instances from the shortest to the longest (total of 25 durations without repetitions); call these times: $[t_1, t_2, \ldots , t_{25}]$. At each time $t_i$, we defined the ’eradicated trials’ for the instances whose durations were $t_i$ and whose outcome was ‘Success’, and ’censored trials’ for attempts with same duration but whose outcomes was ‘Failure’. We also defined the number of trials at risk at time $t_i$ to be the number of trials whose durations were no less than $t_i$. We used these quantities to compute the survival probability corresponding to these $t_i$’s, which are the 25 $P_i$’s; the curve then contains these 25 $[t_i, P_i]$ pairs; see Fig. 7a. The probability can be calculated by the following formula

$$P_i=\Pi _{i:t_i\le t} \left( 1-\frac{d_i}{n_i}\right) ,$$

with $d_i$ be the number of events and $n_i$ be the total individuals at risk at time i. The survival probability at each time point are listed in the following table.

Time (year)	Number of eradicated trials	Number of censoring (failed trials)	Number of trials at risk	Survival probability
$t_0 =0.00$				$P_0 = 1$
$t_1=0.08$	1	0	61	$P_1=1-\frac{1}{61}$
$t_2=0.17$	1	0	60	$P_2=P_1 \cdot (1-\frac{1}{60})$
$t_3=0.25$	3	0	59	$P_3=P_2 \cdot (1-\frac{3}{59})$
$t_4=0.33$	0	1	56	$P_4=P_3 \cdot (1-\frac{0}{56})$
$t_5=0.83$	1	0	55	$P_5=P_4 \cdot (1-\frac{1}{55})$
$t_6=1.00$	2	3	54	$P_6=P_5 \cdot (1-\frac{2}{54})$
$t_7=1.33$	1	2	49	$P_7=P_6 \cdot (1-\frac{1}{49})$
$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$
$t_ {25}=18.00$	0	1	1	$P_{25}=P_{24} \cdot (1-\frac{0}{1})$