Introduction

The need for mark-recapture techniques that minimize post-marking/tagging effects on fish is well recognized for individuals under 70 mm fork length (FL) in which tagging and marking options are limited (Skalski et al. 2009; Wilder et al. 2016; Sandford et al. 2019). Natural marks such as skin pigmentation and scars can provide a non-invasive technique for recognizing individual animals based on photos taken over time. Natural marks can also allow individual identification within a species provided there is sufficient polymorphism and information in the characteristic in question (Pennycuick 1978). The feasibility of using natural marks has been demonstrated in a variety of fish groups, such as ictalurids (Barriga et al. 2015), salmonids (Garcia de Leaniz et al. 1994; Merz et al. 2012), sharks (Castro and Rosa 2005; Speed et al. 2007; Van Tienhoven et al. 2007), syngnathids (Martin-Smith 2011; Freret-Meurer et al. 2013), and osmerids (Castillo et al. 2018). Those studies have shown that the stability of individual natural marks may depend on the interval of time between photos and the life stage of the species considered. However, the extent to which ambient light can influence natural marks in fishes has received limited attention (e.g., Castillo et al. 2018), emphasizing the need to further evaluate the ambient light conditions under which natural marks can be effectively used to identify individuals in a given species.

We evaluated the effects of light intensity on the expression of chromatophores in delta smelt (Hypomesus transpacificus), a small endangered osmerid endemic to the Upper San Francisco Estuary. Two of the world’s largest water diversions operate year round within the habitat of delta smelt, making this species particularly sensitive to entrainment and habitat loss (Grimaldo et al. 2009; Moyle et al. 2016). Delta smelt experienced a long-term decline becoming exceedingly rare in field surveys since the 2012 drought (Moyle et al. 2016), and its population has reached record low levels after the drought (Hobbs et al. 2017). Due to the lack of adequate individual marking and tagging methods for delta smelt and its sensitivity to handling stress (Swanson et al. 1996), no tagging or marking studies have been conducted to track wild delta smelt. Hence, there is a pressing need to evaluate the least invasive marking/tagging methods that could be adapted to track this species in the field. Castillo et al. (2018) reported it is feasible to identify individual cultured delta smelt from winter to spring based on their chromatophore patterns. In that study, adult fish were maintained at low indoor ambient light and photographed two months apart. Those fish were then moved to high outdoors ambient light in early spring and after two months the size of their chromatophores had decreased. However, the separate influence of time and ambient light on chromatophores stability was not evaluated in that study.

Considering that delta smelt move between nearshore and deeper channels to maintain their upstream position or to migrate in their tidally influenced habitat (Bennett and Burau 2014), the ambient light for this species is likely influenced by tides, prey occurrence and spawning migration. Moreover, catch rates of wild delta smelt are typically higher at locations with elevated turbidity (Feyrer et al. 2007), implying higher reliance on habitats with low ambient light, but such habitats have decreased due to long-term declines of turbidity in the upper San Francisco Estuary (Feyrer et al. 2007; Schoellhamer 2011). Hence, the ambient light experienced by wild delta smelt has likely increased over the years but the extent to which ambient light influences the stability of chromatophores within and between life stages is unknown.

To address the uncertainties associated with the use of natural marks on delta smelt, we examined the potential value of chromatophores as natural marks between the subadult and adult stages, and under a range of ambient light more likely include those experienced by wild delta smelt. Our objectives were to evaluate: 1) whether chromatophores can be used to reliably distinguish individual delta smelt from the subadult to adult stages based on visual (naked eye) and automated image recognition software and 2) whether chromatophores stability is influenced by ambient light, life stage and gender.

Methods

Fish culture and tagging

Delta smelt were cultured at the University of California, Davis, Fish Conservation and Culture Lab (FCCL, Byron, CA). Fish were held in black-interior insulated fiberglass tanks (1100 L; working volume 860 L). Details on the fish culture system set up and procedures are reported in Lindberg et al. (2013). Individual fish identification was independently verified throughout the study using plastic fluorescent alpha-numeric tags (1.2 X 2.7 mm; Visible Implant Alphanumeric tags, VIA tags; Northwest Marine Technologies, Olympia, WA). Based on tag shedding tests conducted at the FCCL with this VIA tag (Lindberg et al. 2013), we tagged subadult fish (55.1 ± 5.6 mm FL; 1.16 ± 0.40 g total weight; mean ± SD) to minimize tag shedding. A total of 520 fish were tagged on September 24, 2014. Fish mortalities and tag shedding were recorded for the duration of the study (Table 1).

Table 1 Number of delta smelt used in three photo sessions conducted between fall 2014 and spring 2015 under two ambient light treatments, fish survival between sessions and from the day fish were VIA tagged, and the percent of VIA tags lost between sessions
Full size table

Ambient light treatments

The same day all fish were tagged, they were transferred to two outdoor holding tanks (each 1100 L; working volume 860 L) with a density of 260 fish/tank. Two ambient light treatments were used to evaluate whether different fish exposure to sunlight influenced chromatophore expression and the ability to recognize individuals from fall to spring. One group of fish was exposed to low ambient light (4.06 ± 3.52 lx, mean ± SD) which was provided by a canopy above the tank to attenuate direct sunlight and a shade cloth directly covering the tank. The second group of fish was exposed to high ambient light (223 ± 100 lx, mean ± SD) with the tank only covered from direct sun by a shade cloth. The shade cloth covering both tanks also served prevent avian predators. The first photo session started on day 230 post hatch and the interval between photos sessions was 90 days (sessions 1–2) and 86 days (sessions 2–3), with each photo session completed within a week (Table 1).

Image selection and capture

We used two criteria to select suitable body areas for natural marks evaluation: 1) clearly evident chromatophores and 2) varied chromatophores among fish (Castillo et al. 2018). We selected dorsal head areas for performing visual recognition (areas A, B and C) and automated recognition (cropped areas A, B and C; Fig. 1 a-b). Then, we captured three types of images per fish across all photo sessions: the dorsal head area, the VIA tag, and the full lateral view of the fish; with each of these photo sets comprising 1444 samples. Each sample was the highest resolution image selected among three replicates. The photos of the VIA tag and the lateral view photos were used for improved fish tracking.

Fig. 1
figure 1

Anterior dorsal surface of adult delta smelt used for photo matching, a head image showing pre-, inter-, and post-orbital areas (A, B and C) used in visual recognition, b same fish showing cropped areas A–B − C used in automated recognition and the circle pattern in area A used to crop images

Full size image

Fish were immersed for about 2 min in tricaine methyl sulfonate (MS-222, 0.1 g/L; Finquel, Argent Chemical Laboratories, Redmond, Washington) to reduce handling stress and mobility during photo sessions. It took about 0.5 min to photograph each fish. A digital camera (Canon EOS Rebel XTi) was mounted on a tripod, with a macro lens (Canon USA, Inc., EF 100 mm f/2.8 USM) pointing down about 15 cm above the fish. The fish were held in a pan with water while the first two photos were taken, a partial lateral body photo showing the VIA tag and a second photo showing the top of the head. The full lateral view of the fish body was captured with another camera (Canon Power Shot D10; Canon USA, Inc.), and the fish was placed in an acrylic holder with water (Castillo et al. 2018). After each photo session fish were allowed to recover in a black 20 L bucket and then moved to their treatment tank.

Images of the dorsal head area were standardized between photo sessions (Castillo et al. 2018; adapted from Merz et al. 2012) by controlling three axes: 1) rotation around the long axis of the body so both eyes’ lenses were equally positioned along a horizontal line, 2) the mouth of the fish was slightly tilted upward to position the head areas at similar elevation to optimize focus, and 3) left-right rotation - by orienting the head to the same side of the frame during image acquisition.

Automated image recognition

The size and rotation of digital photos for the head areas A − B − C (Fig. 1a-b) were standardized using manual editing prior to automated matching (Castillo et al. 2018). Automated image recognition was conducted using the TinEye Match Engine’s Application Programming Interface - API; https://www.tineye.com) to enable comparisons with the latter study based on the same high-fidelity version of TinEye. The image that most closely resemble each target image is identified by TinEye whenever possible, with a percentage similarity score assigned to each image pair (automated match quality). The use of a cutoff score value was not considered in our study since TinEye returned only the best match per image between photo sessions. The number of individuals compared between a given photo session pair per treatment was determined by the lowest number of surviving fish that still retained VIA tag in the two corresponding photo sessions (Table 1).

Visual photo recognition

Two testers examined the chromatophores in the head (Fig. 1a-b) and attempted to visually recognize individual delta smelt between each pair of photo sessions (1–2, 1–3, 2–3). None of the testers was trained on visual image recognition prior, or during, the present study, and both had similar experience matching head images of delta smelt and achieved a 100% matching success among three photo sessions (Castillo et al. 2018). Visual recognition was based on blind tests of 30 randomly selected fish per tester, with half of the fish selected from the low and high light treatments. A given fish was only used in one of the three pairs of photo sessions. Testers used a qualitative ranking to assign a measure of confidence to the visual match quality based on areas A, B and C (1: poor; 2: fair; 3: good; and 4: excellent, Castillo et al. 2018). Each tester matched a pair of photo sessions including 30 fish in about 8 to 14 h and their data were combined for analyses since they showed similar patterns between photo sessions.

Light effect on chromatophores

Testers compared random samples of 30 fish between sessions to evaluate the relative changes in chromatophore size for each individual and light treatment. We used the ranking system of Castillo et al. (2018) to evaluate the relative overall difference of chromatophore sizes in the head (1: little, 2: moderate, and 3: substantial), with a sign indicating whether chromatophore size increased (+) or decreased (−) in the most recent photo session compared. The changes in ambient light experienced by fish were estimated by measuring illuminance with a light meter (LI-COR, model LI-250A). Illuminance data collected weekly in winter-spring 2017 was used to complement preliminary data collected in fall-winter 2014–15. Illuminance was measured below the shade cloth of the tanks at noon to reflect the largest likely daily contrast in ambient light between the low and high light treatments.

Statistical evaluation

We used the equality of proportions (z test, Devore and Peck 1986) to compare the number of correctly matched images to the total number of images evaluated (success rate) between each photo session pair (1–2, 2–3, 1–3). We used ANOVA (Sokal and Rohlf 1995) to evaluate whether ambient light, gender, and photo sessions pairs influenced match quality and chromatophore expression. The sample size for each treatment level in ANOVA analyses was inferred from the degrees of freedom of the F statistic (n = total df + 1 / treatment df + 1) since all treatments were orthogonal (Sokal and Rohlf 1995). These analyses were conducted separately for automated and visual recognition as follows: 1) match quality ~ photo session-ambient light combinations (i.e., three session pairs x two ambient light levels) and 2) match quality ~ (photo session + ambient light + photo session x ambient light). For visual analyses the following additional ANOVA were used: 1) match quality ~ (photo session + gender + photo session x gender), 2) chromatophores expression ~ (photo session and ambient light combinations), and 3) chromatophores expression ~ (photo session + ambient light + photo session x ambient light). We used the Ryan/Einot-Gabriel/Welsch multiple range test for multiple comparisons (P < 0.05) and data were analyzed and examined for normality and homogeneity of variance using GenStat 14 and Systat 11.

Results

Experimental fish had relatively high survival and low tag shedding from the subadult to adult stages in both the low and high light treatments (Table 1). These results allowed us to compare images of at least 218 individuals between photo session pairs using automated matching (Table 1). Delta smelt showed high variability of chromatophores in the dorsal head area, both in terms of individual patterns and their temporal stability (Fig. 2).

Fig. 2
figure 2

Head images of four delta smelt used in visual recognition between photo sessions 1 and 3 under low and high ambient light. Two of the fish shown were matched and two were mismatched

Full size image

Automated image recognition

The success rate of the TinEye API varied widely between photo session pairs and was generally similar between the low and high light treatments, ranging respectively from 18.7–22.5% (sessions 1–3) to 69.2–94.0% (sessions 2–3; Fig. 3a). Success rate for each light treatment differed significantly among all three session pairs (Table 2). Unlike sessions 2–3 which involved only adult fish, the TinEye API was unable to match most of the fish from the subadult to the adult stage (sessions 1–2 and 1–3), but for the matched images it produced only one mismatch throughout this study (sessions 1–2 for the high light treatment). The automated matching success between light treatments of the corresponding photo session pair differed only between sessions 2–3 (z = 6.10, P < 0.001), and for this session pair the TinEye API did not match 7.8% of the fish under low-light and 30.8% of the fish under high-light.

Fig. 3
figure 3

Percent matching among photo sessions of the dorsal head area for delta smelt, a automated matching based on combined cropped combined areas A–B − C, b visual matching based on combined areas A, B and C. Sample size for each comparison is shown in Table 2

Full size image
Table 2 Comparisons between pairs of photo sessions for delta smelt under two light treatments and the proportion of matching success for automated and visual recognition (P-values in bold type denote significant differences)
Full size table

Match quality differed among the six combinations of photo sessions and ambient light (F5, 251 = 7.43, P < 0.001; Fig. 4a). When comparing light treatments for the same photo session pairs, match quality was only different between sessions 2 and 3 (Fig. 4a). Two-way ANOVA showed match quality differed among the photo sessions (F2, 251 = 10.31, P < 0.001) and between low and high ambient light (F1251 = 11.15, P < 0.001), but these two factors showed no interaction (F2, 251 = 2.68, P > 0.05).

Fig. 4
figure 4

Match quality (mean ± SD) among photo sessions of the dorsal head area of delta smelt, a automated analyses for combined cropped areas A–B − C per treatment (n = 60), b visual analyses for combined areas A, B and C (wide bars) and match quality for three individual areas A, B and C (narrow bars) per treatment (n = 30). Significant differences in match quality are denoted by no lower letters in common between pairs of photo sessions

Full size image

Visual image recognition

Testers achieved higher matching success than TinEye among the three photo session pairs, ranging from 73% to 100% of the images (Fig. 3b). The matching success between photo session pairs differed significantly when one of the session pairs included only adult fish and the other session pair included juvenile and adults, irrespective of the light treatment (Table 2). Mismatches occurred only when comparing fish between the subadult and adult stages (sessions 1–2 and 1–3, Fig. 3b). The visual matching success between low and high light treatments did not differ for any of the corresponding photo session pairs (z < 0.29, P > 0.75; Fig. 3b).

The average match quality of areas A, B and C differed significantly among the six combinations of photo sessions and ambient light (F5, 179 = 6.46, P < 0.001; Fig. 4b). Two-way ANOVA further showed match quality was different among photo sessions (F2, 179 = 13.2, P < 0.01) but not between the low and high ambient light (F1, 179 = 3.49, P > 0.06), with no interaction between these factors (F2, 179 = 1.2, P > 0.30). The match quality for areas A, B, and C differed among photo sessions (F2, 119 = 6.91, P < 0.001; Fig. 4b), but not between genders (F1, 195 = 1.38, P > 0.24; males: 2.10 ± 0.57; females: 2.22 ± 0.64, mean ± SD), and no interaction was suggested between photo session and gender (F2, 195 = 3.05, P > 0.05).

Light effect on chromatophores

Despite the 100-fold difference in ambient light between treatments (Fig. 5a-b), both the appearance of chromatophores (Fig. 2) and the direction of change in chromatophores expression (Fig. 5a) was generally similar between treatments. However, the change in chromatophore expression was significant across the six combinations of photo sessions and ambient light (F5, 179 = 29.5, P < 0.001). In contrast to sessions 1–2 and 1–3, chromatophores showed decreased expression between sessions 2 and 3 (Fig. 5a). Two-way ANOVA showed the change in chromatophore expression was significant both among photo session pairs (F2, 179 = 56.6, P < 0.001) and between low and high ambient light (F1, 179 = 8.25, P < 0.01), with interaction between photo session and ambient light (F2, 179 = 13.0, P < 0.001).

Fig. 5
figure 5

a Change in the expression of chromatophores (mean ± SD) of delta smelt per treatment (n = 30) between photo sessions 1–2, 2–3 and 1–3 (no lower case letters in common denote significant differences in chromatophore expression), b estimated ambient light (log10 scale, mean ± SD) to which delta smelt were exposed prior to each photo session (n = 40)

Full size image

Discussion

This study supports the conclusion that the chromatophore patterns on the head of delta smelt are more stable from winter to spring than fall to winter. Therefore, the use of chromatophores as natural marks in delta smelt is more feasible within the adult stage than between subadult and adult stages. Unlilke Castillo et al. (2018), the present study considered a larger interval between photo sessions and two greatly different outdoor ambient light scenarios across all photo sessions, hence the present study is more likely to include the range of ambient light experienced by wild delta smelt. Given the limited individual marking and tagging methods available for this species, the stability of natural marks throughout the adult stage of delta smelt under a wide range of ambient light levels is a potentially useful finding. Moreover, over the winter-spring period, automated recognition of natural marks was 100% accurate in all fish for which matching was possible, with better matching success for the low light treatment (92%) than the high light treatment (62%).

After considering the time interval between photo sessions, light treatments, gender, and the life stages of fish throughout the study period, our analyses support the conclusion that ontogenetic changes between the subadult and adult stages is the likely reason for the higher matching success of images between winter and spring (adult stage only) compared to fall and winter or fall and spring (subadult to adult stages). Such conclusion is consistent with the counterintuitive finding that the three month interval between photo sessions 1–2 (fall–winter) did not result in greatly higher matching success compared to the six month interval between sessions 1–3 (fall–spring; Fig. 3a-b; Table 2).

Despite the reported higher frequency of breeding tubercles on the head of male delta smelt during the spawning season (Wang 2007), fish gender did not influence the visual match quality, which is consistent with the findings of Castillo et al. (2018), who also indicated that changes in spawning condition (spawning vs. non-spawning) between photo sessions did not influence the visual match quality. Hence, our results support the conclusion that chromatophores become stable once adult reach pre-spawning condition around January. Such conclusion is consistent with the 100% visual matching success reported for adult delta smelt between January and May 2013 (Castillo et al. 2018), which was achieved despite a 100-fold increase in ambient light occurring between the last two photo sessions. Surprisingly, no differences in the general appearance of chromatophores were apparent to testers between the fish held at low and high ambient light scenarios. Besides delta smelt, to our knowledge, only Atlantic salmon Salmo salar has been tested using visual recognition of natural marks in the eye and jaw areas and such study achieved a 100% matching success over a period of 2 months (Garcia de Leaniz et al. 1994).

The fact that the TinEye API only mistakenly recognized one fish throughout the three photo sessions (i.e., 0.16% of the fish identified) indicates its algorithm only reports matches when the probability of image recognition is extremely high. Yet, the selectivity of such algorithm resulted in the majority of the individuals being unmatched in session pairs 1–2 and 1–3. The output of the TinEye API only included match quality for matched images. This explains why the reported automated match quality between photo sessions (Fig. 4a) did not differ as much as it would be expected based on the corresponding differences in match success (Fig. 3a). Although we did not compare our TinEye results with other automated image recognition methods, and found no such comparative studies in other species of fish, individual matching success varied up to 27% among three automated image recognition methods tested in the freshwater turtle Pseudemys gorzugi (Suriyamongkol and Mali 2018). Given the large differences in the duration of stable natural marks suggested by automated matching in our study, and other species (e.g., Van Tienhoven et al. 2007; Merz et al. 2012), comparisons across different automated methods for a given species should be ideally performed using the same set of images, or at least the same size ranges and life stages.

Based on the percent of match success, our results and those of Castillo et al. (2018) show the stability of natural marks was consistently higher for visual recognition compared to automated recognition. The 100% visual matching success for the adult stage in both light treatments considered in the present study suggest the potential for practical use of natural marks in delta smelt is highest during winter and spring. Although visual recognition further showed natural marks are stable enough to recognize over 70% of the fish between the subadult and adult stages, a higher matching success is required to consistently apply natural marks between the subadult and adult life stages. The differences in matching success between our visual and automated results, and the advantages of automated methods using large sets of images suggest that additional progress in automated image recognition may prove beneficial. For example, through the use of biometrics (Kühl and Burghardt 2013), in combination with several image identification techniques to improve matching success (e.g., Das et al. 2015). Although preliminary recognition for uncropped head images of delta smelt using TinEye showed the matching success did not significantly increase compared to images of cropped head areas B − C (Tien-Chieh Hung, UC Davis FCCL, unpubl. data), further assessments of the combined potential of natural marks and features such as the contour of the head, could prove valuable, particularly if the depth of field is adjusted to improve image resolution over larger body areas.

Fish exposed to either high or low ambient light had a corresponding small to moderate increase in chromatophores size between sessions 1–2 and 1–3. In contrast, a significantly decreased expression of chromatophores was observed between sessions 2–3 in both treatments. Such constriction of chromatophores is likely due to the seasonal increase in sunlight hours from mid-winter to mid-spring, and this could also account for the observed interaction between photo sessions and light treatments. Similarly, Castillo et al. (2018) reported reduced chromatophore expression between early-spring to mid-spring, but unlike the present study, the increased sunlight in that study entailed the transfer of fish from indoors to outdoors tanks in early spring. Importantly, in neither of these studies the smaller size of chromatophores attributed to increased sunlight influenced the visual matching success despite a 100-fold difference in ambient light scenarios. However, such large differences in illuminance could explain the lower success of automated recognition for the high light treatment in sessions 2–3 (Fig. 3a). Despite the limitations of the two image recognition methods evaluated in our study, these results suggest natural marks could also be feasible in wild delta smelt at least during the adult stage. Nevertheless, the skin coloration of wild delta smelt can become lighter following field capture and handling (V. Afentoulis, California Department of Water Resources; pers. comm.). Compared to the chromatophores expression seen in the present study in both light treatments, the skin coloration of wild adult delta smelt was significantly lighter for all fish photographed following capture by surface trawl nets in the upper San Francisco Estuary (G. Castillo, U.S. Fish and Wildlife Service, Lodi, California; unpubl. data). Moreover, wild delta smelt experience higher handling stress compared to cultured delta smelt (Afentoulis et al. 2013). Thus, it is conceivable that stress-induced chromatophore constriction could be more evident in wild delta smelt (e.g., physiological color change, after Sugimoto 2002). Additional studies are needed to evaluate the potential effect of stress on chromatophore expression and on the post-handling survival of wild delta smelt.

Only one person was required to acquire all images used to evaluate natural marks using visual and automated methods. Assuming 500 fish are used per photo session, it would take about 21 h to acquire all images (i.e., 2.5 min per fish). Unlike automated recognition, visual recognition did not require editing images but the time required for visually matching one fish in a pair of 30 images ranged between 16 and 28 min per tester. In contrast, the time required for automated matching was only about 2 min per fish for a pair of 200 images (including 300 min editing images for two photo sessions, 30 min of upload time and 10 min to run 200 automated comparisons). However, when considering the lowest automated matching success of TinEye (sessions 1–3), the required time per reported fish match would be about 8 min. Thus, the extra time required to edit images prior to obtaining automated matching results may outweigh the time requirements of visual matching per fish even when matching success is low. If the goal is to maximize matching success for small sample sizes, our results suggest that visual recognition still may have advantages over automated recognition.

Although VIA tagging could be the most practical method to identify cultured adult delta smelt (Sandford et al. 2019), smaller fish experience increased tag shedding (Lindberg et al. 2013). In addition to tag loss, VIA tagging may not ensure individual identification due to difficulty in reading tag codes in some species (e.g., Barriga et al. 2015). As in the case of natural marks, the feasibility of VIA tags on wild delta smelt has not been evaluated. Although the range of VIA tag shedding in the present study (0.0 to 1.2%) was similar to that reported for tagging conducted in 2013 (0.3 to 1.7%; Castillo et al. 2018), cultured subadult delta smelt in the present study were effectively tagged at a smaller average size (about 8 mm smaller) than the adult fish in the referred 2013 study. Moreover, post-tagging survival during session 3 was higher in the present study (84–87%; Table 1) than in the 2013 study (78%). These results suggest VIA tags could also be considered to evaluate the effectiveness of natural marks in subadult and adult wild delta smelt.