## Abstract

In a standard eyewitness lineup scenario, a witness observes a culprit commit a crime and is later asked to identify the culprit from a set of faces, the lineup. Signal detection theory (SDT), a powerful modeling framework for analyzing data, has recently become a common way to analyze lineup data. The goal of this paper is to introduce a new R package, sdtlu (Signal Detection Theory – LineUp), that streamlines and automates the SDT analysis of lineup data. sdtlu provides functions to process lineup data, determine the best-fitting SDT parameters, compute model-based performance measures such as area under the curve (AUC) and diagnosticity, use bootstrapping to determine uncertainty intervals around these parameters and measures, and compare parameters across two different data sets. The package incorporates closed-form solutions for both simultaneous and sequential lineups that allow for model-based analyses without Monte Carlo simulation. Show-ups are also supported. The package can estimate the base-rate of lineups that include a guilty suspect when the guilt or innocence of each suspect in the data set is unknown, as in “real-world” lineups. The package can also produce a full set of graphs, including data and model-based ROC curves and the underlying SDT model.

This is a preview of subscription content, access via your institution.

## Notes

- 1.
We recognize that suspect position can influence identification in both simultaneous and sequential lineups (e.g., Palmer, Sauer, & Holt, 2017). Whereas the sequential model naturally incorporates position effects, the baseline simultaneous model does not, and doing so introduces a host of difficulties because the witness can consider the faces in any order and can return to previously considered faces. To avoid the additional model complexity involved in incorporating position effects into the simultaneous model, we leave that change for future work.

- 2.
Confidence ratings for rejections are not used because it is unclear how one should make these confidence ratings and because confidence is often fairly unsystematic for rejections in data sets.

- 3.
The

*G*^{2}method provides maximum likelihood estimates of parameter values, as*G*^{2}is determined by the likelihood of the data under a given parameter set in the model of interest (the "research" model) and the likelihood of the data in a "full" model in which the probability of each response category matches its proportion in the data. The latter value does not change for different parameter sets, so*G*^{2}differences across parameter sets are determined only by the likelihood of the "research" model. - 4.
Although this should technically be sigma_l, we use sigma_f to avoid confusion with sigma_1.

- 5.
Trial-by-trial data are not used. The data and predictions are collapsed over suspect positions.

- 6.
Note that Gronlund et al. (2009) did not rely on SDT modeling to interpret their data, so noting that their paradigm violated the model’s assumptions is not a criticism of these authors and does not undermine the original purpose of their study.

- 7.
Note that the package code implements the joint, rather than conditional probabilities, for example,

*P*(*resp*=*sus*∩*conf*=*i*) =*P*(*resp*=*sus*∩*conf*=*i*∩*tar*=*pres*) +*P*(*resp*=*sus*∩*conf*=*i*∩*tar*=*abs*),*P*(*resp*=*sus*∩*conf*=*i*∩*tar*=*pres*) =*pP*(*resp*=*sus*∩*conf*=*i*|*tar*=*pres*), and*P*(*resp*=*sus*∩*conf*=*i*∩*tar*=*abs*) = (1 −*p*)*P*(*resp*=*sus*∩*conf*=*i*|*tar*=*abs*). For ease of exposition, the conditionals are presented here, but the results are equivalent. The same transformation holds for all equations below. - 8.
Note that the package code implements the joint, rather than conditional probabilities. See FN 7.

## References

Bothwell, R. K., Deffenbacher, K. A., Brigham, J. C. (1987). Correlation of eyewitness accuracy and confidence: Optimality hypothesis revisited.

*Journal of Applied Psychology, 72*, 691–695.Brewer, N., & Burke, A. (2002). Effects of testimonial inconsistencies and eyewitness confidence on mock-juror judgments.

*Law and Human Behavior*,*26*, 353–364.Carlson, C. A., & Carlson, M. A. (2014). An evaluation of lineup presentation, weapon presence, and a distinctive feature using ROC analysis.

*Journal of Applied Research in Memory and Cognition, 3*, 45–53.Cohen, A.L., Starns, J.J., Rotello, C.M., & Cataldo, A.M. (2020). Estimating the proportion of guilty suspects and posterior probability of guilt in lineups using signal-detection models.

*Cognitive Research: Principles and Implications, 5*, 21. https://doi.org/10.1186/s41235-020-00219-4Colloff, M. F., Wade, K. A., Wixted, J. T., & Maylor, E. A. (2017). A signal-detection analysis of eyewitness identification across the adult lifespan.

*Psychology and Aging, 32*, 243–258. DOI: https://doi.org/10.1037/pag0000168Cutler, B. L., Penrod, S. D., & Dexter, H. R. (1990). Juror sensitivity to eyewitness identification evidence.

*Law and Human Behavior, 14*, 185–191.Deffenbacher, K. A. (1980). Eyewitness accuracy and confidence: Can we infer anything about their relationship? Law and Human Behavior, 4, 243–260. https://doi.org/10.1007/BF01040617

Dobolyi, D. G., & Dodson, C. S. (2013). Eyewitness confidence in simultaneous and sequential lineups: A criterion shift account for sequential mistaken identification overconfidence.

*Journal of Experimental Psychology: Applied, 19*, 345–357.Gonzalez, R., Ellsworth, P. C., & Pembroke, M. (1993). Response biases in lineups and showups.

*Journal of Personality and Social Psychology, 64*, 525–537Gronlund, S. D., Carlson, C. A., Dailey, S. B., Goodsell, C. A. (2009). Robustness of the sequential lineup advantage.

*Journal of Experimental Psychology: Applied, 15*, 140–152. DOI: https://doi.org/10.1037/a0015082Horry, R., Brewer, N., Weber, N., & Palmer, M. A. (2015). The effects of allowing a second sequential lineup lap on choosing and probative value.

*Psychology, Public Policy, and Law, 21*, 121–133. doi:https://doi.org/10.1037/law0000041Horry, R., Palmer, M., & Brewer, N. (2012). Backloading in the sequential lineup prevents within-lineup criterion shifts that undermine eyewitness identification performance.

*Journal of Experimental Psychology: Applied, 18*, 346–360.Levi, A. M. (2012). Much better than the sequential lineup: a 120-person lineup.

*Psychology, Crime & Law, 18*, 631–640. DOI: https://doi.org/10.1080/1068316X.2010.526120Macmillan, N. A., & Creelman, D. C. (2005).

*Detection Theory: A User’s Guide.*New Jersey: Lawrence Erlbaum Associates.McClish, D. K. (1989). Analyzing a portion of the ROC curve.

*Medical Decision Making, 9*, 190–195.Mickes, L. (2015). Receiver operating characteristic analysis and confidence–accuracy characteristic analysis in investigations of system variables and estimator variables that affect eyewitness memory.

*Journal of Applied Research in Memory and Cognition, 4,*93–102. DOI: https://doi.org/10.1016/j.jarmac.2015.01.003Mickes, L., Flowe, H. D., & Wixted, J. T. (2012). Receiver operating characteristic analysis of eyewitness memory: Comparing the diagnostic accuracy of simultaneous versus sequential lineups.

*Journal of Experimental Psychology: Applied, 18*, 361–376.Palmer, M. A., Brewer, N., Weber, N., & Nagesh, A. (2013). The confidence–accuracy relationship for eyewitness identification decisions: Effects of exposure duration, retention interval, and divided attention.

*Journal of Experimental Psychology: Applied, 19*, 55–71. DOI: https://doi.org/10.1037/a0031602Palmer, M. A., Sauer, J. D., & Holt, G. A. (2017). Undermining position effects in choices from arrays, with implications for police lineups.

*Journal of experimental psychology: applied*, 23(1), 71.Police Executive Research Forum. (2013). A national survey of eyewitness identification procedures in law enforcement agencies. Retrieved from https://www.ncjrs.gov/pdffiles1/nij/grants/242617.pdf

Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”.

*BMC Bioinformatics, 7,*77. DOI: https://doi.org/10.1186/1471-2105-12-77Rotello, C. M., & Chen, T. (2016). ROC curve analyses of eyewitness identification decisions: An analysis of the recent debate.

*Cognitive Research: Principles and Implications*, 1:10. DOI https://doi.org/10.1186/s41235-016-0006-7Sing, T., Sander, O., Beerenwinkel, N., & Lengauer, T. (2005). ROCR: visualizing classifier performance in R.

*Bioinformatics, 21*, 3940–3941.Starr, S. J., Metz, C. E., Lusted, L. B., & Goodenough, D. J. (1975). Visual detection and localization of radiographic images.

*Radiology, 116*, 553–538.Swets, J. A., & Pickett, R. M. (1982).

*Evaluation of diagnostic systems: Methods from signal detection theory*. New York: Academic Press.Thomas, E. A. C., & Myers, J. L. (1972). Implications of latency data for threshold and nonthreshold models of signal detection.

*Journal of Mathematical Psychology, 9*, 253–285. DOI: https://doi.org/10.1016/0022-2496(72)90018-1Tunnicliff, J. L., & Clark, S. E. (2000). Selecting foils for identification lineups: Matching suspects or descriptions?

*Law and Human Behavior, 24*, 231–258.Wang, X.-F. (2012). Nonparametric smooth ROC curves for continuous data. R package version 0.1–2.

Wells, G. L. (2014). Eyewitness identification: Probative value, criterion shifts, and policy regarding the sequential lineup.

*Current Directions in Psychological Science, 23*, 11–16. doi:https://doi.org/10.1177/0963721413504781Wells, G. L., & Lindsay, R. C. (1980). On estimating the diagnosticity of eyewitness nonidentifications.

*Psychological Bulletin, 88*, 776–784. doi:https://doi.org/10.1037/0033-2909.88.3.776Wells, G. L., & Olson, E. A. (2003). Eyewitness testimony.

*Annual Review of Psychology, 54,*277–295.Wells, G. L., Rydell, S. M., & Seelau, E. P. (1993). The selection of distractors for eyewitness lineups,

*Journal of Applied Psychology, 78*, 835–844.Wells, G. L., & Turtle, J. W., (1986). Eyewitness identification: The importance of lineup models.

*Psychological Bulletin, 99,*320–329.Wells, G. L., Yang, Y., & Smalarz, L. (2015). Eyewitness identification: Bayesian information gain, base-rate effect equivalency curves, and reasonable suspicion.

*Law and Human Behavior, 39(2)*, 99.Wetmore, S. A., Neuschatz, J. S., Gronlund, S. D., Wooten, A., Goodsell, C. A., Carlson C. A. (2015). Effect of retention interval on showup and lineup performance.

*Journal of Applied Research in Memory and Cognition, 4*, 8–14. DOI: https://doi.org/10.1016/j.jarmac.2014.07.003Wixted, J. T., Mickes, L. (2012). The field of eyewitness memory should abandon “probative value” and embrace receiver operating characteristic analysis.

*Perspectives on Psychological Science, 7*, 275–278. doi:https://doi.org/10.1177/1745691612442906Wixted, J. T., Mickes, L., Dunn, J. C., Clark, S. E., & Wells, W. (2016). Estimating the reliability of eyewitness identifications from police lineups.

*Proceedings of the National Academy of Sciences, 113*, 304-309. https://doi.org/10.1073/pnas.1516814112Wixted, J. T., Vul, E., Mickes, L., & Wilson, B. M. (2018). Models of lineup memory.

*Cognitive Psychology, 105*, 81–114.Wogalter, M. S., Malpass, R. S., & McQuiston, D. E. (2004). A national survey of U.S. police on preparation and conduct of identification lineups.

*Psychology, Crime & Law, 10*, 69 –82. DOI: https://doi.org/10.1080/10683160410001641873

## Acknowledgements

The authors would like to thank Andrea Cataldo for her help collating the example data sets, Tanya Leise for help verifying AUC, and to the authors of the data sets for making them available.

### Open Practices Statement

The R package and all data are available at https://osf.io/mfk4e.

## Author information

### Affiliations

### Corresponding author

## Additional information

### Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Appendices

### Appendix A Equations for simultaneous lineups

Let ϕ(*s*, *μ*, *σ*) be the density of a normal distribution with mean *μ* and standard deviation *σ* and let \( \Phi \left(s,\mu, \sigma \right)=\underset{-\infty }{\overset{s}{\int }}\upphi \left(x,\mu, \sigma \right) dx \) be the cumulative normal.

Let *resp*=response, with values *sus*=suspect, *fil*=filler, and *rej*=reject. Let *conf*=the response confidence level, with values 1…max confidence level. Note that, for notational convenience, 1 is the lowest confidence level here. Let *tar*=target, with values *pres*=present and *abs*=absent.

The following model parameters are used: *p*=P(target present), *μ*_{t}=target distribution mean, *σ*_{t}=target distribution standard deviation, *μ*_{l} =lure distribution mean, *σ*_{l} =lure distribution standard deviation, *c*_{i}=the values of the *i*^{th} response criterion, where *c*_{1} is the lowest response criterion.

The lineup size is given by *l*.

Also see Fig. 1.

### Suspect response

The probability of a suspect response at confidence level *i* is given by^{Footnote 7}

where

and

In Equation A1a finds the overall probability of observing a suspect ID at a given confidence level by taking the weighted average of the corresponding probability for target present and target absent lineups, where the weight is the proportion of target-present lineups (*p*). This equation is needed to get predictions for restricted data, such as data from real lineups, where the guilt status of each suspect is unknown.

Equations A1b and A1c provide the probabilities of choosing the suspect at a given confidence level for target-present and target-absent lineups individually. These equations are used for full data in which each suspect can be classified as guilty or innocent.

In Equation A1b, the first term in the integral is the probability density at a value of *s* for a guilty suspect (i.e., a draw from the target distribution). The second term in the integral is the probability that a given filler (i.e., a draw from the lure distribution) has a strength value below *s*, raised to the power of the number of fillers (*l –* 1) to give the joint probability that all of the fillers have a strength value below *s* (the model assumes that filler strengths are independent of one another and independent of the suspect strength). Multiplying the two terms gives the joint probability density that the suspect has a strength value of *s* and has a higher strength than all of the fillers, i.e., the suspect is selected as the lineup member whose strength value is used to make the identification decision. Integrating this equation between *c*_{i} and *c*_{i+1} gives the joint probability that the suspect both has a strength value in this range and has the highest strength value in the lineup, i.e., the probability of selecting the suspect with confidence level *i*.

In Equation A1c has the same structure as Equation A1b, but the probability density for the suspect is based on the lure distribution to represent a target absent lineup.

### Filler response

The probability of a filler response at confidence level *i* is given by

where

and

Equation A2a finds that the overall probability of selecting a filler at confidence level *i* by taking the weighted average of the corresponding probability for target-present and target-absent lineups, which would be needed for restricted data.

Equations A2b and A2c give the probability of selecting a filler at confidence level *i* for target-present and target-absent lineups individually, so these equations would be used for full data.

For Equation A2b, the first term in the integral is the probability density at strength value *s* for a filler *F*_{1} (i.e., a draw from the lure distribution). The second term in the integral is the probability that one of the other *l* – 2 fillers has a strength value below *s*. Exponentiating provides the joint probability that all of these other fillers all have a strength value below *s*. The third term in the integral is the probability that the suspect (i.e., a random draw from the target distribution) has a strength value below *s*. Multiplying these three terms gives the probability density that *F*_{1} has a strength value of *s* and this strength value is higher than the strength values for all the other fillers and the suspect. That is, *F*_{1} is selected as the lineup member whose strength value will inform the identification decision and has a strength value of *s*. Integrating this equation between *c*_{i} and *c*i+1 gives the joint probability that filler *F*_{1} both has a strength value in this range and has the highest strength value in the lineup, that is, the probability of selecting filler *F*_{1} with confidence *i*. Finally, multiplying this value by the number of fillers (*l –* 1) gives the probability of selecting any of the fillers at confidence level *i*.

Equation A2c has the same structure as Equation A2b, except that the suspect becomes a draw from the lure distribution to represent a target-absent lineup.

### No identification

The probability of rejecting the lineup, that is, not identifying any lineup member as the culprit, is given by

where

and

Equation A3a indicates that the overall probability of rejecting a lineup is the weighted average of the probability of rejection for target-present and target-absent lineups. This value is needed to fit restricted data.

Equations A3b and A3c give the probability of rejection for target-present and target-absent lineups, and so are used for full data.

In Equation A3b, the first integral has the same structure as Equation A1b and gives the probability that the suspect has the highest strength value and a strength value below *c*_{1}. The second integral has the same structure as Equation A2b and gives the probability that a given filler has the highest strength value and a strength value below *c*_{1}, which is multiplied by the number of fillers that could potentially have the highest strength value (*l* – 1). Adding these two terms gives the total probability that the lineup member with the highest strength value (whether suspect or filler) has a strength below *c*_{1}, that is, the probability that the lineup will be rejected.

Equation A3c has the same structure as Equation A3b, except that the suspect is now a draw from the lure distribution to represent a target-absent lineup.

### Appendix B Equations for sequential lineups

Let ϕ(*s*, *μ*, *σ*) be the density of a normal distribution with mean *μ* and standard deviation *σ* and let \( \Phi \left(s,\mu, \sigma \right)=\underset{-\infty }{\overset{s}{\int }}\upphi \left(x,\mu, \sigma \right) dx \) be the cumulative normal.

Let *resp*=response, with values *sus*=suspect, *fil*=filler, and *rej*=reject. Let *conf*=the response confidence level, with values 1…max confidence level. Note that, for notational convenience, 1 is the lowest confidence level here. Let *tar*=target, with values *pres*=present and *abs*=absent. Let *spos*=the subject position, with values=1…lineup size.

The following model parameters are used: *p*=P(target present), *μ*_{t}=target distribution mean, *σ*_{t}=target distribution standard deviation, *μ*_{l} =lure distribution mean, *σ*_{l} =lure distribution standard deviation, *c*_{i}=the values of the *i*^{th} response criterion, where *c*_{1} is the lowest response criterion.

The lineup size is given by *l*.

For the sequential lineups, suspect and filler responses depend on the suspect position.

Also see Fig. 1.

### Suspect response

The probability of a suspect response at confidence level *i* is given by^{Footnote 8}

where

where *P*(*spos* = *j*)is the probability that the suspect is in position j, as determined by the lineup designer and

and

where

Equation B1a provides the overall probability of a suspect ID at confidence level *i*, used for restricted data, by taking the weighted average of the probability of a suspect ID at confidence level *i* for target-present and target-absent lineups. The weight is determined by the proportion of target-present lineups (*p*).

Equations B1b and B1c give the probability of a suspect ID at confidence level *i* for target-present lineups. Equation B1c assumes a given suspect position and Equation B1b calculates this value across the full distribution of suspect positions. In Equation B1c, the first term is the probability that the witness would reach the suspect position (*j*) in the lineup; that is, that all *j –* 1 preceding fillers would have strength values below the identification criterion *c*_{1}. The second term is the probability that the witness would identify the suspect with confidence level *i*, found by integrating the probability density of the target distribution between the criteria defining the bounds of confidence region *i*. Multiplying the two terms gives the probability that the witness would reach the suspect in the lineup sequence and would identify them with confidence level *i* once they do so. Equation B1b takes the weighted average of the values returned by Equation B1c for each suspect position, where the weights are taken from the probability distribution of suspect positions.

Equations B1d and B1e have the same structure as Equations B1b and B1c, except that the suspect strength comes from the lure distribution instead of the target distribution to represent a target-absent lineup.

### Filler response

The probability of a filler response at confidence level *i* is given by

where

where *P*(*spos* = *j*)is the probability that the suspect is in position *j*, as determined by the lineup designer, and

where *k* denotes each lineup position, and

where

Equations B2a, B2b, and B2d are directly analogous to Equations B1a, B1b, and B1d. See the explanation of those equations. Equations B2c and B2e are similar to Equations B1c and B1e, but they give the probability of selecting any of the fillers (as opposed to the single suspect) with a given confidence level.

First consider Equation B2c. The sum goes over all the positions (*k*) in the lineup (1 through *l*). For each position, the bracketed equations give the probability of selecting a filler at confidence level *i* for that position. This value is 0 if the suspect (and not a filler) is in that position (the first “if” statement). Recall that the suspect is in position *j*. The second “if” statement applies to filler positions that come before the suspect position in the lineup. For these positions, the equation is the probability of rejecting all of the *k –* 1 fillers that came before the filler in position *k –* that is, the probability that *k –* 1 random draws from the lure distribution would all fall below the identification criterion *c*_{1} *–* multiplied by the probability of selecting confidence level *i* for the filler in position *k,* which is found by integrating the probability density of the lure distribution between the confidence criteria defining this confidence region. The third “if” statement applies to filler positions that come after the suspect position. The only change from the equation just discussed is that now the probability of getting “past” the faces before position *k* is found by multiplying the probability that each of the *k –* 2 preceding fillers fell below the identification criterion and the probability that the one preceding guilty suspect fell below the identification criterion.

Equation B2e is like Equation B2c, except that, because this is for a target-absent lineup, the suspect term in the third “if” statement is defined by the lure distribution.

### No identification

Lineup rejections do not depend on suspect position. The probability of rejecting the lineup, that is, failing to identify any individual, is given by

where

and

Equation B3a provides the overall probability of rejection across all lineups, used for fitting restricted data, by taking the weighted average of the probability of rejection for target-present and target-absent lineups. Equation B3b gives the probability that all members of a target-present lineup would fall below the identification criterion *c*_{1}, obtained by multiplying the probability that a draw from the target distribution falls below this criterion by the probability that all of the *l – 1* fillers also fall below this criterion. Equation B3c is the same as equation B3b, except now all lineup members are assumed to be draws from the lure distribution.

### Appendix C Equations for AUC and diagnosticity

### Area under the curve

Let *T*_{a} = *P*(*suspect pick* | *target absent*) for a given set of model parameters *θ.* Let \( {T}_a^{\ast } \) be the highest *T*_{a} on the ROC curve, which can be less than 1 for lineups. Let *T*_{p}(*x*) = *P*(*suspect pick* | *target present*) when *T*_{a} = *x*. Then,

Analytic solutions do not typically exist for AUC. Here, AUC is computed numerically using iterative quadrature.

### Diagnosticity

Let ϕ(*s*, *μ*, *σ*) be the density of a normal distribution with mean *μ* and standard deviation *σ* and let \( \Phi \left(s,\mu, \sigma \right)=\underset{-\infty }{\overset{s}{\int }}\upphi \left(x,\mu, \sigma \right) dx \) be the cumulative normal. Let *μ*_{t}=target distribution mean, *σ*_{t}=target distribution standard deviation, *μ*_{l} =lure distribution mean, *σ*_{l} =lure distribution standard deviation, *c*_{i}=the values of the *i*^{th} response criterion, where *c*_{1} is the lowest response criterion. Let the lineup size be given by *l*. Also see Fig. 1.

### Simultaneous lineups

First, consider diagnosticity for simultaneous lineups. Overall diagnosticity, collapsing over response threshold, is given by

and diagnosticity at confidence level *i* is given by

### Sequential lineups

Next, consider diagnosticity for sequential lineups. Overall diagnosticity, collapsing over confidence, is given by

where *P(spos=j)* is the probability of a suspect at position *j*.

The diagnosticity at confidence level *i* is given by

### Appendix D Full sdtlu_compare_2 figure

## Rights and permissions

## About this article

### Cite this article

Cohen, A.L., Starns, J.J. & Rotello, C.M. sdtlu: An R package for the signal detection analysis of eyewitness lineup data.
*Behav Res* **53, **278–300 (2021). https://doi.org/10.3758/s13428-020-01402-7

Published:

Issue Date:

### Keywords

- Eyewitness lineups
- Computational modeling
- Signal detection
- R package