Square Root Transformation of Data in an Effluent Limit Setting Program

Abstract

Environmental data is generally expected to be lognormally distributed and hence the logarithmetic transformation of the data has been used to calculate effluent limits in various regulatory programs (E.P.A., 1987). The choice of a distribution to characterize a set of data is a common cause of subjective bias and a wrong choice of distribution could lead to improper effluent limits. The problem becomes more acute when some of the data are below the laboratory detection limit (DL) leading to censoring of data. This study is concerned with the case where the square root transformation of the data is more appropriate than the logarithmetic transformation for setting effluent limits. The maximum likelihood estimator of the mean of the raw data is unbiased and is shown to have smaller mean square error than the estimator obtained by transforming back using the parameters of the corresponding distribution. For the estimation of the 99-th percentile, the maximum likelihood estimator, the estimator obtained by transforming back and the unbiased estimator are compared. The 95-th percentile of the mean of four observations is based on the 95-th percentile of a noncentral distribution with four degrees of freedom. When some of the data are below the laboratory detection limit, simulation is used to calculate the 95-th percentile of the mean of four observations. Using Ontario’s Municipal/Industrial Strategy for Abatement (MISA) data, methods of this paper are compared with United States Environmental Protection Agency (EPA) methods, the methods of El-Shaarawi and Dolan for censored data (1989), and methods based on estimation of parameters by probability plots.