# Cyclostationarity-based DOA estimation algorithms for coherent signals in impulsive noise environments

- 225 Downloads

## Abstract

Estimating direction of arrival (DOA) is important in a variety of practical applications. Conventional cyclostationarity-based coherent DOA estimation algorithms are not robust to non-Gaussian α-stable impulsive noise. Additionally, fractional lower-order statistics (FLOS)-based algorithms are tolerant to impulsive noise; however, they experience performance degradation for coherent signals and interference. To overcome these drawbacks, two types of fractional lower-order cyclostationarity-based subspace DOA estimation methods are proposed for coherent signals in the presence of interference and α-stable impulsive noise. The new proposed algorithms exploit the fractional lower-order cyclostationarity properties of the signals and are immune to the impulsive noise and interference. Moreover, they can provide more accurate DOA estimates of coherent signals than conventional cyclostationarity-based and FLOS-based methods. The simulation results illustrate the robustness and effectiveness of the proposed methods for coherent signals based on a comparison with traditional methods. The new algorithms can be used in the presence of a wide range of interference, Gaussian noise, and α-stable distribution impulsive noise environments.

## Keywords

Cyclostationarity Direction of arrival (DOA) Coherent signals Fractional lower-order statistics α-Stable process## Abbreviations

- AM
Amplitude modulation

- BPSK
Binary phase shift keying

- DOA
Direction of arrival

- FLOM
Fractional lower-order moment

- FLOS
Fractional lower-order statistics

- GSNR
Generalized signal-to-noise ratio

- MUSIC
Multiple signal classification

- PFLOM
Phased fractional lower-order moment

- QPSK
Quaternary phase shift keying

- RMSE
Root-mean-square error

- SIR
Signal-to-interference ratio

- SNR
Signal-to-noise ratio

- SOI
Signal of interest

- SVD
Singular value decomposition

*SαS*Symmetric

*α*-stable distribution- U-ESPRIT
Unitary estimation of signal parameters via rotational invariance techniques

## 1 Introduction

The direction of arrival (DOA) is the base problem in array signal processing and is the core of many civilian and military applications, such as communication regulation enforcement, search-and-rescue operations, and military reconnaissance [1, 2, 3]. In realistic radio environments, correlated signals are not negligible. Evans and Shane et al. proposed derived spatial smoothing techniques for the DOA estimation of coherent signals; in the study, a uniform linear array was divided into several subarrays, and the covariance matrices of the subarrays were then calculated and averaged together [4]. Additionally, subspace smoothing, modified smoothing, and other smoothing methods have been proposed [5, 6, 7, 8]. A new formulation of the Khatri-Rao has been used to estimate DOA which can cope with more coherent signals than classical multiple signal classification (MUSIC) with the spatial smoothing [9]. The unitary estimation of signal parameters via rotational invariance techniques (U-ESPRIT) incorporates forward-backward averaging that can lead to higher resolution than classical ESPRIT [10, 11, 12]. Since the performance of subspace-based algorithms is limited by the number of snapshots, the sparse reconstruction-based methods have been proposed to deal with the DOA estimation of coherent signals [13, 14, 15, 16].

Many artificial modulated communications signals and nature signals have the cyclostationarity properties, which can break through previous limitations and further improve the DOA estimation performance of subspace algorithms. If received signals can be divided into the signal of interest (SOI) and interference (including noise) based on their cyclostationary characteristics, the signals of interest can be selected and the noise and interfering signals can be removed by utilizing cyclostationarity [17]. Therefore, there are many advantages of DOA estimation using cyclostationary characteristics compared to conventional methods [18, 19]. These advantages include selective direction finding, interference and noise suppression, and breakthrough-limited multi-signal processing. Therefore, DOA estimation based on cyclostationarity has been widely studied [20, 21, 22, 23].

Noise, interference, and coherent signals can affect the performance of DOA estimation in wireless communication, radar, sonar, and other systems. The noise and interference can affect the estimation accuracy, and the coherent and correlated signals caused by multipath propagation can, in turn, cause the correlation function to produce several peaks and widen the main peak, which may lead to erroneous estimations by the corresponding algorithms [24]. However, most of the traditional cyclostationarity-based DOA methods for coherent signals assume that the additive noise obeys a Gaussian distribution. Many theoretical studies and experiments have shown that underwater acoustic noise, atmospheric noise, artificial noise, and other noise types contain numerous impulsive components. Previous studies [25, 26, 27] demonstrated that it is inappropriate to model these noises as Gaussian. In fact, the phenomena with impulsive components encountered in practice are appropriate to be modeled as non-Gaussian processes. Although cyclostationarity-based methods are tolerant to coherent signals and Gaussian noise, a significant degradation in their performance can occur when non-Gaussian impulsive noise is present. To suppress the impulsive components, robust fractional lower-order statistics (FLOS)-based DOA estimation methods have been developed [26, 27, 28, 29]. The FLOS-based algorithms can provide accurate DOA estimates for Gaussian and non-Gaussian impulsive noises; however, they suffer from poor performance when interfering signal is present which occupies the same spectral band as the source signal. Recent advances on communications, array processing, and identification have indicated that non-Gaussian impulsive noise can be modeled by *α*-stable distributions [29, 30]. According to the Generalized Central Limit Theorem, the Gaussian distribution is the limiting case of the *α*-stable distribution (*α* = 2). Therefore, the *α*-stable distribution is a more realistic class of distribution than the Gaussian distribution in communication, radar, sonar, and similar application. In real applications, we are interested in developing DOA algorithms that are robust to interference, Gaussian noise, and non-Gaussian *α*-stable distribution noise and that account for the coherent signals in real-world environments.

In this paper, we address the issue of the DOA estimation of cyclostationary coherent signals in impulsive noise environments. Two novel types of robust fractional lower-order cyclostationarity-based signal-selective subspace DOA estimation algorithms for coherent signals are developed. Compared with conventional cyclic subspace DOA estimation algorithms, the proposed methods are not only immune to the interference and Gaussian noise, but can account for impulsive noise and coherent signals. Moreover, because the proposed methods are applicable to coherent signals in the presence of impulsive noise and interference, they are superior to the methods that are merely robust against impulsive noise or suitable for coherent sources. The proposed DOA estimation algorithms have three notable advantages over traditional methods: (1) in the presence of interfering signals, the proposed methods exhibit signal selectivity and performance improvements over the FLOS-based MUSIC and ESPRIT algorithms; (2) in impulsive environments, the proposed methods are more robust to impulsive noise than the cyclic MUSIC and cyclic ESPRIT algorithms; and (3) the proposed methods are effective for cyclostationary coherent signals in impulsive noise and interference environments, while conventional coherent cyclic methods are limited to DOA estimation from Gaussian noise, and FLOS-based coherent methods exhibit degradation in the presence of interference.

## 2 Methods

DOA estimation has been widely used in various important applications (e.g., radar, sonar, communications, and wireless sensor networks) and has garnered considerable attention in recent years. The conventional subspace DOA estimation methods are not robust against non-Gaussian *α*-stable impulsive noise. Although FLOS-based algorithms are robust to the impulsive noise, they do not achieve satisfactory performance in the presence of coband interference. Conventional cyclostationarity-based methods are highly immune to interference; however, they are severely degraded in the presence of *α*-stable impulsive noise and useless for coherent signals. Therefore, we address the problem of DOA estimation for coherent signals in the presence of impulsive noise and interference based on cyclostationarity.

First, we briefly introduce the signal model, cyclic statistics, and *α*-stable distribution. Based on these concepts, we formulate a fractional lower-order cyclostationarity. Since the conventional cyclostationary-based subspace DOA estimation methods are not robust against impulsive noise and coherent signals, we propose a type of spatial smoothing fractional lower-order cyclic algorithm. The proposed spatial smoothing fractional lower-order cyclic algorithms use *p*th-order cyclic statistics to exploit cyclostationarity in impulsive noise and employ spatial smoothing technique to account for coherent signals. Thus, they are applicable to coherent signals in impulsive noise. To further circumvent the coherent signals in impulsive noise, we propose a type of modified fractional lower-order cyclic DOA estimation method. Compared with spatial smoothing fractional lower-order cyclic algorithms, the modified fractional lower-order cyclic algorithms combine the cyclic and conjugate-cyclic statistics together to improve the performance for coherent signals in impulsive noise. Finally, we perform experiments to evaluate the performance of the new proposed algorithms. The designed methods are implemented by using MATLAB. The parameters in the experiments are introduced in Section 5.

## 3 Preliminary

*d*, the number of elements is

*M*, and the wavelength of the signal is

*λ*, the array output can be expressed as

*s*

_{i}(

*t*) is the

*i*th signal received at the sensor, \( {a}_m\left({\theta}_i\right)={e}^{-j\frac{2\pi }{\lambda}\left(m-1\right)d\sin \left({\theta}_i\right)} \) is the

*m*th steering coefficient of the sensor towards the angle

*θ*

_{i}, and

*n*

_{m}(

*t*) is the noise at the

*m*th sensor.

**S**(

*t*) = [

*s*

_{1}(

*t*)

*s*

_{2}(

*t*) ⋯

*s*

_{K}(

*t*)]

^{T}is the signal vector, \( \mathbf{X}(t)={\left[{x}_1(t)\kern0.5em {x}_2(t)\kern0.5em \cdots \kern0.5em {x}_M(t)\right]}^T \) is the vector of signals received by the array sensors, \( \mathbf{A}=\left[{a}_1\left({\theta}_1\right)\kern0.5em {a}_2\left({\theta}_2\right)\kern0.5em \cdots \kern0.5em {a}_M\left({\theta}_K\right)\right] \) is the matrix of array steering vectors, and \( \mathbf{N}(t)={\left[{n}_1(t)\kern0.5em {n}_2(t)\kern0.5em \cdots \kern0.5em {n}_M(t)\right]}^T \) is the noise vector.

In wireless communications, multipath signals are ubiquitous phenomena in radio propagation that can result in signals reaching the antenna array via two or more paths. For example, if a transmitted wave or echo propagates along multiple paths, the received signals are coherent in wireless communications systems. At the same time, the complexity of the propagation path can result in multiple propagation paths between the base station and users are reached, thereby creating a multipath problem. Therefore, it is important to consider efficient signal processing techniques to limit coherent signals.

*s*

_{i}(

*t*) and

*s*

_{j}(

*t*), the correlation coefficients can be written as follows,

*u*

_{i}is a complex.

Many manmade signals encountered in communications, radar, and sonar systems are from a special class of processes whose statistical functions are periodic functions of time, such as signals with amplitude modulation (AM), binary phase shift keying (BPSK), and quaternary phase shift keying (QPSK). These signals are referred to as cyclostationary signals, which exhibit inherent cyclostationarity properties. The cyclostationarity can be used for effective signal processing and make the signals immune to interference and noise, which have different cyclostationarity as that of the signals of interest or are not cyclostationary signals.

*x*(

*t*) is defined by

*ε*represents all the harmonics of the fundamental cycle frequencies of

*x*(

*t*), and “∗” denotes the conjugate operation. The cyclic autocorrelation function is a characteristic property of second-order cyclostationarity. The signal

*x*(

*t*) is said to contain cyclostationarity if and only if \( {R}_x^{\varepsilon}\left(\tau \right)\ne 0 \) for some non-zero cycle frequencies

*ε*. The cyclic spectrum (also called the spectral correlation function) is defined as the Fourier transform of the cycle autocorrelation function:

Furthermore, when *ε* = 0, the cyclic correlation becomes a conventional correlation function, and the cyclic spectrum is the same as the power spectrum.

*α*-stable distribution is appropriate for describing these non-Gaussian noises and has been effectively used to characterize impulsive phenomena. However, the

*α*-stable distribution only has a closed form probability density function in some special cases. The characteristic function of the

*α*-stable distribution is given as follows:

We find that the stable distribution can be determined by four parameters, the characteristic exponent *α* (0 < *α* ≤ 2), symmetric parameter *β* (−1 ≤ *β* ≤ 1), scale parameter *γ* (*γ* > 0), and location parameter *a* (− ∞ < *α* < + ∞). When *β* = 0, the stable distribution is called symmetric *α*-stable distribution (*SαS*). The smaller the characteristic exponent *α* is, the heavier the tails of the *SαS* distribution. The Gaussian distribution is a special case of the *SαS* family (*α* = 2).

Compared with Gaussian noise which has an exponential tail, *α*-stable distributions have algebraic tails. This property makes the second-order or higher order moments of stable distribution non-existence (*α* > 2). In particular, even the first-order moment of it is non-existent when *α* > 1. Thus, the *α*-stable impulsive components of *x*(*t*) make the second-order autocorrelation function *R*_{x}(*t*, *τ*) non-existent. As a result, the cyclic autocorrelation function \( {R}_x^{\varepsilon}\left(\tau \right) \) and spectral correlation function \( {S}_x^{\varepsilon }(f) \) become useless. Therefore, the conventional second-order statistics-based subspace DOA algorithms and the cyclostationarity-based subspace DOA algorithms degrade in the presence of impulsive noise.

*M*= 8, the signal-to-noise ratio (SNR) is 0 dB, the number of snapshots is

*N*= 1024, and the sampling frequency is

*f*

_{s}= 10

^{9}Hz. The SOI is a BPSK signal. The carrier frequency of the BPSK signal is

*f*

_{1}= 0.25

*f*

_{s}, and the symbol rate is

*ε*

_{1}= 0.025

*f*

_{s}. The interfering signal is a QPSK signal with a carrier frequency of

*f*

_{2}= 0.2

*f*

_{s}and a symbol rate of

*ε*

_{2}= 0.02

*f*

_{s}. The angles of incidence of the BPSK and QPSK signals are 20

^{∘}and 50

^{∘}, respectively. The cycle frequency exploited by the cyclic algorithms is

*ε*= 2

*f*

_{1}, which is the cycle frequency of the BPSK signal.

The simulation results indicate that the traditional MUSIC, ESPRIT, cyclic MUSIC, and cyclic ESPRIT algorithms work normally in Gaussian noise. However, the cyclic MUSIC and cyclic ESPRIT are signal selective and can suppress the interfering QPSK signal for only one peak corresponding to the measured DOA. In contrast, MUSIC and ESPRIT cannot suppress the effects of interference. There are two peaks for MUSIC and ESPRIT, one for the DOA of the SOI and one for the DOA of the interference. Although all the algorithms are robust to Gaussian noise, Figs. 1b and 2b show that all the algorithms exhibit severe degradation when impulsive noise is encountered. Moreover, they are not robust to *α*-stable distributed impulsive noise.

^{∘}and 40

^{∘}, respectively. All estimation results are obtained from 10 realizations. Figure 3 shows that cyclic MUSIC and cyclic ESPRIT are not applicable for coherent signals, and they cannot provide accurate DOA estimates for coherent signals.

The simulation results (Figs. 1, 2, and 3) indicate that the cyclic MUSIC and cyclic ESPRIT algorithms can suppress Gaussian noise and interference signals, but cannot be applied to coherent sources and impulsive noise. To overcome the limitations of these cyclic subspace DOA estimation algorithms, it is necessary to exploit the additional signal properties and develop novel signal processing techniques that enable the use of cyclostationarity-based subspace algorithms and are tolerant to coherent signals and robust to impulsive noise.

## 4 Methods of cyclostationarity-based DOA estimation for coherent signals

Because of the uselessness of the conventional cyclic correlation function and the spectral correlation function, conventional cyclic DOA algorithms cannot be effectively applied to impulsive noise. Furthermore, conventional cyclic DOA algorithms cannot account for coherent signals. To circumvent these issues, we develop new DOA estimation algorithms based on fractional lower-order cyclostationarity. The new proposed algorithms share the signal selectivity traits that make them immune to the interference, Gaussian noise, and impulsive noise and are applicable to coherent signals.

### 4.1 The proposed smoothing fractional lower-order cyclic DOA methods

The performance of conventional cyclic DOA algorithms degrades in impulsive noise and coherent signal scenes. It is necessary to develop new cyclostationarity-based DOA estimation algorithms that are tolerant to coherent signals and interference and are robust to both Gaussian noise and impulsive noise.

**C**is the

*K*× 1 complex constant attenuation vector and

**S**(

*t*) =

**C**

*s*(

*t*). The

*K*sources are coherent, and thus, the signals can be described as

*c*

_{1}= 1. The covariance matrix of

**X**(

*t*) can be obtained as follows:

**S**(

*t*) can be given by,

The coherent signals have a certain relationship, that is, they change in a unified way with time, causing the information to be hidden. Therefore, the signal and noise subspaces cannot be constructed by eigenvalue decomposition, and direction estimates based on the conventional feature structure subspace are invalid for coherent signals.

*M*(

*M*>

*K*) element uniform line array is divided into

*L*uniformly overlapping subarrays of dimension

*Q*(

*Q*>

*K*), in such a way that each sub-array shares all but one of its sensors with an adjacent sub-array [6]. The signal of the

*l*th sub-array is

**A**

_{M}is the direction matrix of

*Q*×

*K*dimension, which is the

*Q*dimensional steering vector

*a*

_{M}(

*θ*

_{i}) (

*i*= 1, 2, ⋯,

*K*), and \( \mathbf{D}=\operatorname{diag}\left({e}^{j\frac{2\pi }{\lambda}\sin {\theta}_1},{e}^{j\frac{2\pi }{\lambda}\sin {\theta}_2},\cdots, {e}^{j\frac{2\pi }{\lambda}\sin {\theta}_K}\right) \).

*p*th-order cyclostationarity has been developed. Considering a cyclostationary signal

*x*(

*t*), the

*p*th-order correlation of

*x*(

*t*) is defined by

*p*th-order phased fractional lower-order moment (PFLOM) is defined as

*z*

^{〈p〉}= |

*z*|

^{p − 1}

*z*. The PFLOM can be re-expressed in a polar form (

*z*=

*re*

^{jθ}) as follows:

Note that the PFLOM acts only on the magnitude of the operand and preserves the corresponding phase; thus, *z*^{〈p〉} has the same period as *z*, and *x*(*t*) and (*x*(*t*))^{〈p〉} have the same period.

*ε*represents all cycle frequencies of

*x*(

*t*). The Fourier coefficient \( {R}_x^{\varepsilon, p}\left(\tau \right) \) is referred to as the

*p*th-order cyclic correlation function and given as follows:

*p*th-order and stable process theory [31], the covariation 〈

*x*(

*t*+

*τ*/2)[

*x*

^{∗}(

*t*−

*τ*/2)]

^{〈p〉}〉 is robust to the impulsive noise when 1 ≤

*p*<

*α*; thus, the

*p*th-order cyclic correlation function can refrain the effects of impulsive noise. In fact, the cyclic correlation function can be represented as the cross-correlation of frequency-shifted versions of \( {R}_x^{\varepsilon}\left(\tau \right)=\left\langle u\left(t+\tau /2\right){v}^{\ast}\left(t-\tau /2\right)\right\rangle \) where

*u*(

*t*) =

*x*(

*t*)

*e*

^{−jπεt}and

*v*(

*t*) =

*x*(

*t*)

*e*

^{jπεt}. By using the properties of

*z*

^{〈k〉}

*e*

^{±jπεt}= (

*ze*

^{±jπεt})

^{〈k〉}and (

*z*

^{∗})

^{〈k〉}= (

*z*

^{〈k〉})

^{∗}, the

*p*th-order cyclic correlation function (19) can be rewritten as

Based on Eq. (17), the PFLOM acts only on the magnitude of *v*^{'}(*t*), and *v*^{'}(*t*) has the same period as that of *v*(*t*). Therefore, the *p*th-order cyclic correlation function is an alternative but equivalent characterization of the second-order cyclic function. Furthermore, compared to conventional second-order cyclic correlation function, *p*th-order cyclic correlation function is robust to *SαS* distributed (1 < *α* ≤ 2) impulsive noise. When *p* = 2, the *p*th-order cyclic autocorrelation function \( {R}_x^{\varepsilon, p}\left(\tau \right) \) becomes the traditional cyclic autocorrelation function \( {R}_x^{\varepsilon}\left(\tau \right) \). When the cycle frequency *ε* = 0, the *p*th-order cyclic autocorrelation function \( {R}_x^{\varepsilon, p}\left(\tau \right) \) reduces to the *p*th-order correlation function \( {R}_x^p\left(\tau \right) \). The *p*th-order cyclic correlation function plays an essential role in high-resolution direction finding.

*l*th sub-array is obtained as

*ε*is one cycle frequency of

*s*(

*t*). The spatial smoothing fractional lower-order cyclic covariance matrix is given by

**U**

_{S}and

**U**

_{Q}. The spectral estimation is achieved based on the orthogonality of

**A**(

*θ*) and

**U**

_{Q},

Then, the DOA of the proposed smoothing fractional lower-order cyclic MUSIC algorithm can be obtained from a search for peaks in the spectrum of (25).

**U**

_{S}and

**U**

_{N}can be obtained from (25). To develop the smoothing fractional lower-order cyclic ESPRIT algorithm,

**U**

_{S}is divided into

**U**

_{S1}and

**U**

_{S2}. Specifically, the relation between

**U**

_{S1}and

**U**

_{S2}can be determined by

**Ψ**=

**T**

^{−1}

**ΦT**. Additionally,

*μ*

_{k}=

*ω*

_{0}Δ sin

*θ*

_{k}/

*c*(

*k*= 1, 2, … ,

*K*). Note that (28) indicates that the characteristic value of

**Ψ**is a diagonal element of

**Φ**. Therefore, after obtaining \( \boldsymbol{\Psi} ={\mathbf{U}}_{S1}^{+}{\mathbf{U}}_{S2} \), the eigenvalues and diagonal elements of

**Φ**can be obtained. The DOA estimate can be obtained by,

*λ*

_{k}is the eigenvalue of

**Ψ**.

### 4.2 The proposed modified fractional lower-order cyclic DOA methods

**X**

^{∗}(

*t*) is the complex conjugation of

**X**(

*t*), and

**J**

^{2}=

**I**; therefore, the correlation matrix of

**Y**(

*t*)

**= JX**

^{∗}(

*t*) is given by

**Y**(

*t*) is defined in the following form:

**R**

^{ε, p}(

*τ*) is not a Hermitian matrix, we can use SVD to decompose the covariance matrix

**R**

^{ε, p}(

*τ*),

*K*

_{a}non-zero singular values of

**R**

^{ε, p}(

*τ*), and the other

*M*−

*K*

_{a}singular values equals to zero. For rank(

**R**

^{ε, p}(

*τ*)) =

*K*

_{a},

**U**and

**V**can then be divided,

**U**

_{S}and

**U**

_{N}can be obtained from (35). The modified fractional lower-order cyclic MUSIC method is obtained using the orthogonality of

**A**(

*θ*) and

**U**

_{N},

Thus, the DOA can be obtained from a search for the peaks in the spectrum of (38).

**Y**(

*t*) is defined as the product of

**J**and the complex conjugate of

**X**

^{∗}(

*t*). Based on the SVD of

**R**

^{ε, p}(

*τ*), the signal subspace

**U**

_{S}and noise subspace

**U**

_{N}are determined. Then, we can extract

**U**

_{S1}and

**U**

_{S2}from

**U**

_{S}. According to the property of rotational invariance, we obtain,

**Ψ**is the transformation matrix \( \boldsymbol{\Psi} ={\mathbf{U}}_{S1}^{+}{\mathbf{U}}_{S2} \).

**R**

^{ε, p}(

*τ*) is nonsingular, the space of

**A**and

**U**

_{S}is the same, and the linear transformation matrix of

**A**and

**U**

_{S}is

**T**. By establishing the relation between the rotationally invariant factor

**Φ**and the transformation matrix

**Ψ**, we obtain the following equation,

**Φ**by finding the eigenvalue of

**ψ**, that is the DOA estimates is obtained as

## 5 Simulation results and discussion

In this section, we present various simulation results that are used to assess the performance of the proposed algorithms. We verify the effectiveness and robustness of the proposed smoothing fractional lower-order cyclic MUSIC (SFCyclic-MUSIC), smoothing fractional lower-order cyclic ESPRIT (SFCyclic-ESPRIT), modified fractional lower-order cyclic MUSIC (MFCyclic-MUSIC), and modified fractional lower-order cyclic ESPRIT (MFCyclic-ESPRIT) algorithms and compare their performance to the performance of four classes of existing FLOS-based, cyclostationarity-based, sparse representation algorithms, namely, the fractional lower-order moment-based MUSIC (FLOM-MUSIC) and ESPRIT (FLOM-ESPRIT), the modified MUSIC (M-MUSIC) and ESPRIT (M-ESPRIT), the cyclic MUSIC (Cyclic-MUSIC) and ESPRIT (Cyclic-ESPRIT), the Bayes-optimal algorithm [24], and conventional sparse Bayesian learning (SBL) algorithms for those frequently encountered communication signals (e.g., BPSK and QPSK).

*α*-stable distribution process has finite variance only for

*α*= 2, the traditional SNR is inappropriate for determining the power of

*SαS*noise. Thus, we use a generalized signal-to-noise ratio (GSNR) which is defined as [27],

*SαS*impulsive noise samples are power scaled by the dispersion parameter

*γ*. The performance of the algorithms is evaluated by the root-mean-square error (RMSE) of the DOA estimates and the probability of resolution. The RMSE is defined by

The probability of resolution is also called the success probability which is defined as the ratio of the number of successful runs to the total number of Monte Carlo runs. The number of arrays is *M* = 10, the number of the snapshots is *N* = 1024, and the sampling frequency is *f*_{s} = 10^{9} Hz.

### 5.1 DOA estimation of the proposed methods

In this case, the incident signals are *s*_{1}(*t*), *s*_{2}(*t*), *s*_{3}(*t*), and *s*_{4}(*t*) with DOA(*θ*_{1}, *θ*_{2}, *θ*_{3}, *θ*_{4}) = (10^{∘}, 15^{∘}, 40^{∘}, 50^{∘}). The carrier frequency and symbol rate of the coherent BPSK signals of *s*_{1}(*t*) and *s*_{2}(*t*) are *f*_{1} = 0.25*f*_{s} and *ε*_{1} = 0.025*f*_{s}, respectively. *s*_{3}(*t*) is a BPSK signal with a carrier frequency of *f*_{2} = 0.25*f*_{s} and keying rate of *ε*_{2} = 0.025*f*_{s}. *s*_{4}(*t*) is a QPSK signal with a carrier frequency and symbol rate of *f*_{3} = 0.1*f*_{s} and *ε*_{3} = 0.05*f*_{s}, respectively. *s*_{3}(*t*) and *s*_{4}(*t*) are independent of *s*_{1}(*t*) and *s*_{2}(*t*). The cycle frequency used by the cyclic algorithms is *ε* = 2*f*_{1}, which is the cycle frequency of *s*_{1}(*t*), *s*_{2}(*t*), and *s*_{3}(*t*).

*α*= 1.6 (GSNR = 10 dB) is shown in Figs. 4 and 5. The simulation results indicate that all the proposed methods are robust to impulsive noise. However, the MFCyclic-MUSIC and MFCyclic-ESPRIT methods make better use of fractional lower-order cyclostationarity and conjugate fractional lower-order cyclic cyclostationarity and are superior to the SFCyclic-MUSIC and SFCyclic-ESPRIT methods. Figure 5 illustrates that the MFCyclic-MUSIC and MFCyclic-ESPRIT methods can still suppress extremely impulsive noise (

*α*= 1.6) and provide accurate DOA estimates for cyclostationary coherent sources in the presence of impulsive noise and interference signals.

### 5.2 Influence of impulsive noise

We consider a BPSK communication signal representing the SOI. The carrier frequency of the BPSK signal is *f*_{c} = 0.25*f*_{s}, the keying rate is *ε*_{k} = 0.0625*f*_{s}, and the DOA of the signal is 10^{∘}. The interference in this case is a BPSK signal with a carrier frequency of *f*_{1} = 0.2*f*_{s}, keying rate of *ε*_{1} = 0.025*f*_{s}, and a DOA of 20^{∘}. The signal-to-interference ratio (SIR) is 3 dB. We use a cycle frequency of *ε* = *ε*_{k} = 0.0625/*T*_{s} and *p* = 1.2. The estimation results are obtained from 1000 Monte Carlo realizations.

*α*is conducted. The simulation scenario includes a GSNR = 10 dB and

*p*= 1.1. As shown in Fig. 6a and 6b, the impulsive characteristics of the stable distribution are weakened as α increases, and the performance of all the algorithms improved. Moreover, the simulation results indicate that MFCyclic-MUSIC and MFCyclic-ESPRIT slightly outperformed SFCyclic-MUSIC and SFCyclic-ESPRIT when the impulsive characteristics are strong. When

*α*≥ 1.9, the two types of algorithms are basically the same.

### 5.3 Influence of coherent signals

In this case, we consider two coherent QPSK communication signals. The carrier frequency of the QPSK signals is *f*_{c} = 0.2*f*_{s}, and the keying rate is *ε*_{k} = 0.025*f*_{s}. The DOA of the signals are 10^{∘} and 15^{∘}, and the angle to be estimated is 10^{∘}. The coband interference is an AM signal that has the same carrier frequency and bandwidth as that of the QPSK signals. The DOA of the AM signal is 20^{∘}. The cycle frequency exploited by the cyclic algorithm is set to *ε* = *ε*_{k} and GSNR = 0 dB. The estimation results are obtained from 1000 Monte Carlo realizations.

*α*= 1.5) and in the coband interfering environments, respectively. Because the FLOM-MUSIC cannot estimate coherent signals, it can be seen from Fig. 10 that the FLOM-MUSIC achieves the worst performance. Moreover, comparing Fig. 10a with b, we can see that the Bayes-optimal algorithm achieves the best suppression performance for the impulsive noise and the proposed method outperforms Bayes-optimal algorithm against interference.

Cycle frequency is one of the most important representation of cyclostationarity. Typical cycle frequencies for many communications signals include the double carrier frequency, the keying rate and the corresponding harmonics, as well as the sums and differences of these cycle frequencies. For example, for a BPSK signal or a QPSK signal, the cycle frequencies are ±2*f*_{c}, *ε*_{k}, and *ε*_{i} = ± 2*f*_{c} + *iε*_{k} (*i* = 0, ± 1, ± 2, …), where *f*_{c} is the carrier frequency and *ε*_{k} represents the keying rate. If we use the cycle frequency at which the spectral correlation is stronger, the performance of the proposed cyclic algorithms can be improved.

*α*= 1.8 is added only in the simulation. The simulation results indicate that the proposed methods perform effectively at the two cycle frequencies. However, for the QPSK signal, the cyclostationarity characteristic is stronger at

*ε*= 2

*f*

_{c}than at

*ε*=

*ε*

_{k}, so the performance of the two proposed algorithms is slightly better at

*ε*= 2

*f*

_{c}than at

*ε*=

*ε*

_{k}.

## 6 Conclusions

In this paper, new fractional lower-order cyclostationarity-based DOA estimation algorithms for coherent signals in the presence of *α*-stable impulsive noise and interference are introduced. Traditional second-order cyclostationarity-based and FLOS-based DOA estimation methods severely degrade for coherent signals, under *α*-stable distributed impulsive noise and interfering signals. By exploiting the fractional lower-order cyclostationarity properties of signals, new smoothing fractional lower-order cyclic subspace algorithms and modified fractional lower-order cyclic subspace algorithms are proposed. The new proposed algorithms not only provide the signal selectivity in the presence of impulsive noise, but also are highly tolerant to coherent signals. Moreover, these algorithms yield more reliable DOA estimates of cyclostationary coherent signals than do cyclostationarity-based and FLOS-based methods in the presence of interference and impulsive noise. The effectiveness and robustness of the proposed algorithms are evaluated via simulations, and the results indicate that the new algorithms are applicable to a wide range of interference, Gaussian noise, and non-Gaussian impulsive noise environments for coherent signals.

## Notes

### Acknowledgements

The authors would like to thank the editors and anonymous reviewers. The authors are grateful to the National Science Foundation of China for its support of this research. This work is supported by the National Science Foundation of China under Grant 61461036, 61761033, and 61501325.

### Funding

This work is supported by the National Science Foundation of China under Grant 61461036, 61761033, and 61501325.

### Availability of data and materials

Not applicable.

### Authors’ contributions

The algorithms proposed in this paper have been conceived by Dr. YL, Dr. YZ, and Prof. TQ. Dr. YL, Dr. YZ, and Dr. JG designed the experiments. Dr. JG and B.S. QW performed the experiments and analyzed the results. Dr. YL and Dr. YZ wrote the paper. All authors read and approved the final manuscript.

### Competing interests

The authors declare that they have no competing interests.

### Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## References

- 1.H. Krim, M. Viberg, Two decades of array signal processing research: the parametric approach. IEEE Signal Process Mag
**13**(4), 67–94 (1996)CrossRefGoogle Scholar - 2.B. Ottersten, M. Viberg, T. Kailath, Performance analysis of the total least squares ESPRIT algorithm. IEEE Trans Signal Process
**39**(5), 1122–1135 (1991)zbMATHCrossRefGoogle Scholar - 3.A. Liu, F. Li, B. Li, et al., Spatial polarimetric time-frequency distribution based DOA estimation: combing ESPRTI with MUSIC. EURASIP J Wirel Commun Netw
**2018**(51), 1–8 (2018)Google Scholar - 4.M. TJ Shan, T.K. Wax, On spatial smoothing for direction-of-arrival estimation of coherent signals. IEEE Trans Acoust Speech Signal Process
**33**(4), 806–811 (1985)CrossRefGoogle Scholar - 5.H. Shi, Z. Li, D. Liu, H. Chen, Efficient method of two-dimensional DOA estimation for coherent signals. EURASIP J Wirel Commun Netw
**2017**(53), 1–10 (2017)Google Scholar - 6.D. W, R.L. Kirlin, Improved spatial smoothing techniques for DOA estimation of coherent signals. IEEE Trans Signal Process
**39**(5), 1208–1210 (1991)CrossRefGoogle Scholar - 7.H. Shi, Z. Li, J. Cao, H. Chen, Two-dimensional DOA estimation of coherent sources using two parallel uniform linear arrays. EURASIP J Wirel Commun Netw
**2017**(60), 1–9 (2017)Google Scholar - 8.D. Kundu, Modified MUSIC algorithm for estimating DOA of signals. Signal Process
**48**(1), 85–90 (1996)MathSciNetCrossRefGoogle Scholar - 9.M. Inoue, K. Hayashi, H. Mori, T. Nabetani, A DOA estimation method with Kronecker subspace for coherent signals. IEEE Commun Lett
**22**(11), 2306–2309 (2018)CrossRefGoogle Scholar - 10.Z. Liu, J. Wang, F. Wang, Conjugate unitary ESPRIT for real sources adapted to the coherent case. Signal Process
**89**(7), 1403–1411 (2009)zbMATHCrossRefGoogle Scholar - 11.T. Wang, B. Ai, R. He, Z. Zhong, Two-dimension direction-of-arrival estimation for massive MIMO systems. IEEE Access
**2015**(3), 2122–2128 (2015)CrossRefGoogle Scholar - 12.Y. Zhou, Z. Fei, S. Yang, J. Kuang, S. Chen, L. Hanzo, Joint angle estimation and signal reconstruction for coherently distributed sources in massive MIMO systems based on 2-D unitary ESPRIT. IEEE Access
**2017**(5), 9632–9646 (2017)CrossRefGoogle Scholar - 13.W. Cui, T. Qian, J. Tian, Enhanced covariances matrix sparse representation method for DOA estimation. Electron Lett
**51**(16), 1288–1290 (2015)CrossRefGoogle Scholar - 14.J. Cai, D. Bao, P. Li, DOA estimation via sparse recovering from the smoothed covariance vector. J Syst Eng Electron
**27**(3), 555–561 (2016)CrossRefGoogle Scholar - 15.B. Li, X. Wang, Sparse representation-based DOA estimation of coherent wideband LFM signals in FRFT domain. EURASIP J Wirel Commun Netw
**2017**(210), 1–9 (2017)Google Scholar - 16.J. Liu, W. Zhou, X. Wang, D. Huang, A sparse direction-of-arrival estimation algorithm for MIMO radar in the presence of gain-phase errors. Digital Signal Process
**2017**(69), 193–203 (2017)CrossRefGoogle Scholar - 17.W.A. Gardner, A. Napolitano, L. Paura, Cyclostationarity: half a century of research. Signal Process
**86**(4), 639–697 (2006)zbMATHCrossRefGoogle Scholar - 18.A. Napolitano, Generalizations of cyclostationarity: a new paradigm for signal processing for mobile communications, radar, and sonar. IEEE Signal Process Mag
**30**(6), 53–63 (2013)CrossRefGoogle Scholar - 19.A. Napolitano, Cyclostationarity: limits and generalizations. Signal Process
**120**(3), 323–347 (2016)CrossRefGoogle Scholar - 20.W.A. Gardner, Simplification of MUSIC and ESPRIT by exploitation of cyclostationarity. Proc IEEE
**76**(7), 845–847 (1988)CrossRefGoogle Scholar - 21.G. Xu, T. Kailath, Direction of arrival estimation via exploitation of cyclostationary - a combination of temporal and spatial processing. IEEE Trans Signal Process
**40**(7), 1775–1786 (1992)zbMATHCrossRefGoogle Scholar - 22.H.Q. Yan, H.H. Fan, Signal-selective DOA tracking for wideband cyclostationary sources. IEEE Trans Signal Process
**55**(5), 2007–2015 (2007)MathSciNetzbMATHCrossRefGoogle Scholar - 23.W.J. Zeng, X.L. Li, X.D. Zhang, X. Jiang, in
*IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei*. An improved signal-selective direction finding algorithm using second-order cyclic statistics (2009), pp. 2141–2144Google Scholar - 24.J.S. Dai, H.C. So, Sparse Bayesian learning approach for outlier-resistant direction-of-arrival estimation. IEEE Trans Signal Process
**66**(3), 744–756 (2018)MathSciNetzbMATHCrossRefGoogle Scholar - 25.W.J. Zeng, H.C. So, Outlier-robust matrix completion via ℓ
_{p}-minimization. IEEE Trans Signal Process**66**(5), 1125–1140 (2018)MathSciNetzbMATHCrossRefGoogle Scholar - 26.H. Belkacemi, S. Marcos, Robust subspace-based algorithms for joint angle/Doppler estimation in non-Gaussian clutter. Signal Process
**87**(7), 1547–1558 (2007)zbMATHCrossRefGoogle Scholar - 27.P. Tsakalides, C.L. Nikias, The robust covariation-based MUSIC (ROC-MUSIC) algorithm for bearing estimation in impulsive noise environments. IEEE Trans Signal Process
**44**(7), 1623–1633 (1996)CrossRefGoogle Scholar - 28.T.H. Liu, J.M. Mendel, A subspace-based direction finding algorithm using fractional lower order statistics. IEEE Trans Signal Process
**49**(8), 1605–1613 (2001)MathSciNetzbMATHCrossRefGoogle Scholar - 29.M. Rupi, P. Tsakalides, E.D. Re, C.L. Nikias, Robust spatial filtering of coherent source for wireless communications. Signal Process
**80**(3), 381–396 (2000)zbMATHCrossRefGoogle Scholar - 30.Y. Liu, Y.H. Zhang, T.S. Qiu, J. Gao, S. Na, Improved time difference of arrival estimation algorithms for cyclostationary signals in
*α*-stable impulsive noise. Digital Signal Process**2018**(76), 94–105 (2018)MathSciNetCrossRefGoogle Scholar - 31.S. Cambanis, G. Miller, Linear problems in
*p*th order and stable processes. SIAM J Appl Math**41**(1), 43–69 (1981)MathSciNetzbMATHCrossRefGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.