Attractor Basin Analysis of the Hopfield Model: The Generalized Quadratic Knapsack Problem

Abstract

The Continuous Hopfield Neural Network (CHN) is a neural network which can be used to solve some optimization problems. The weights of the network are selected based upon a set of parameters which are deduced by mapping the optimization problem to its associated CHN. When the optimization problem is the Traveling Salesman Problem, for instance, this mapping process leaves one free parameter; as this parameter decreases, better solutions are obtained. For the general case, a Generalized Quadratic Knapsack Problem (GQKP), there are some free parameters which can be related to the saddle point of the CHN. Whereas in simple instances of the GQKP, this result guarantees that the global optimum is always obtained, in more complex instances, this is far more complicated. However, it is shown how in the surroundings of the saddle point the attractor basins for the best solutions grow as the free parameter decreases, making saddle point neighbors excellent starting point candidates for the CHN. Some technical results and some computational experiences validate this behavior.

A Some Technical Results of the Illustrative Example

Recalling the GQKP problem introduced in Sect. 2, a stability analysis of the valid solutions is carried out, looking for the parameters that guarantee that the CHN is stable in the feasible solutions. The energy function obtained for the simple GQKP problem (see Eq. 3) was:

$$\begin{aligned} \begin{array}{ll} E\left( \mathbf {v}\right) &{} = \alpha {}\, \left( 2\, {v_{1}}^2 - {v_{2}}^2\right) + \frac{1}{2} \phi _{1,1} \left( v_{1} + v_{2}\right) ^2 - \gamma _{1} v_{1} \left( v_{1} - 1\right) - \gamma _{2} v_{2}\left( v_{2} - 1\right) \\ &{} {=} + \beta _{1} \left( v_{1} + v_{2}\right) \end{array} \end{aligned}$$

Referencing Talaván and Yáñez [12], the stability of any $\mathbf {v} \in H$ is ensured if:

$$\begin{aligned} \min _{\mathbf {v} \in H_F} \underline{E}^0(\mathbf {v}) \ge 0 \end{aligned}$$

(4)

$$\begin{aligned} \max _{\mathbf {v} \in H_F} \overline{E}^1(\mathbf {v}) \le 0 \end{aligned}$$

(5)

$$\begin{aligned} E_{n+k}(\mathbf {v}) = 0 \quad \forall k \in \{1,\ldots ,m_1\}{\ /}\ v_{n+k} \in (0,1) \quad \forall \mathbf {v} \in H_F \end{aligned}$$

(6)

where

$$\begin{aligned} E_i(\mathbf {v}) \equiv \displaystyle \frac{\partial E(\mathbf {v})}{\partial v_i} \qquad \underline{E}^0(\mathbf {v}) \equiv \displaystyle \min _{v_i = 0}E_{i}(\mathbf {v}) \qquad \overline{E}^1(\mathbf {v}) \equiv \displaystyle \min _{v_i = 1} E_{i}(\mathbf {v}) \end{aligned}$$

Although analyzing all feasible solutions would be a very difficult task with a GQKP, this simple example allows rigorous analysis. Thus, the partial derivatives of the energy function will be of the form:

$$\begin{aligned} \begin{array}{ll} E_{1}\left( \mathbf {v}\right) &{}= {-} 4\alpha v_{1} + \phi _{1,1} v_{1} + \phi _{1,1} v_{2} - 2\gamma _{1}v_{1} + \gamma _{1} + \beta _{1}\\ E_{2}\left( \mathbf {v}\right) &{}=-2\alpha v_{2} + \phi _{1,1} v_{1} + \phi _{1,1} v_{2} - 2\gamma _{2}v_{2} + \gamma _{2} + \beta _{1} \end{array} \end{aligned}$$

Condition 4 is satisfied if:

$$\begin{aligned} \begin{array}{rcl} \underline{E}^{0}( \left[ {\begin{matrix}1\\ 0\end{matrix}}\right] ) &{} = &{} \min \{E_{2}( \left[ {\begin{matrix}1\\ 0\end{matrix}}\right] )\} = \phi _{1,1} + \gamma _{2} + \beta _{1} \ge 0 \\ \underline{E}^{0}( \left[ {\begin{matrix}0\\ 1\end{matrix}}\right] ) &{} = &{} \min \{ E_{1}( \left[ {\begin{matrix}0\\ 1\end{matrix}}\right] ) \} = \phi _{1,1} + \gamma _{1} + \beta _{1} \ge 0 \end{array} \end{aligned}$$

and Condition 5 is satisfied if:

$$\begin{aligned} \begin{array}{rcl} \overline{E}^{1}( \left[ {\begin{matrix}1\\ 0\end{matrix}}\right] ) &{} = &{} \max \{E_{1}( \left[ {\begin{matrix}1\\ 0\end{matrix}}\right] )\} = {-}4\alpha + \phi _{1,1} - \gamma _{1} + \beta _{1} \le 0 \\ \overline{E}^{1}( \left[ {\begin{matrix}0\\ 1\end{matrix}}\right] ) &{} = &{} \max \{ E_{2}( \left[ {\begin{matrix}0\\ 1\end{matrix}}\right] ) \} = -2\alpha + \phi _{1,1} -\gamma _{2} + \beta _{1} \le 0 \end{array} \end{aligned}$$

This example does not use Condition 6 as no slack variables are needed.

The instability of any interior point $\mathbf {v} \in H \setminus H_C$ is guaranteed if:

$$ T_{i,i} \ge 0 \quad \forall i \in \{1,\dots ,n\} $$

and the stability of valid solutions is obtained if $\mathbf {\varPhi }$ is positive semidefinite:

$$\phi _{k,l} \ge 0 \quad \forall k,l \in \{1,\dots ,m\}$$

The instability of any non-feasible corner is obtained by creating a partition of the set $H_{C}\setminus H_{F}$ and forcing the instability conditions for each of the elements in the partition:

$$ \vee \left\{ \begin{array}{rcl} \underline{E}^0(\mathbf {v}) &{} \le &{} - \varepsilon \\ \overline{E}^1(\mathbf {v}) &{} \ge &{} \varepsilon \\ E_{n+k}(\mathbf {v}) &{} \ne &{} 0 \quad \text {for any } v_{n+k} \in (0,1) \end{array} \right. \qquad \text {with } \varepsilon > 0$$

Going forward the partition $H_C \setminus H_F$ is created using the direct method introduced by Talaván [11]. In summary, this partition is created considering the different cases which may occur when the constraints for $H_F$ are violated. Thus, given $\mathbf {v} \in H_C \setminus H_F$, the instability will be found from the first unsatisfied constraint, distinguishing inequalities and equations (which may not be satisfied by excess or defect):

$$H_C \setminus H_F = \bigcup _{k=1}^{m_{1}}W_{k,0} \bigcup _{k=1}^{m}W_{k,1} \bigcup _{k=1}^{m}W_{k,2} $$

where

For the GQKP problem being studied, the partition gets reduced to:

$$ H_C \setminus H_F = W_{1,1} \cup W_{1,2} $$

where

• $W_{1,1} = \{e_{1}(\mathbf {v})\> 1\} = \{ v_{1} + v_{2} > 1\}$, which is satisfied if $v_{1} = 1$ and $v_{2} = 1$:

  $$ \left\{ \begin{array}{l} E_{1}(\mathbf {v})> {-}4\alpha + 2\phi _{1,1} - \gamma _{1} + \beta _{1}\\ E_{2}(\mathbf {v}) > -2\alpha + 2\phi _{1,1} - \gamma _{2} + \beta _{1} \end{array} \right. $$

  and the instability is guaranteed if:

  $$ \vee \left\{ \begin{array}{l} {-}4\alpha + 2\phi _{1,1} - \gamma _{1} + \beta _{1} \ge \varepsilon \\ -2\alpha + 2\phi _{1,1} - \gamma _{2} + \beta _{1} \ge \varepsilon \end{array} \right. $$

• $W_{1,2} = \{e_{1}(\mathbf {v})< 1\} = \{ v_{1} + v_{2} < 1\}$, which is satisfied if $v_{1} = 0$ y $v_{2} = 0$:

  $$ \left\{ \begin{array}{l} E_{1}(\mathbf {v}) \le \gamma _{1} + \beta _{1}\\ E_{2}(\mathbf {v}) \le \gamma _{2} + \beta _{1} \end{array} \right. $$

  and the instability is guaranteed if:

  $$ \vee \left\{ \begin{array}{l} \gamma _{1} + \beta _{1} \le -\varepsilon \\ \gamma _{2} + \beta _{1} \le -\varepsilon \end{array} \right. $$

Therefore, considering the initial conditions and choosing from the recently obtained inequalities, the following set of linear inequalities guarantees the stability of feasible solutions and instability of non-feasible ones:

$$\begin{aligned} {\left\{ \begin{array}{ll} \begin{array}{rrr} T_{1,1} = &{}- 4\alpha - \phi _{1,1} + 2\gamma _{1} \ge &{} 0\\ T_{2,2} = &{} {-}2\alpha - \phi _{1,1} + 2\gamma _{2} \ge &{} 0\\ &{} \phi _{1,1} \ge &{} 0\\ W_{1,1}: &{} 4\alpha + 2\phi _{1,1} - \gamma _{1} + \beta _{1} \ge &{} \varepsilon \\ W_{1,2}: &{} \gamma _{2} + \beta _{1} \le &{}-\varepsilon \end{array} \end{array}\right. } \end{aligned}$$