Advertisement

SN Computer Science

, 1:21 | Cite as

Newton-2EGSOR Method for Unconstrained Optimization Problems with a Block Diagonal Hessian

  • Khadizah GhazaliEmail author
  • Jumat Sulaiman
  • Yosza Dasril
  • Darmesah Gabda
Original Research

Abstract

We find that finding the unconstrained optimizer for large-scale problems with the Newton method can be very expensive. Therefore, in this paper, we developed a new efficient method for solving large-scale unconstrained optimization problems with block diagonal Hessian matrices to reduce the cost of computing Newton’s direction. This method is a combination of the Newton method and the 2-point explicit group successive-over relaxation (2EGSOR) block iterative method. To calculate the performance of the developed method, we used a combination of the Newton method with Gauss–Seidel (Newton-GS) point iteration and the Newton method with successive-over relaxation (Newton-SOR) point iteration as reference methods. The numerical experiment has proven that the developed algorithms generate results that are more efficient compared to the reference methods with less execution time and fewer iterations. Through 90 test cases, we observe that the speedup ratio for our proposed method is up to 659.87 times faster compared to the Newton-SOR method and up to 1963.57 times more rapidly than the Newton-GS method.

Keywords

Large-scale unconstrained optimization problems Newton method Explicit group iterative method 

Introduction

In this paper, we consider a large-scale unconstrained optimization problem defined as
$$ \mathop {\hbox{min} }\limits_{{\underline{{\mathbf{x}}} \in {\mathbb{R}}^{n} }} f\left( {\underline{{\mathbf{x}}} } \right), $$
(1)
where \( f:{\mathbb{R}}^{n} \to {\mathbb{R}} \) is the objective function and at least twice continuously differentiable. This problem is one of the most challenging problems in applied mathematics and physics, which extensively and increasingly used in engineering, medical, computational economics, finance, marketing, telecommunications, and manufacturing. There are numerous existing methods for solving problem (1) such as pattern search [1], steepest descent method [2], Newton method [3], Barzilai–Borwein gradient method [4], Quasi-Newton method [5] or conjugate gradient method [6]. Among all these methods, the Newton method is probably one of the best-known methods that have an excellent performance when the initial point is chosen appropriately [3]. Not limited to that, the Newton method is a local method; thus, it has a local quadratic rate of convergence [7]. Therefore, in this paper, we choose Newton’s method for solving unconstrained optimization problems.

Although theoretically, Newton method has a fast convergence rate [8], it is practically also related with the difficulty of finding its second derivative when involving the large-scale problems and it has possible failure to converge to the solution of problem (1) from weak initial points. Therefore, in practice, many researchers tend to modify or combine it with another method for solving large-scale unconstrained optimization to make it more computationally efficient and to achieve an accurate solution [9, 10, 11, 12, 13, 14]. Bouaricha et al. in [9] presented a Newton method with trust region for finding large-scale unconstrained optimization problems and used two types of an iterative method which are the incomplete Cholesky decomposition and conjugate gradient method to obtain the trust region step instead of using the classical Newton direction. Shi [10] obtained a new globalization strategy for solving unconstrained optimization problems based on combining Newton’s direction with the steepest descent direction to achieved both global convergence and high local convergence in which the search direction is determined first so that the direction is a descent direction. Kahya [11] introduced and proven the convergence of some modification of Newton method with an exponential iteration formula for solving unconstrained optimization and finally showed that the introduced method is as efficient as the classical Newton method.

Next, Grapsa [12] proposed a modified Newton’s direction method using a proper gradient’s vector modification to have a descent property without a line search technique for solving problems of unconstrained optimization and proved its rate of convergence based on a Dennis modified Newton–Kantorovich theorem. Taheri et al. [13] developed a new algorithm for solving unconstrained optimization problems by combining the anti-gradient direction with the Newton direction to improve the convergence rate of Newton’s method. Abiodun and Adelabu [14] recently presented two iterative modifications of Newton’s method using an updated formula based on the recurrence of matrix factorizations to replaces an inverse matrix and maintains its positive definiteness property.

From what we have noticed, all of these improvements are caused by the problem of finding Newton’s direction thru solving the linear system. If it involves a small-scale problem, using the classical Newton method is very effective, but when it involves a large-scale problem, this method is very unhelpful unless we solve it using an iterative approach. One of the effective iterative methods is as suggested by either Young [15] or Evans [16]. Young [15] introduced several iterative solutions for solving a large linear system, and one of them is the SOR point iterative method, while Evans [16] introduced the block iterative method through the Explicit Group (EG) iterative method whereby this method grouped the linear system to several points for generating a new system of linear equations. This EG iterative method is known as one of the efficient block iterative methods which later have been further developed by Yousif and Evans [17, 18], Abdullah [19] and Othman and Abdullah [20].

Thus, in this paper, motivated by how to improve the computational efficiency and to save the storage requirements by cut down the work in the classical Newton method. We proposed a more reliable method for finding large-scale unconstrained optimizer by combining the Newton method with the 2-point explicit group successive-over relaxation (2EGSOR) iterative methods, namely as Newton-2EGSOR method. This combination uses the 2EGSOR iterative method as an inner iteration for finding Newton’s direction and used the Newton method as the outer iteration to estimate the solution of problem (1). To analyze the performance of the Newton-2EGSOR method, we consider a combination of the Newton method with successive-over relaxation iteration and the Newton method with Gauss–Seidel iteration as reference methods and they are called as the Newton-SOR method and the Newton-GS method, respectively. Noted that, solving unconstrained optimization problems using Newton-2EGSOR method as proposed in this paper is different from the existing combination of Newton’s method with other methods as stated in the previous paragraphs.

A brief description of the formulation for the Newton scheme with a block diagonal Hessian matrix is given in Sect. 2, and then the Newton-2EGSOR iteration is presented in Sect. 3. In Sect. 4, we report the result of our numerical experiments for the Newton-2EGSOR method, along with the result of the reference methods. Finally, we give some conclusions in Sect. 5.

Derivation of the Newton Scheme with a Block Diagonal Hessian Matrix

The Newton iterative scheme for solving large-scale unconstrained optimization was formulated for minimizing the quadratic approximation to the objective function, \( f\left( {\underline{{\mathbf{x}}} } \right) \) in the problem (1) at the current point \( \underline{{\mathbf{x}}}^{\left( k \right)} \) using the second order of Taylor expansion as:
$$ f\left( {\underline{{\mathbf{x}}} } \right) \approx f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) + \left[ {\nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)} \right]^{T} \left( {\underline{{\mathbf{x}}} - \underline{{\mathbf{x}}}^{\left( k \right)} } \right) + \frac{1}{2}\left( {\underline{{\mathbf{x}}} - \underline{{\mathbf{x}}}^{\left( k \right)} } \right)^{T} \nabla^{2} f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)\left( {\underline{{\mathbf{x}}} - \underline{{\mathbf{x}}}^{\left( k \right)} } \right), $$
(2)
where \( \nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) \) is the gradient of the first partial derivatives of \( f\left( {\underline{{\mathbf{x}}} } \right) \) and \( \nabla^{2} f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) = {\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) \) is the Hessian matrix of second partial derivatives of \( f\left( {\underline{{\mathbf{x}}} } \right) \). This quadratics approximation (2) will achieve its minimum value by differentiating it with respect to \( \underline{{\mathbf{x}}} \) and equating the resulting expression to zero as;
$$ \nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) + {\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)\left( {\underline{{\mathbf{x}}} - \underline{{\mathbf{x}}}^{\left( k \right)} } \right) = 0. $$
(3)
Thus, we can simplify Eq. (3) to get:
$$ \underline{{\mathbf{x}}} = \underline{{\mathbf{x}}}^{\left( k \right)} - \left[ {{\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)} \right]^{ - 1} \nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right). $$
(4)
This Eq. (4) produced the Newton iteration at a current point near the solution, and defined as:
$$ \underline{{\mathbf{x}}}^{{\left( {k + 1} \right)}} = \underline{{\mathbf{x}}}^{\left( k \right)} - \left[ {{\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)} \right]^{ - 1} \nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right), $$
(5)
where \( \left[ {{\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)} \right]^{ - 1} \) is the inverse of the Hessian matrix \( {\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) \). Thus, the Newton direction is
$$ \underline{d}^{\left( k \right)} = \underline{{\mathbf{x}}}^{{\left( {k + 1} \right)}} - \underline{{\mathbf{x}}}^{\left( k \right)} = - \left[ {{\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)} \right]^{ - 1} \nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right), $$
(6)
which can be determined by solving the Newton equation;
$$ {\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)\underline{d}^{\left( k \right)} = - \nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right). $$
(7)
We have supposed that \( {\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) \) is positive definite, so the Newton direction is a descent direction and it satisfies;
$$ \left[ {\nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)} \right]^{T} \underline{d}^{\left( k \right)} = - \left[ {\nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)} \right]^{T} \left[ {{\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right)} \right]^{ - 1} \nabla f\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) < 0. $$
(8)
In this study, we only consider solving the problem (1) for a large-scale unconstrained optimization where the dimension of the Hessian matrix, \( {\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) \) is greater or equal to 1000. As a particularly interesting case, in this paper also we consider the Hessian of a block diagonal matrix of order \( n \) with the general form is given by [21];
$$ {\mathbf{H}}\left( {\underline{{\mathbf{x}}}^{\left( k \right)} } \right) = \left[ {\begin{array}{*{20}c} {\frac{{\partial^{2} f}}{{\partial x_{1}^{2} }}} & {\frac{{\partial^{2} f}}{{\partial x_{1} \partial x_{2} }}} & 0 & \cdots & \cdots & \cdots & \cdots & 0 \\ {\frac{{\partial^{2} f}}{{\partial x_{2} \partial x_{1} }}} & {\frac{{\partial^{2} f}}{{\partial x_{2}^{2} }}} & 0 & \cdots & \cdots & \cdots & \cdots & 0 \\ 0 & 0 & {\frac{{\partial^{2} f}}{{\partial x_{3}^{2} }}} & {\frac{{\partial^{2} f}}{{\partial x_{3} \partial x_{4} }}} & 0 & \cdots & \cdots & 0 \\ \vdots & \vdots & {\frac{{\partial^{2} f}}{{\partial x_{4} \partial x_{3} }}} & {\frac{{\partial^{2} f}}{{\partial x_{4}^{2} }}} & 0 & \cdots & \cdots & 0 \\ \vdots & \vdots & 0 & 0 & \ddots & \ddots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \ddots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & 0 & 0 & {\frac{{\partial^{2} f}}{{\partial x_{n - 1}^{2} }}} & {\frac{{\partial^{2} f}}{{\partial x_{n - 1} \partial x_{n} }}} \\ 0 & 0 & 0 & 0 & 0 & 0 & {\frac{{\partial^{2} f}}{{\partial x_{n} \partial x_{n - 1} }}} & {\frac{{\partial^{2} f}}{{\partial x_{n}^{2} }}} \\ \end{array} } \right], $$
(9)

Formulation of the Proposed Iterative Method

Solving Eq. (7) with a coefficient matrix (9) using a direct method such as Gauss elimination method [22] or simultaneous method [23] needs more memory space and computing time since the coefficient matrix (9) is sparse and large scale. Furthermore, the implementations of the Newton method that use a direct method to obtain the Newton direction are not suitable for many large problems because of their cost in terms of computing the first and second partial derivative. Therefore, we choose a variation of the Newton method that uses an iterative method to obtain the Newton direction. Notice that Eq. (7) is a linear system which can be solved iteratively. Therefore, we propose an iterative method as in Sulaiman et al. [24] and Ghazali et al. [25] for solving this linear system. Let the linear system (7) be rewritten in general form as
$$ {\mathbf{A}}\underline{d} = \underline{f} $$
(10)
where,
$$ {\mathbf{A}} = \left[ {\begin{array}{*{20}c} {b_{1} } & {c_{1} } & 0 & \cdots & \cdots & \cdots & \cdots & 0 \\ {a_{2} } & {b_{2} } & 0 & \cdots & \cdots & \cdots & \cdots & 0 \\ 0 & 0 & {b_{3} } & {c_{3} } & 0 & \cdots & \cdots & 0 \\ \vdots & \vdots & {a_{4} } & {b_{4} } & 0 & \cdots & \cdots & 0 \\ \vdots & \vdots & 0 & 0 & \ddots & \ddots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \ddots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & 0 & 0 & {b_{n - 1} } & {c_{n - 1} } \\ 0 & 0 & 0 & 0 & 0 & 0 & {a_{n} } & {b_{n} } \\ \end{array} } \right],\underline{d} = \left[ {\begin{array}{*{20}c} {d_{1} } \\ {d_{2} } \\ {d_{3} } \\ {d_{4} } \\ \vdots \\ \vdots \\ {d_{n - 1} } \\ {d_{n} } \\ \end{array} } \right],\underline{f} = \left[ {\begin{array}{*{20}c} {f_{1} } \\ {f_{2} } \\ {f_{3} } \\ {f_{4} } \\ \vdots \\ \vdots \\ {f_{n - 1} } \\ {f_{n} } \\ \end{array} } \right] $$
with, \( b_{1} ,b_{2} \ldots ,b_{n} ,a_{2} , \ldots ,a_{n} ,c_{1} ,c_{2} \ldots ,c_{n - 1} \in {\mathbb{R}} \). The following subsections discussed the formulation of the SOR point iteration and the 2-point EG block iteration.

Formulation of the SOR Point Iteration

To derive the formulation of the SOR point iteration, we start by considering the implementation of a parameter \( \omega \) as a relaxation factor into Eq. (10) and rewrite it as [15];
$$ \omega {\mathbf{A}}\underline{d} = \omega \underline{f} . . $$
(11)
Let us decompose the real coefficient matrix \( {\mathbf{A}} \) of the linear system (11) as;
$$ {\mathbf{A}} = {\mathbf{D}} - {\mathbf{L}} - {\mathbf{U}}, $$
(12)
where \( {\mathbf{D}} \) is the nonzero diagonal matrix of \( {\mathbf{A}} \), \( {\mathbf{L}} \) and \( {\mathbf{U}} \) are strictly lower and upper triangular matrices of \( {\mathbf{A}} \), respectively. By applying the decomposition in Eq. (12) into a linear system (11), the iterative formulation of the SOR method can be expressed in vector form as [26, 27];
$$ \underline{d}^{{\left( {k + 1} \right)}} = \left( {{\mathbf{D}} - \omega {\mathbf{L}}} \right)^{ - 1} \left( {\omega {\mathbf{U}} + \left( {1 - \omega } \right){\mathbf{D}}} \right)\underline{d}^{\left( k \right)} + \omega \left( {{\mathbf{D}} - \omega {\mathbf{L}}} \right)^{ - 1} \underline{f} . $$
(13)
Referring to the linear system (10), Eq. (13) can be rewritten in iterative form as follows;
$$ \begin{aligned} b_{i} d_{i} + c_{i} d_{i + 1} = f_{i} ,\;i = 1,3, \ldots ,n - 1 \hfill \\ a_{i} d_{i - 1} + b_{i} d_{i} = f_{i} ,\;i = 2,4, \ldots ,n. \hfill \\ \end{aligned} $$
(14)
Therefore, based on Eq. (14), the implementation of the SOR point iterations, each component \( d_{i}^{{\left( {k + 1} \right)}} \), \( i = 1,2, \ldots ,n \) can be computed as;
$$ \begin{aligned} d_{i}^{{\left( {k + 1} \right)}} = \left( {1 - \omega } \right)d_{i}^{\left( k \right)} + \frac{\omega }{{b_{i} }}\left( {f_{i} - c_{i} d_{i + 1}^{\left( k \right)} } \right),\;i = 1,3, \ldots ,n - 1 \hfill \\ d_{i}^{{\left( {k + 1} \right)}} = \left( {1 - \omega } \right)d_{i}^{\left( k \right)} + \frac{\omega }{{b_{i} }}\left( {f_{i} - a_{i} d_{i - 1}^{{\left( {k + 1} \right)}} } \right),i = 2,4, \ldots ,n, \hfill \\ \end{aligned} $$
(15)
where \( \omega \) with the optimal value in the range of [1, 2) and selected based on the smallest number of inner iterations. Equation (13) is turning out to be the Gauss–Seidel method, with \( \omega = 1 \). The performance of the SOR iteration can be dramatically improved with a good choice of \( \omega \). For the optimal choice of \( \omega \), the SOR iteration may converge faster than the Gauss–Seidel iteration. Theoretically, the optimal value of \( \omega \) is given by [15];
$$ \omega_{\text{OPT}} = \frac{2}{{1 + \sqrt {1 - \rho^{2} } }}. $$

With \( \rho \) denotes as the spectral radius of the iteration matrix. Using the formulation of the SOR point iterative method, the next subsection discussed the formulation of block SOR iteration, known as the Explicit Group SOR block iterative method.

Formulation of the 2-Point EGSOR Block Iteration

Based on our selection of n for this paper, this iteration involves a complete group of two points accordingly for generating 2 × 2 systems of a linear equation. To derive the formulation of the proposed iterative method, let us expressed a group of two points from the linear system (10) as follows [28, 29];
$$ \left[ {\begin{array}{*{20}c} {b_{i} } & {c_{i} } \\ {a_{i + 1} } & {b_{i + 1} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {d_{i} } \\ {d_{i + 1} } \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {f_{i} } \\ {f_{i + 1} } \\ \end{array} } \right]. $$
(16)
Hence, solving and simplifying Eq. (16) gives that the explicit form of the 2-point EGSOR iteration can be stated as;
$$ \begin{aligned} d_{i}^{{\left( {k + 1} \right)}} = \left( {1 - \omega } \right)d_{i}^{\left( k \right)} + \frac{\omega }{\alpha }\left( {b_{i + 1} f_{i} - c_{i} f_{i + 1} } \right) \hfill \\ d_{i + 1}^{{\left( {k + 1} \right)}} = \left( {1 - \omega } \right)d_{i + 1}^{\left( k \right)} + \frac{\omega }{\alpha }\left( { - a_{i + 1} f_{i} - b_{i} f_{i + 1} } \right), \hfill \\ \end{aligned} $$
(17)
where
$$ \alpha = \left( {b_{i} } \right)\left( {b_{i + 1} } \right) - \left( {c_{i} } \right)\left( {a_{i + 1} } \right), $$
and the optimal range of \( \omega \) is still in [1,2). Therefore, using Eqs. (10) and (17), we propose the reliable algorithm of the Newton-2EGSOR scheme with a block diagonal Hessian matrix for solving problem (1), as stated in Algorithm 1.

Numerical Experiments and Results

In this section, we described the numerical experiments we have carried out using Algorithm 1 and highlight the comparison results observed. Besides that, we also specify several details left unspecified in Algorithm 1. We start by representing the test functions used in our numerical experiments.

The Test Functions

We use six unconstrained optimization test functions that are obtained from the collection collected by Andrei [30] and Moghrabi [31]. The selection of all these test functions is based on its Hessian type, which is a block diagonal Hessian matrix. In Table 1, we specified the description for each of the six test functions. The test number shown in Table 1 for each test will be used to denote to that test function in reporting comparison results. For the initial point selection, we used three different initial points (designated as \( {\mathbf{x}}^{\left( 0 \right)} \)). Since each test function collected by Andrei [30] and Moghrabi [31] have provided a standard initial point, therefore, we set it as one of our initial points and used it as an option (a), while for the selection of a nonstandard initial point, we use random initial points chosen from a range surrounding the standard initial point. We indicated it as an option (b) for nonstandard initial point 1 and as an option (c) for nonstandard initial point 2 in Table 1. Thus for each test function, we have three test cases. For example, we indicate the test case defined by test number 3 with the initial point option (c) as 3(c). Therefore, we have 18 test cases from six test functions in Table 1. In addition to that, all these test cases were run five times using five different order of Hessian matrix as n = {1000, 5000, 10000, 20000, 30000}. Thus, this gives a total of 90 test cases
Table 1

The test functions used in the numerical experiment

Test number

Test name, algebraic expression, local optimal value and optimal point

Initial point, \( {\mathbf{x}}^{\left( 0 \right)} \)

(a) Standard

(b) Nonstandard 1

(c) Nonstandard 2

1

Extended White and Holts

\( f\left( {\underline{\text{x}} } \right) = \sum\nolimits_{i = 1}^{n/2} {100\left( {x_{2i} + x_{2i - 1}^{3} } \right)^{2} + \left( {1 - x_{2i - 1} } \right)^{2} } \)

\( f^{*} = 0 \) and \( {\mathbf{x}}^{*} = \left( {1.0, - 1.0, \ldots ,1.0, - 1.0} \right) \)

\( \left( { - 1.2,1.0, \ldots , - 1.2,1.0} \right) \)

\( \left( {0.65,0.65, \ldots ,0.65,0.65} \right) \)

\( \left( {1.5, - 1.0, \ldots ,1.5, - 1.0} \right) \)

2

Extended Rosenbrock

\( f\left( {\underline{\text{x}} } \right) = \sum\nolimits_{i = 1}^{n/2} {100\left( {x_{2i} - x_{2i - 1}^{2} } \right)^{2} + \left( {1 - x_{2i - 1} } \right)^{2} } \)

\( f^{*} = 0 \) and \( {\mathbf{x}}^{*} = \left( {1.0,1.0, \ldots ,1.0,1.0} \right) \)

\( \left( { - 1.2,1.0, \ldots , - 1.2,1.0} \right) \)

\( \left( { - 1.2, - 1.2, \ldots , - 1.2, - 1.2} \right) \) \( \left( {1.5,1.5, \ldots ,1.5,1.5} \right) \)

3

Extended Tridiagonal 1

\( f\left( {\underline{\text{x}} } \right) = \sum\nolimits_{i = 1}^{n/2} {\left( {x_{2i - 1} + x_{2i} - 3} \right)^{2} + } \left( {x_{2i - 1} - x_{2i} + 1} \right)^{4} \)

\( f^{*} = 0 \) and \( {\mathbf{x}}^{*} = \left( {1.0,2.0, \ldots ,1.0,2.0} \right) \)

\( \left( {2.0,2.0, \ldots ,2.0,2.0} \right) \)

\( \left( {2.0, - 2.0, \ldots ,2.0, - 2.0} \right) \) \( \left( {1.0, - 1.0, \ldots ,1.0, - 1.0} \right) \)

4

Extended Himmelblau

\( f\left( {\underline{\text{x}} } \right) = \sum\nolimits_{i = 1}^{n/2} {\left( {x_{{_{2i - 1} }}^{2} + x_{2i} - 11} \right)^{2} + } \left( {x_{2i - 1} + x_{{_{2i} }}^{2} - 7} \right)^{2} \)

\( f^{*} = 0 \) and \( {\mathbf{x}}^{*} = \left( {3.0,2.0, \ldots ,3.0,2.0} \right) \)

\( \left( {1.0,1.0, \ldots ,1.0,1.0} \right) \)

\( \left( {3.0,3.0, \ldots ,3.0,3.0} \right) \) \( \left( {1.0,1.2, \ldots ,1.0,1.2} \right) \)

5

Extended DENSCHNF

\( f\left( {\underline{\text{x}} } \right) = \sum\nolimits_{i = 1}^{n/2} {\left[ {2\left( {x_{2i - 1} + x_{2i} } \right)^{2} + \left( {x_{2i - 1} - x_{2i} } \right)^{2} - 8} \right]^{2} + } \left[ {5x_{2i - 1}^{2} + \left( {x_{2i} - 3} \right)^{2} - 9} \right]^{2} \)

\( f^{*} = 0 \) and \( {\mathbf{x}}^{*} = \left( {1.0,1.0, \ldots ,1.0,1.0} \right) \)

\( \left( {2.0,0.0, \ldots ,2.0,0.0} \right) \)

\( \left( {2.0,2.0, \ldots ,2.0,2.0} \right) \) \( \left( {2.0, - 1.0, \ldots ,2.0, - 1.0} \right) \)

6

Generalized Shallow

\( f\left( {\underline{\text{x}} } \right) = \sum\nolimits_{i = 1}^{n/2} {\left( {2x_{2i - 1}^{2} - x_{2i} } \right)^{2} } + \left( {1 - x_{2i - 1} } \right)^{2} \)

\( f^{*} = 0 \) and \( {\mathbf{x}}^{*} = \left( {1.0,2.0, \ldots ,1.0,2.0} \right) \)

\( \left( { - 2.0, - 2.0, \ldots , - 2.0, - 2.0} \right) \)

\( \left( {2.5,2.5, \ldots ,2.5,2.5} \right) \) \( \left( { - 2.0,1.0, \ldots , - 2.0,1.0} \right) \)

Convergence Tolerance

Often previous researchers set the convergence tolerance as 10−4 [6], 10−5 [9] or 10−6 [32] for the case of unconstrained optimization problems. Since there is no particular method for choosing the convergence tolerance, we decided to select the convergence tolerances by looking at the effect of tolerance on simulation performance (accuracy and computational time) for Algorithm 1. Therefore, we look at the effect of adjusting the convergence tolerances (\( \varepsilon_{1} \) and \( \varepsilon_{2} \)) for various value by implementing Algorithm 1 on test case 2(a) with n = {1000}. We report the relationship between decreasing convergence tolerance (with \( \varepsilon_{1} = \varepsilon_{2} \)) and the number of outer iterations in Fig. 1. Clearly from Fig. 1, we can see that as we reduced the convergence tolerance, the number of outer iterations increased up to a certain point and then stop showing a significant increase. We also notice that the simulation time is increasing according to the number of iterations. Thus, to avoid the excessive increased of the simulation time, we chose \( 10^{ - 6} \) as our convergence tolerance to terminate Algorithm 1.
Fig. 1

The relationship between the number of outer iterations and convergence tolerance (\( \varepsilon_{1} = \varepsilon_{2} \))

However, to get better accuracy, we set up the Algorithm 1 to look at how much the error from the exact minimum value changes with a changing of convergence tolerance, \( \varepsilon_{2} \) and maintained the convergence tolerance \( \varepsilon_{1} \) as \( 10^{ - 6} \). Thus, we compute the error from the exact minimum value with decreasing convergence tolerance, \( \varepsilon_{2} \) using \( \varepsilon_{1} = 10^{ - 6} \) and stated the result in Fig. 2. Thru Fig. 2, we can see that the error decreased exponentially and did not provide much gain in performance at a convergence tolerance, \( \varepsilon_{2} \) of about \( 10^{ - 8} \). As a result, a tolerance of \( 10^{ - 8} \) was chosen to terminate the inner iteration with a tolerance \( 10^{ - 6} \) to terminate the outer iteration.
Fig. 2

Exponential decay of error for decreasing convergence tolerance \( \varepsilon_{2} \)

Intuitively, we fixed our convergence tolerances as \( \varepsilon_{2} = 10^{ - 8} \) (for inner iteration) and \( \varepsilon_{1} = 10^{ - 6} \) (for outer iteration) on all our test cases.

The Optimal Value of \( \omega \)

To determine the optimal value of \( \omega \), we implement several computer programs, and then we choose the optimal value when their number of inner iterations is the smallest. For Newton-SOR method to do this, we set up the Algorithm 1 to search the optimal parameter \( \omega \) by

Notation Used in the Table of Comparison Results

We use the following notation in reporting the comparison results.
\( {\text{NI}}_{i} \)

Number of inner iterations

\( {\text{NI}}_{O} \)

Number of outer iterations

t

Execution time in seconds

\( {\text{FV}}_{t} \)

Function value at the iterate where execution terminated

\( l_{2} {-}n_{t} \)

l2 norm of the gradient at the iterate where execution terminated

Ʃt

Total execution time in seconds

N GS

Newton-GS method

N SOR

Newton-SOR method

N 2EGSOR

Newton-2EGSOR method

Comparison Results

In this subsection, we shall report the numerical comparison of the three methods mentioned earlier. The numerical comparisons have been completed using Algorithm 1, and we indicate the comparison results observed. All the numerical calculations are compiled using C language with double precision arithmetic. The results of the numerical comparison of the number of inner iterations and the number of outer iterations, the execution time is in seconds, the function value at the iterate where the execution terminated and the l2 norm of the gradient at the iterate where the execution terminated are tabulated in Table 2. All values tabulated in Table 2 are rounded up to two decimal points. Hence, the value l2 norm of the gradient at the iterate where the execution terminated is smaller than the convergence tolerance, \( \varepsilon_{1} \). Noted that in Table 2, 11 test cases show its approximate value is the optimal value. These values are obtained using the method (Newton-2EGSOR) we proposed, 1(c) for n = {1000, 5000, 10000, 20000, 30000}, 2(b) for n = {30000} and 5(c) for n = {1000, 5000, 10000, 20000, 30000}. Therefore, the selection of a good initial point can influence the efficiency of the early search process. In Table 2 also, we stated the optimal value of \( \omega \) for Newton-SOR method and its value is in the range between 1 and 2. Using the comparison results in Table 2, we computed the decrement percentage of the number of iteration for the Newton-2EGSOR method and the Newton-SOR method compared to the Newton-GS method and stated it in Table 3. Finally, to better understand the comparison results, we have computed the speedup ratio for the Newton-2EGSOR method with both reference methods in Table 4.
Table 2

Computational result of the Newton-GS, Newton-SOR, and Newton-2EGSOR

Test cases

n

NIi

NIo

t

FVt

l2-nt

N GS

NSOR (ω)

N 2EGSOR

N GS

N SOR

N 2EGSOR

N GS

N SOR

N 2EGSOR

N GS

N SOR

N 2EGSOR

N GS

N SOR

N 2EGSOR

1(a)

1000

34,490

17,458 (1.362)

12

4281

1851

6

1.35

0.68

0.00

2.50E−13

9.98E−14

3.49E−23

9.99E−07

9.98E−07

3.74E−10

5000

35,215

17,797 (1.362)

12

5006

2190

6

7.39

3.62

0.00

2.50E−13

9.98E−14

1.75E−22

9.99E−07

9.98E−07

8.36E−10

10,000

35,527

17,943 (1.362)

12

5318

2336

6

15.18

7.67

0.01

2.50E−13

9.99E−14

3.49E−22

9.99E−07

9.98E−07

1.18E−09

20,000

35,839

18,089 (1.362)

12

5630

2482

6

31.60

15.58

0.03

2.50E−13

9.99E−14

6.98E−22

9.99E−07

9.98E−07

1.67E−09

30,000

36,021

18,174 (1.362)

12

5812

2567

6

48.28

25.24

0.04

2.50E−13

1.00E−13

1.05E−21

1.00E−06

9.99E−07

2.05E−09

1(b)

1000

10,541

1017 (1.940)

10

4417

60

5

0.90

0.06

0.00

2.50E−13

2.36E−16

6.16E−30

9.99E−07

9.72E−07

4.97E−15

5000

11,265

1023 (1.940)

10

5141

66

5

5.19

0.26

0.00

2.50E−13

2.97E−16

3.08E−29

1.00E−06

9.34E−07

1.11E−14

10,000

11,577

1025 (1.940)

10

5453

68

5

10.85

0.47

0.01

2.50E−13

4.36E−16

6.16E−29

1.00E−06

9.25E−07

1.57E−14

20,000

11,889

1027 (1.940)

10

5765

70

5

22.92

0.88

0.02

2.50E−13

7.23E−16

1.23E−28

1.00E−06

8.50E−07

2.22E−14

30,000

12,072

1028 (1.940)

10

5948

71

5

35.52

1.46

0.03

2.50E−13

1.03E−15

1.85E−28

9.99E−07

8.02E−07

2.72E−14

1(c)

1000

58,846

1917 (1.972)

10

4411

71

5

1.86

0.06

0.00

2.50E−13

9.67E−14

0.00E + 00

9.99E−07

7.77E−07

0.00E + 00

5000

59,571

1969 (1.972)

10

5136

123

5

9.89

0.36

0.01

2.50E−13

2.51E−14

0.00E + 00

9.99E−07

4.69E−07

0.00E + 00

10,000

59,883

1969 (1.972)

10

5448

123

5

20.31

0.64

0.01

2.50E−13

5.01E−14

0.00E + 00

9.99E−07

6.63E−07

0.00E + 00

20,000

60,195

1969 (1.972)

10

5760

123

5

42.18

1.34

0.02

2.50E−13

1.00E−13

0.00E + 00

9.99E−07

9.37E−07

0.00E + 00

30,000

60,377

1970 (1.972)

10

5942

124

5

63.21

1.94

0.03

2.50E−13

1.50E−13

0.00E + 00

1.00E−06

5.99E−07

0.00E + 00

2(a)

1000

12,320

997 (1.915)

12

1806

20

6

0.43

0.03

0.00

2.50E−13

4.16E−14

1.72E−17

9.99E−07

3.02E−07

1.85E−07

5000

12,642

997 (1.915)

12

2128

20

6

2.42

0.12

0.01

2.50E−13

2.08E−13

8.58E−17

1.00E−06

6.76E−07

4.14E−07

10,000

12,781

997 (1.915)

12

2267

20

6

5.08

0.24

0.01

2.50E−13

4.16E−13

1.72E−16

9.99E−07

9.56E−07

5.86E−07

20,000

12,920

1018 (1.913)

12

2406

24

6

10.26

0.50

0.02

2.49E−13

2.60E−13

3.43E−16

9.99E−07

7.44E−07

8.29E−07

30,000

13,001

1018 (1.913)

13

2487

24

7

16.16

0.78

0.03

2.50E−13

3.90E−13

0.00E + 00

9.99E−07

9.11E−07

0.00E + 00

2(b)

1000

14,018

902 (1.921)

10

1804

11

5

0.47

0.03

0.00

2.50E−13

8.69E−13

5.68E−22

9.99E−07

9.54E−07

4.87E−11

5000

14,340

952 (1.921)

10

2126

61

5

2.53

0.14

0.00

2.50E−13

6.40E−16

2.84E−21

1.00E−06

8.05E−07

1.09E−10

10,000

14,479

953 (1.921)

10

2265

62

5

5.30

0.28

0.01

2.50E−13

1.11E−15

5.68E−21

9.99E−07

8.55E−07

1.54E−10

20,000

14,618

954 (1.921)

10

2404

63

5

10.94

0.56

0.01

2.50E−13

2.04E−15

1.14E−20

9.99E−07

8.58E−07

2.18E−10

30,000

14,699

955 (1.921)

10

2485

64

5

16.54

0.91

0.02

2.50E−13

2.92E−15

1.70E−20

9.99E−07

6.76E−07

2.67E−10

2(c)

1000

15,173

985 (1.937)

10

1669

54

5

0.48

0.03

0.00

2.49E−13

1.74E−15

2.59E−25

9.99E−07

9.17E−07

1.39E−12

5000

15,495

987 (1.937)

10

1991

56

5

2.61

0.15

0.00

2.50E−13

8.06E−15

1.29E−24

9.99E−07

9.01E−07

3.11E−12

10,000

15,634

988 (1.937)

10

2130

57

5

5.26

0.29

0.01

2.49E−13

1.60E−14

2.59E−24

9.99E−07

5.91E−07

4.40E−12

20,000

15,773

988 (1.937)

10

2269

57

5

11.06

0.58

0.02

2.49E−13

3.20E−14

5.18E−24

9.98E−07

8.36E−07

6.22E−12

30,000

15,854

989 (1.937)

10

2350

58

5

17.20

0.86

0.02

2.49E−13

4.79E−14

7.77E−24

9.99E−07

2.02E−07

7.62E−12

3(a)

1000

36,264

4812 (1.936)

32

16

16

16

0.76

0.21

0.00

4.53E−09

2.71E−09

2.69E−09

7.33E−07

5.49E−07

4.46E−07

5000

49,992

7024 (1.936)

32

19

17

16

5.04

1.11

0.02

8.48E−09

2.80E−09

1.34E−08

1.00E−06

7.70E−07

9.98E−07

10,000

58,811

12,456 (1.967)

34

8838

19

17

23.20

3.60

0.04

1.07E−08

2.95E−10

5.31E−09

1.00E−06

1.00E−06

4.18E−07

20,000

69,922

19,410 (1.967)

34

19,949

2276

17

83.45

17.82

0.06

1.35E−08

9.17E−11

1.06E−08

1.00E−06

1.00E−06

5.91E−07

30,000

77,716

20,977 (1.970)

34

27,743

3735

17

155.78

35.59

0.09

1.54E−08

9.22E−11

1.59E−08

1.00E−06

1.00E−06

7.24E−07

3(b)

1000

37,654

5850 (1.923)

40

20

20

20

0.76

0.15

0.00

4.39E−09

2.62E−09

2.56E−09

7.18E−07

5.40E−07

4.30E−07

5000

51,001

8180 (1.937)

40

23

21

20

5.23

1.02

0.02

8.48E−09

2.67E−09

1.28E−08

1.00E−06

7.66E−07

9.62E−07

10,000

59,820

14,827 (1.971)

42

8842

23

21

24.52

3.62

0.04

1.07E−08

2.75E−10

5.05E−09

1.00E−06

1.00E−06

4.03E−07

20,000

70,931

21,229 (1.959)

42

19,953

2454

21

84.41

16.57

0.08

1.35E−08

1.23E−10

1.01E−08

1.00E−06

1.00E−06

5.70E−07

30,000

78,725

23,040 (1.964)

42

27,747

3977

21

164.71

32.34

0.11

1.54E−08

1.18E−10

1.51E−08

1.00E−06

1.00E−06

6.98E−07

3(c)

1000

40,121

6117 (1.940)

38

19

19

19

0.82

0.16

0.00

3.41E−09

1.71E−09

1.68E−09

6.20E−07

4.48E−07

3.13E−07

5000

50,344

6711 (1.956)

38

22

19

19

5.36

0.85

0.02

8.48E−09

8.56E−09

8.38E−09

1.00E−06

1.00E−06

7.01E−07

10,000

59,163

14,784 (1.964)

38

8841

22

19

24.03

3.61

0.04

1.07E−08

2.13E−10

1.68E−08

1.00E−06

1.00E−06

9.91E−07

20,000

70,274

20,648 (1.964)

40

19,952

2348

20

82.52

16.84

0.06

1.35E−08

1.03E−10

6.62E−09

1.00E−06

1.00E−06

4.15E−07

30,000

78,069

22,277 (1.964)

40

27,747

3977

20

160.88

31.63

0.11

1.54E−08

1.18E−10

9.94E−09

1.00E−06

1.00E−06

5.09E−07

4(a)

1000

62

52 (1.026)

15

8

8

8

0.01

0.00

0.00

3.92E−16

1.95E−16

8.31E−28

2.90E−07

2.10E−07

3.66E−13

5000

62

52 (1.026)

15

8

8

8

0.02

0.01

0.01

1.96E−15

9.73E−16

4.15E−27

6.49E−07

4.70E−07

8.19E−13

10,000

62

52 (1.026)

15

8

8

8

0.03

0.02

0.02

3.92E−15

1.95E−15

8.31E−27

9.17E−07

6.65E−07

1.16E−12

20,000

63

52 (1.026)

15

9

8

8

0.07

0.06

0.04

4.75E−17

3.89E−15

1.66E−26

1.01E−07

9.41E−07

1.64E−12

30,000

63

53 (1.026)

15

9

9

8

0.10

0.09

0.06

7.12E−17

3.62E−18

2.49E−26

1.24E−07

2.92E−08

2.01E−12

4(b)

1000

44

32 (1.045)

10

6

5

5

0.00

0.00

0.00

1.54E−15

7.59E−17

5.58E−17

4.38E−07

6.37E−08

5.87E−08

5000

44

32 (1.045)

10

6

5

5

0.01

0.01

0.01

7.70E−15

3.80E−16

2.79E−16

9.79E−07

1.42E−07

1.31E−07

10,000

45

32 (1.045)

10

7

5

5

0.03

0.02

0.01

3.89E−16

7.59E−16

5.58E−16

2.20E−07

2.01E−07

1.86E−07

20,000

45

32 (1.045)

10

7

5

5

0.05

0.04

0.02

7.79E−16

1.52E−15

1.12E−15

3.11E−07

2.85E−07

2.63E−07

30,000

45

32 (1.045)

10

7

5

5

0.07

0.05

0.04

1.17E−15

2.28E−15

1.67E−15

3.81E−07

3.49E−07

3.22E−07

4(c)

1000

116

111 (1.015)

22

11

12

11

0.01

0.00

0.01

1.18E−15

6.86E−17

3.45E−16

4.25E−07

1.23E−07

2.38E−07

5000

116

111 (1.015)

22

11

12

11

0.03

0.02

0.01

5.89E−15

3.43E−16

1.73E−15

9.51E−07

2.76E−07

5.31E−07

10,000

117

111 (1.015)

22

12

12

11

0.05

0.04

0.03

5.00E−16

6.86E−16

3.45E−15

3.27E−07

3.90E−07

7.52E−07

20,000

117

111 (1.015)

23

12

12

12

0.10

0.09

0.06

1.00E−15

1.37E−15

1.66E−26

4.63E−07

5.51E−07

1.64E−12

30,000

117

111 (1.015)

23

12

12

12

0.15

0.14

0.07

1.50E−15

2.06E−15

2.49E−26

5.67E−07

6.75E−07

2.01E−12

5(a)

1000

31

30 (1.015)

12

6

6

6

0.00

0.00

0.00

4.81E−15

7.48E−19

1.01E−18

9.92E−07

1.19E−08

1.41E−08

5000

32

30 (1.015)

12

7

6

6

0.02

0.01

0.03

4.68E−17

3.74E−18

1.01E−17

9.78E−08

2.66E−08

4.45E−08

10,000

32

30 (1.015)

12

7

6

6

0.04

0.03

0.02

9.36E−17

7.48E−18

1.01E−17

1.38E−07

3.76E−08

4.45E−08

20,000

32

30 (1.015)

12

7

6

6

0.08

0.07

0.06

1.87E−16

1.50E−17

2.02E−17

1.96E−07

5.32E−08

6.29E−08

30,000

32

30 (1.015)

12

7

6

6

0.12

0.10

0.08

2.81E−16

2.24E−17

3.03E−17

2.40E−07

6.51E−08

7.70E−08

5(b)

1000

48

39 (1.051)

12

6

6

6

0.00

0.00

0.00

9.06E−16

4.22E−15

9.47E−26

3.85E−07

8.61E−07

5.68E−07

5000

48

40 (1.051)

13

6

7

7

0.02

0.02

0.01

4.53E−15

4.49E−17

6.31E−26

8.61E−07

1.13E−07

2.54E−12

10,000

49

40 (1.051)

13

7

7

7

0.05

0.04

0.03

5.89E−16

8.99E−17

3.16E−26

3.47E−07

1.59E−07

3.60E−12

20,000

49

40 (1.051)

13

7

7

7

0.09

0.07

0.07

1.18E−15

1.80E−16

1.58E−26

4.91E−07

2.25E−07

5.09E−12

30,000

49

40 (1.051)

13

7

7

7

0.12

0.11

0.09

1.77E−15

2.70E−16

7.81E−16

6.01E−07

2.76E−07

6.23E−12

5(c)

1000

51

45 (1.083)

13

7

7

7

0.01

0.00

0.00

3.18E−15

8.29E−16

0.00E + 00

8.07E−07

3.51E−07

0.00E + 00

5000

52

45 (1.083)

13

8

7

7

0.02

0.02

0.01

3.07E−17

4.15E−15

0.00E + 00

7.92E−08

7.84E−07

0.00E + 00

10,000

52

46 (1.083)

13

8

8

7

0.04

0.04

0.03

6.13E−17

1.94E−16

0.00E + 00

1.12E−07

2.37E−07

0.00E + 00

20,000

52

46 (1.083)

13

8

8

7

0.09

0.09

0.06

1.23E−16

3.89E−16

0.00E + 00

1.58E−07

3.35E−07

0.00E + 00

30,000

52

46 (1.083)

13

8

8

7

0.13

0.13

0.10

1.84E−16

5.83E−16

0.00E + 00

1.94E−07

4.11E−07

0.00E + 00

6(a)

1000

1481

361 (1.768)

12

16

6

6

0.07

0.01

0.00

2.34E−13

1.63E−14

2.56E−23

9.68E−07

4.81E−07

4.17E−11

5000

1494

362 (1.768)

12

29

7

6

0.29

0.09

0.00

2.42E−13

7.47E−14

1.28E−22

9.84E−07

3.68E−07

9.33E−11

10,000

1500

362 (1.768)

12

35

7

6

0.47

0.23

0.01

2.34E−13

1.49E−13

2.56E−22

9.67E−07

5.21E−07

1.32E−10

20,000

1506

362 (1.768)

12

41

7

6

0.92

0.28

0.02

2.26E−13

2.99E−13

5.12E−22

9.51E−07

7.37E−07

1.87E−10

30,000

1509

362 (1.768)

12

44

7

6

1.35

0.39

0.03

2.36E−13

4.48E−13

7.68E−22

9.71E−07

9.02E + 00

2.29E−10

6(b)

1000

2039

400 (1.819)

10

15

6

5

0.07

0.05

0.00

2.34E−13

9.40E−15

1.16E−17

9.67E−07

5.39E−07

6.81E−09

5000

2052

401 (1.819)

10

28

7

5

0.31

0.16

0.00

2.42E−13

3.89E−14

5.80E−17

9.83E−07

1.45E−07

1.52E−08

10,000

2058

401 (1.819)

10

34

7

5

0.76

0.17

0.01

2.34E−13

7.78E−14

1.16E−16

9.66E−07

2.05E−07

2.16E−08

20,000

2064

401 (1.819)

10

40

7

5

1.22

0.32

0.01

2.26E−13

1.56E−13

2.32E−16

9.50E−07

2.91E−07

3.05E−08

30,000

2067

401 (1.819)

10

43

7

5

1.84

0.57

0.03

2.35E−13

2.33E−13

3.48E−16

9.70E−07

3.56E−07

3.73E−08

6(c)

1000

1421

367 (1.763)

12

16

6

6

0.07

0.03

0.00

2.24E−13

1.28E−13

8.25E−18

9.47E−07

7.74E−07

2.37E−08

5000

1434

373 (1.756)

12

29

6

6

0.26

0.10

0.01

2.32E−13

3.51E−13

4.13E−17

9.63E−07

5.58E−07

5.30E−08

10,000

1440

373 (1.756)

12

35

6

6

0.50

0.15

0.01

2.24E−13

7.03E−13

8.25E−17

9.47E−07

7.89E−07

7.49E−08

20,000

1445

374 (1.755)

12

40

6

6

0.87

0.30

0.02

2.44E−13

1.28E−12

1.65E−16

9.89E−07

8.55E−07

1.06E−07

30,000

1449

381 (1.755)

12

44

11

6

1.36

0.41

0.03

2.26E−13

2.60E−14

2.48E−16

9.50E−07

5.76E−07

1.30E−07

Table 3

Decrement percentage of the number of iterations Newton-SOR and Newton-2EGSOR compared to Newton-GS

Test cases

NIi (%)

Nio (%)

Test cases

NIi (%)

Nio (%)

N SOR

N 2EGSOR

N SOR

N 2EGSOR

N SOR

N 2EGSOR

N SOR

N 2EGSOR

1(a)

49.38–49.55

99.97

55.83–56.76

99.86–99.90

4(a)

15.87–17.46

75.81–76.19

0.00–11.11

0.00–11.11

1(b)

90.35–91.48

99.91–99.92

98.64–98.81

99.89–99.92

4(b)

27.27–28.89

77.27–77.78

16.67–28.57

16.67–28.57

1(c)

96.69–96.74

99.98

97.61–98.39

99.89–99.92

4(c)

4.31–5.13

80.34–81.20

0.00

0.00–8.33

2(a)

91.91–92.20

99.90–99.91

98.89–99.12

99.67–99.75

5(a)

3.23–6.25

61.29–62.50

0.00–14.29

0.00–14.29

2(b)

93.36–93.57

99.93

97.13–99.39

99.72–99.80

5(b)

16.67–13.46

72.92–75.00

0.00

0.00

2(c)

93.51–93.76

99.93–99.94

96.76–97.53

99.70–99.79

5(c)

11.54–13.46

74.51–75.00

0.00–12.50

0.00–12.50

3(a)

4812–86.73

99.91–99.96

0.00–99.79

0.00–99.94

6(a)

75.62–76.01

99.19–99.20

62.50–84.09

62.50–86.36

3(b)

70.07–84.46

99.89–99.95

0.00–99.74

0.00–99.92

6(b)

80.38–80.60

99.51–99.52

60.00–83.72

66.67–88.37

3(c)

70.62–86.67

99.91–99.95

0.00–99.75

0.00–99.93

6(c)

73.71–74.17

99.16–99.17

62.50–85.00

62.50–86.36

Table 4

Comparison of speedup ratio for Newton-2EGSOR method with Newton-GS method and Newton-SOR method

Test cases

Ʃt

Speedup ratio

NGS (I)

NSOR (II)

N2EGSOR (III)

\( \frac{I}{II} \)

\( \frac{I}{III} \)

\( \frac{II}{III} \)

1(a)

103.80

52.79

0.08

1.97

1297.50

659.87

1(b)

75.38

3.13

0.06

24.08

1256.33

52.17

1(c)

137.45

4.34

0.07

31.67

1963.57

62.00

2(a)

34.35

1.67

0.07

20.57

490.71

23.86

2(b)

35.78

1.92

0.04

18.64

894.50

48.00

2(c)

36.61

1.91

0.05

19.17

732.20

38.20

3(a)

268.23

58.33

0.21

4.60

1277.29

277.76

3(b)

279.63

53.70

0.25

5.21

1118.52

214.80

3(c)

273.61

53.09

0.23

5.15

1189.61

230.83

4(a)

0.23

0.18

0.13

1.28

1.77

1.38

4(b)

0.16

0.12

0.08

1.33

2.00

1.50

4(c)

0.34

0.29

0.18

1.17

1.89

1.61

5(a)

0.26

0.21

0.19

1.24

1.37

1.11

5(b)

0.28

0.24

0.20

1.17

1.40

1.20

5(c)

0.29

0.28

0.20

1.04

1.45

1.40

6(a)

3.10

1.00

0.06

3.10

51.67

16.67

6(b)

4.20

1.27

0.05

3.31

84.00

25.40

6(c)

3.06

0.99

0.07

3.09

43.71

14.14

Conclusion

As presented in this paper, our proposed combination of the Newton method with the 2-point Explicit Group (2EG) block iterative method is more efficient in the process for solving large-scale unconstrained optimization problems with a block diagonal Hessian matrix compared to the reference methods. This can be seen clearly through the comparison results in Table 2, which demonstrates the efficiency of our proposed method. From the column of the function value at the iterate where execution terminated in Table 2, it is found that the accuracy of our proposed method is more superior compared to the reference methods. We also can observe from the decrement percentage of the number of iteration in Table 3 that the Newton-2EGSOR method reduces the use of computational storage in finding the approximate value. It is apparent that the number of inner iteration for the Newton-2EGSOR method decreased from the Newton-GS methods with the least percentage decrease is 61.29%, and the most significant percentage decrease is 99.96%. Whereas for the decrement of the number of outer iterations is up to 99.94%. As expected, with the use of a block iterative method together with the relaxation factor, \( \omega \), the speedup ratio for our proposed method was much faster than the reference methods. In Table 4, we can see that the speedup ratio of our proposed method is up to 1963.57 times faster compared to the Newton-GS method and up to 659.87 times faster compared to the Newton-SOR method. Thus, it can be concluded that our proposed iterative method is able to show substantial improvement in the number of iterations and execution time compared to the Newton-GS and Newton-SOR point iterative methods. To extend this study, we will develop an algorithm base on the combination of the Newton method with 4-point Explicit Group (4EG) block iterative approach as done by Ghazali et al. [33].

Notes

Funding

This study was funded by Universiti Malaysia Sabah (Grant number: GUG0160-2/2017).

Compliance with Ethical Standards

Conflict of Interest

Authors (Khadizah Ghazali and Jumat Sulaiman) have received research grants from Universiti Malaysia Sabah, while authors (Yosza Dasril and Darmesah Gabda) declare that they have no conflict of interest.

References

  1. 1.
    Mariani VC, Leandro DS. A hybrid shuffled complex evolution approach with pattern search for unconstrained optimization. Math Comput Simul. 2011;81(9):1901–9.MathSciNetCrossRefGoogle Scholar
  2. 2.
    Johari R, Rivaie M, Mamat M. A new scaled steepest descent method for unconstrained optimization with global convergence properties, Journal of Engineering and Applied Sciences. 2018; http://medwelljournals.com/abstract/?doi=jeasci.2018.5442.5445. Accessed 8 Jan 2019.
  3. 3.
    Kumar S, Kanwar V, Tomar SK, Singh S. Geometrically constructed families of Newton’s method for unconstrained optimization and nonlinear equations. Int J Math Math Sci. 2011.  https://doi.org/10.1155/2011/972537.MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Nosratipour H, Solymani O, Borzabadi AH. An Adaptive nonmonotone global Brazilai-Borwein gradient method for unconstrained optimization. Optimization. 2017;66(4):641–55.MathSciNetCrossRefGoogle Scholar
  5. 5.
    Zhang H, Ni Q. A new regularized quasi-Newton method for unconstrained optimization. Optim Lett. 2018;12:1639–58.MathSciNetCrossRefGoogle Scholar
  6. 6.
    Moyi AU, Leong WJ. A sufficient descent three-term conjugate gradient method via symmetric rank-one update for largescale optimization. Optimization. 2016;65(1):121–43.MathSciNetCrossRefGoogle Scholar
  7. 7.
    Sun W, Yuan Y. Optimization theory and methods: nonlinear programming. Boston, MA: Springer; 2006zbMATHGoogle Scholar
  8. 8.
    Nocedal J, Wright SJ. Numerical optimization. 2nd ed. Berlin: Springer-Verlag; 2000.zbMATHGoogle Scholar
  9. 9.
    Bouaricha A, More JJ, Wu Z. Newton’s method for large-scale optimization, Preprint MCS-P635-0197. Argonne: Argonne National Laboratory; 1997.Google Scholar
  10. 10.
    Shi Y. Globally convergent algorithms for unconstrained optimization. Comput Optim Appl. 2000;16:295–308.MathSciNetCrossRefGoogle Scholar
  11. 11.
    Kahya E. A class of exponential quadratically convergent iterative formulae for unconstrained optimization. Appl Math Comput. 2007;186:1010–7.MathSciNetzbMATHGoogle Scholar
  12. 12.
    Grapsa TN. A modified Newton direction for unconstrained optimization. Optimization. 2014;63(7):983–1004.MathSciNetCrossRefGoogle Scholar
  13. 13.
    Taheri S, Mammadov M, Seifollahi S. Globally convergent algorithms for solving unconstrained optimization problems. Optimization. 2015;64(2):249–63.MathSciNetCrossRefGoogle Scholar
  14. 14.
    Abiodun TL, Adelabu OA. A comparison of newton’s method and two of its modifications for the solution of nonlinear optimization problems. Inte J Educ Sci. 2017;17(1–3):116–28.Google Scholar
  15. 15.
    Young DM. Iterative solution of large linear systems. London: Academic Press; 1971.zbMATHGoogle Scholar
  16. 16.
    Evans DJ. Group explicit iterative methods. Int J Comput Math. 1985.  https://doi.org/10.1080/00207168508803452.CrossRefzbMATHGoogle Scholar
  17. 17.
    Yousif WS, Evans DJ. Explicit de-coupled group iterative methods and their implementations. Parallel Algorithms Appl. 1995;7:53–71.CrossRefGoogle Scholar
  18. 18.
    Yousif WS, Evans DJ. Explicit group over-relaxation methods for solving elliptic partial differential equations. Math Comput Simul. 1986;28:453–66.MathSciNetCrossRefGoogle Scholar
  19. 19.
    Abdullah AR. The four point explicit decoupled group (EDG) method: a fast Poisson solver. Int J Comput Math. 1991;38:61–70.CrossRefGoogle Scholar
  20. 20.
    Othman M, Abdullah AR. An efficient four points modified explicit group poisson solver. Int J Comput Math. 2000;76:203–17.MathSciNetCrossRefGoogle Scholar
  21. 21.
    Savenkov SN, Muttiah RS, Oberemok E, Klimov A. Incomplete active polarimetry: measurement of the block-diagonal scattering matrix. J Quant Spectrosc Radiat Transf. 2011;112(11):1796–802.CrossRefGoogle Scholar
  22. 22.
    Killingbeck JP, Grosjean A. A Gauss elimination method for resonances. J Math Chem. 2010;47:1027–37.MathSciNetCrossRefGoogle Scholar
  23. 23.
    Liu Y, Lv J, Gao X. The application of simultaneous elimination and back substitution method (SEBSM) in finite element method. Eng Comput. 2016;33(8):2339–55.CrossRefGoogle Scholar
  24. 24.
    Sulaiman J, Hasan MK, Othman M, Karim SAA. Application of block iterative methods with newton scheme for fisher’s equation by using implicit finite difference. Jurnal KALAM. 2015;8(1):39–46. https://jurnalkalam.wordpress.com/volume-8-%E2%99%A6-number-1-%E2%99%A6-june-2015/. Accessed 15 Dec 2017.
  25. 25.
    Ghazali K, Sulaiman J, Dasril Y, Gabda D. Newton method with explicit group iteration for solving large scale unconstrained optimization problems. J Phys Conf Ser. 2018.  https://doi.org/10.1088/1742-6596/1132/1/012056.CrossRefGoogle Scholar
  26. 26.
    Saad Y. Iterative methods for sparse linear systems, 2nd edn. Philadelphia, PA: SIAM; 2003.zbMATHGoogle Scholar
  27. 27.
    Sulaiman J, Hasan MK, Othman M, Karim SAA. Fourth-order solutions of nonlinear two-point boundary value problems by Newton-HSSOR iteration. AIP Conf Proc. 2014;1602:69–75.CrossRefGoogle Scholar
  28. 28.
    Sulaiman J, Hasan MK, Othman M, Karim SAA. Newton-EGMSOR methods for solution of second order two-point. J Math Syst Sci. 2012;2:185–90. http://www.davidpublisher.org/index.php/Home/Article/index?id=7598.html. Accessed 6 Aug 2018.
  29. 29.
    Sulaiman J, Hasan MK, Othman M, Karim SAA. Numerical solutions of nonlinear second-order two-point boundary value problems using half-sweep SOR with Newton method. J Concr Appl Math. 2013;11(1):112–20.MathSciNetzbMATHGoogle Scholar
  30. 30.
    Andrei N. An unconstrained optimization test function collection. Adv Model Optim. 2008;10(1):147–61.MathSciNetzbMATHGoogle Scholar
  31. 31.
    Moghrabi IAR. Minimization of extended quadratic functions with inexact line searches. J Korean Soc Ind Appl Math. 2005;9(1):55–61.MathSciNetGoogle Scholar
  32. 32.
    Shuai H, Zhong W, Jing Z. An extended nonmonotone line search technique for large-scale unconstrained optimization. J Comput Appl Math. 2018;330:586–604.MathSciNetCrossRefGoogle Scholar
  33. 33.
    Ghazali K, Sulaiman J, Dasril Y, Gabda D. Application of Newton-4EGSOR Iteration for solving large scale unconstrained optimization problems with a tridiagonal hessian matrix. In: Alfred R, Lim Y, Ibrahim A, Anthony P, editors. Computational science and technology. Lecture Notes in Electrical Engineering. Singapore: Springer; 2019.  https://doi.org/10.1007/978-981-13-2622-6_39.CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd 2019

Authors and Affiliations

  1. 1.Faculty of Science and Natural Resources, Mathematics with Economics ProgrammeUniversiti Malaysia SabahSabahMalaysia
  2. 2.Faculty of Electronic and Computer EngineeringUniversiti Teknikal Malaysia MelakaMelakaMalaysia

Personalised recommendations