Advanced algorithmic approach to optimization using MATLAB programming

Introduction

Different approaches can be used to solve large-scale unconstrained optimization issues. Backgroundique, which takes the following form, is one efficient way to solve unconstraint optimization problems:

s.t. 𝐹𝐹: 𝑅𝑅 𝑛𝑛 → 𝑅𝑅 this continuously differentiable function. From the initial point𝑥𝑥 0 ∈ 𝑅𝑅 𝑛𝑛 , the iterative sequence {𝑥𝑥 𝑘𝑘 } is generated to solution this problem by the recurrence

and 𝛼𝛼 𝑘𝑘 > 0 is obtained through a line search and 𝑑𝑑 𝑘𝑘 is a search direction given by: In addition the space 𝑅𝑅 𝑖𝑖 is vector space with Euclidean norm ‖. ‖ defined as follows [1] :

In this work, we aim to introduce a new two-term conjugate gradient approach to solve largescale equations due to the limited memory and simplicity. The authors introduced numerous techniques to solve various optimization and reliability problems (see) [2][3][4][5][6] . The suggested method satisfied both the steepest descent property and the Wolfe condition formula for global convergence.

The New Suggested Algorithm (MOD)

In this hand, we describe the framework of the suggested approach and satisfies a nice descent property. We used the new technique with (CGM) for solving a large scale problems of unconstraint optimization [7] constructed appropriate hyperplane ℋ 𝑘𝑘 where strictly separates 𝑥𝑥 𝑘𝑘 from the solution set of problem (1) i.e.

Dependent on Solodov and Svaiters advised total the other iteration point 𝑥𝑥 𝑘𝑘+1 is constructed by projecting 𝑥𝑥 𝑘𝑘 onto ℋ 𝑘𝑘 that is 𝑥𝑥 𝑘𝑘+1 is determined by:

the new search direction of our method can be computed by (5) Where

According to (4), ( 5), and ( 6), one appealing feature of our algorithm is that Conversely, according to (2), ( 5), and ( 6), the value of d k as determined by our method nearly satisfies the descent condition, i.e. we have thus 𝑑𝑑 𝑘𝑘 determined by the method almost holds the descent condition i.e.

Therefore, the direction 𝑑𝑑 𝑘𝑘 determined by our technique usually holds the sufficient descent condition such that

The steps of a new algorithm of the proposed method are as follows:

The algorithm for conjugate gradient projection (MO4) 1. The parameters 𝛾𝛾,𝜏𝜏>0 and 𝜌𝜌,𝜎𝜎∈ 0,1 are provided, along with the starting point 𝑥𝑥 0 ∈ 𝑅𝑅 𝑛𝑛.

Remark (A)

According to the above discussion, in our method, the search Direction 𝑑𝑑 𝑘𝑘 holds ( 7) is a sufficient descent direction of 𝑓𝑓 at iteration 𝑥𝑥 𝑘𝑘.Also, by (8) thus

This means the denominators of 𝛽𝛽 𝑘𝑘 𝑀𝑀𝑀𝑀𝑀𝑀 and 𝜉𝜉 𝑘𝑘 are greater than zero usually when ‖𝐹𝐹 𝑘𝑘 ‖ = 0 does not hold. This implies that these parameters are well defined.

The New Method's Global Convergence

To examine the new method's global analysis, we need to make a few assumptions.

Assumptions (B)

i) The function 𝑓𝑓(𝑥𝑥) is continuous on the level set R n, bounde below, and differentiable in a neighborhood 𝑁𝑁 of the level set.

By using the Assumptions (10) imply that ∃U > 0 such that

We assume, at most, that Assumptions (B) are satisfied in the final section of this work. Furthermore, we assume that 𝐹𝐹 𝑘𝑘 ≠0, Ψ𝑘𝑘≥0.In order to analyze the globally convergent, we work to determine the selection of 𝛼𝛼 𝑘𝑘 for all that 𝑑𝑑 𝑘𝑘 created by the new technique holds (8).

In the following lemma, we demonstrate a lower bound for 𝛼𝛼 𝑘𝑘 under the initial Wolfe line search.

Lemma: Suppose the sequence of search direction {𝑑𝑑 𝑘𝑘 }and{𝐹𝐹 𝑘𝑘 } be determined by algorithm4.1 Lemma: Assume that algorithm (MO4) determines the search direction sequence { 𝑑𝑑 𝑘𝑘 } and { 𝐹𝐹 𝑘𝑘 }. This suggests that Proof: By the given condition (8) we have

𝐿𝐿

, the step 2 comes by Cauchy-Schwarz inequality and step 3 comes by Lipschitz continuous. Since 𝑑𝑑 𝑘𝑘 hold (8) and 𝜎𝜎 < 1, this implies (12) satisfy. The proof is complete. To obtain the globally convergent conjugate gradient projection method, we used the Zoutendijk condition, which is first given [8] along with the Wolfe condition. The line search of the proposed algorithms now satisfies the Zoutendijk condition, as the following lemma shows.

Lemma: The step length 𝛼𝛼 𝑘𝑘 is determined by the original Wolfe line search if assumption (B) is satisfied and the suggested method finds the search direction sequence { 𝑑𝑑 𝑘𝑘 }, { 𝑓𝑓 𝑘𝑘 }, then we obtain

∞ 𝑘𝑘=0 (13) Proof: From (8) for any 𝑘𝑘 that

which the second and third inequality comes by ( 8) and (13). Therefore, by Assumptions (B)(i) that we get (13). The proof is complete.

As a result, we know that the iteration of (CGM) may fail, which means that only if 𝑑𝑑 𝑘𝑘 →∞ sufficiently quickly will 𝐹𝐹 𝑘𝑘 ≥𝜇𝜇, ∑𝑘𝑘≥0.An alternative interpretation is that the sequence { 𝐹𝐹 𝑘𝑘 } can only be bounded away from zero if 𝑘𝑘=0 ∞ 1 𝑑𝑑 𝑘𝑘 <+∞. The global convergence of the proposed approach is now examined in the subsequent lemma.

Theorem: Suppose the sequence {𝐹𝐹 𝑘𝑘 }and{𝑑𝑑 𝑘𝑘 }be generated by the suggested method, we have

Proof: Assume that ( 14) does not satisfy, let a constant 𝑀𝑀 > 0 s.t.

From ( 6), ( 7) and ( 8) we have

Cauchy-Schwarz inequality, ( 8), ( 9), and trigonometric inequality, respectively, are the sources of the second, third, and fourth inequalities. The last inequality comes by ( 11), (15) and Assumption (B)(i).

In synopsis, the sequence { 𝑑𝑑 𝑘𝑘 } determined by our algorithm has a common upper bound such that.

Additionally, 𝜍𝜍=𝜃𝜃. By applying (15) and ( 16), it is demonstrated that

This implies a contradiction (16), thus conclude that (17) holds. The proof is complete.

Numerical Experiments

In this part, we give some insight into the conduct of our method. Some problems have been used to test and analyze the efficiency of a new method compared to some two-terms conjugate gradient method (NTT) [9] , a three terms conjugate gradient method (NS) [10] and a conjugate gradient-like method (CGL) [11] . For everyone the test problems seven or eight initial point starting are used. Similar in [12] , we check the test problem when the variables number n=5000, 10000,…. and some given starting points 𝑥𝑥 0 = (10,10, … ,10) 𝑇𝑇 , 𝑥𝑥 1 = (-10, -10, … , -10) 𝑇𝑇 , All runs are proposed using MATLAB R2016a and run PC with 16GH, CPU2.30-Windows11 operation system. Numerical results are compared dependent on the number of iterations, the number of evaluation functions and CPU time.

The terminating executions we used some criterion whenever ‖𝐹𝐹 𝑘𝑘 ‖ ≤ 10 -6 and the total number of iteration exceeds 500000. The parameter of the new algorithm were set as follows:𝜏𝜏 = 10 -4 , 𝜎𝜎 = 0.5 and 𝛾𝛾 = 0.1. The parameter of the other methods comes from [8,13,14] . The step-size 𝛼𝛼 𝑘𝑘 is computed by

Conclusions

The (MOD) method under Wolfe condition is a novel approach to solving unconstrained optimization problems using the conjugate gradient (CG) algorithm. This method works well for large data sets because of its limited storage capacity. The global convergence has been proven for the proposed algorithm. Our approach has a competitive advantage over the other three conjugate gradient methods, and numerical results showed that it is a promising and practical tool for solving optimization problems.

References

1. Yu G, Guan L, Chen W. Spectral conjugate gradient methods with sufficient descent property for large-scale unconstrained optimization. Optimization Methods and Software. 2008;23:275-293.
2. Abbo KK. New CG method for large-scale unconstrained optimization based on Nazareth theorem. Iraqi Journal of Statistical Sciences. 2008;13:53-65.
3. Pang D, Du S. The smoothing Fletcher-Reeves conjugate gradient method for solving finite minimax problems. ScienceAsia. 2016;42:40-45. doi:10.2306/scienceasia1513-1874.2016.42.040.
4. Wasi HA, Shiker MAK. A new conjugate gradient method for solving large-scale systems of monotone equations. International Journal of Advanced Science and Technology. 2020;29(4):2303-2314.
5. Shiker MAK, Sahib Z. A modified trust-region method for solving unconstrained optimization. Journal of Engineering and Applied Sciences. 2018;13(22):96679671. doi:10.3923/jeasci.2018.9667.9671.
6. Kafaki SB, Ghanbari R. A descent extension of the Polak-Ribière-Polyak conjugate gradient method. Computers and Mathematics with Applications. 2014;68:2005-2011. doi:10.1016/j.camwa.2014.10.013.
7. Hussein HA, Shiker MAK. A modification to Vogel's approximation method to solve transportation problems. In: IOP Conference Series; Proceedings of the 5th International Scientific Conference for Iraqi AlKhwarizmi Society; Iraq; 2020. (Accepted for publication).
8. Hassan BA, Wais OMT, Mahmood AA. A class of descent conjugate gradient methods for solving optimization problems. Applied Mathematical Sciences. 2019;13:559-567. doi:10.12988/ams.2019.9656.
9. Hassan ZAHH, Shiker MAK. Using generalized Bayes' theorem to evaluate the reliability of aircraft systems. Journal of Engineering and Applied Sciences. 2018;(Special Issue 13):10797-10801. doi:10.36478/jeasci.2018.10797.10801.
10. Mahdi MM, Shiker MAK. A new projection technique for developing a Liu-Storey method to solve nonlinear systems of monotone equations. In: IOP Conference Series; Proceedings of the 5th International Scientific Conference for Iraqi Al-Khwarizmi Society; Iraq; 2020. (Accepted for publication).
11. Mahdi MM, Shiker MAK. Three-term derivative-free projection technique for solving nonlinear monotone equations. In: IOP Conference Series; Proceedings of the 5th International Scientific Conference for Iraqi AlKhwarizmi Society; Iraq; 2020. (Accepted for publication).
12. Shiker MAK, Amini K. A new projection-based algorithm for solving a large-scale nonlinear system of monotone equations. Croatian Operational Research Review. 2018;9(1):63-73. doi:10.17535/crorr.2018.0006.
13. Li Q, Huili D. A class of derivative-free methods for large-scale nonlinear monotone equations. IMA Journal of Numerical Analysis. 2011;31:1625-1635. doi:10.1093/imanum/drq047.