1 Introduction

The rapid advancement of machine learning has sparked significant progress in the field of optimization algorithms. Currently, there exists a vast array of diverse optimization algorithms, each employing various approaches and methodologies. Broadly speaking, these methods can be categorized into two groups: gradient-based methods and second-order methods. The foundation for these groups can be traced back to the inception of the gradient descent method and the Newton-Raphson method, respectively. This paper aims to present a comprehensive review of 38 optimization algorithms, highlighting their key characteristics and interrelationships. Selecting an appropriate optimization algorithm is not a straightforward task, often proving to be quite challenging. Moreover, identifying the ideal algorithm for a specific problem is not always feasible. To address this issue, we will delve into the main theoretical concepts underlying these algorithms, systematically organize them, and evaluate various performance aspects. These aspects include Rosenbrock convergence, sensitivity to Rosenbrock coefficient b, robustness to hyper-parameters, and learning rate variance. By analyzing and interpreting the obtained results, the expressive diagrams will be plotted. Through this comprehensive examination of optimization algorithms, we endeavor to shed light on their intricacies and provide valuable insights for practitioners and researchers.