Research article Special Issues

The WuC-Adam algorithm based on joint improvement of Warmup and cosine annealing algorithms


  • The Adam algorithm is a common choice for optimizing neural network models. However, its application often brings challenges, such as susceptibility to local optima, overfitting and convergence problems caused by unstable learning rate behavior. In this article, we introduce an enhanced Adam optimization algorithm that integrates Warmup and cosine annealing techniques to alleviate these challenges. By integrating preheating technology into traditional Adam algorithms, we systematically improved the learning rate during the initial training phase, effectively avoiding instability issues. In addition, we adopt a dynamic cosine annealing strategy to adaptively adjust the learning rate, improve local optimization problems and enhance the model's generalization ability. To validate the effectiveness of our proposed method, extensive experiments were conducted on various standard datasets and compared with traditional Adam and other optimization methods. Multiple comparative experiments were conducted using multiple optimization algorithms and the improved algorithm proposed in this paper on multiple datasets. On the MNIST, CIFAR10 and CIFAR100 datasets, the improved algorithm proposed in this paper achieved accuracies of 98.87%, 87.67% and 58.88%, respectively, with significant improvements compared to other algorithms. The experimental results clearly indicate that our joint enhancement of the Adam algorithm has resulted in significant improvements in model convergence speed and generalization performance. These promising results emphasize the potential of our enhanced Adam algorithm in a wide range of deep learning tasks.

    Citation: Can Zhang, Yichuan Shao, Haijing Sun, Lei Xing, Qian Zhao, Le Zhang. The WuC-Adam algorithm based on joint improvement of Warmup and cosine annealing algorithms[J]. Mathematical Biosciences and Engineering, 2024, 21(1): 1270-1285. doi: 10.3934/mbe.2024054

    Related Papers:

    [1] Weidi Zhang, Lei Liang . The Great Tang All-Day Mall attractive cause analysis and guidance strategy. Urban Resilience and Sustainability, 2024, 2(1): 6-26. doi: 10.3934/urs.2024002
    [2] Layda B. Spor Leal, Jungseok Ho, Sara Davila . Geospatial analysis of impact on evacuation routes and urban areas in South Texas due to flood events. Urban Resilience and Sustainability, 2023, 1(1): 20-36. doi: 10.3934/urs.2023002
    [3] Fasil Kebede Tesfaye, Ayitenew Mogninet Getaneh, Mequanint Birhan . The management of sustainable energy sources in the mechanical sector. Urban Resilience and Sustainability, 2023, 1(2): 138-145. doi: 10.3934/urs.2023010
    [4] Heitor Castro Brasiel, Danielli Araújo Lima . Clustered-map probabilistic cellular automata for fire propagation in the Brazilian Cerrado with heterogeneous vegetation and wind interference. Urban Resilience and Sustainability, 2024, 2(1): 45-75. doi: 10.3934/urs.2024004
    [5] Erik Velasco . Circular economy in Singapore: waste management, food and agriculture, energy, and transportation. Urban Resilience and Sustainability, 2024, 2(2): 110-150. doi: 10.3934/urs.2024007
    [6] Francis Deng, Armin Mehdipour, Ali Soltani . Construction quality control of concrete structures in architectural engineering—A case in Shanghai, China. Urban Resilience and Sustainability, 2024, 2(3): 256-271. doi: 10.3934/urs.2024013
    [7] Fairuz Amanina, Zul Ilham . Placemaking strategies in greening Universiti Malaya Main Library. Urban Resilience and Sustainability, 2024, 2(1): 76-92. doi: 10.3934/urs.2024005
    [8] Jichao Wang, Xiaoning Sui, Jie Zhang, Wenjie Shi, Wayne L. Thompson . Relocating disaster-prone villages and improving villager well-being: Evidence from Beijing, China. Urban Resilience and Sustainability, 2024, 2(3): 236-255. doi: 10.3934/urs.2024012
    [9] Sunny Han Han, Yujing Li, Peiheng Yu . What makes a successful industrial heritage park?—China's experience based on the ecosystem cultural services perspective. Urban Resilience and Sustainability, 2024, 2(2): 93-109. doi: 10.3934/urs.2024006
    [10] João C. G. Lanzinha . Rehabilitation of existing building parks and its relationship with urban agglomerates–An urgent and demanding task for our common future. Urban Resilience and Sustainability, 2023, 1(2): 86-90. doi: 10.3934/urs.2023006
  • The Adam algorithm is a common choice for optimizing neural network models. However, its application often brings challenges, such as susceptibility to local optima, overfitting and convergence problems caused by unstable learning rate behavior. In this article, we introduce an enhanced Adam optimization algorithm that integrates Warmup and cosine annealing techniques to alleviate these challenges. By integrating preheating technology into traditional Adam algorithms, we systematically improved the learning rate during the initial training phase, effectively avoiding instability issues. In addition, we adopt a dynamic cosine annealing strategy to adaptively adjust the learning rate, improve local optimization problems and enhance the model's generalization ability. To validate the effectiveness of our proposed method, extensive experiments were conducted on various standard datasets and compared with traditional Adam and other optimization methods. Multiple comparative experiments were conducted using multiple optimization algorithms and the improved algorithm proposed in this paper on multiple datasets. On the MNIST, CIFAR10 and CIFAR100 datasets, the improved algorithm proposed in this paper achieved accuracies of 98.87%, 87.67% and 58.88%, respectively, with significant improvements compared to other algorithms. The experimental results clearly indicate that our joint enhancement of the Adam algorithm has resulted in significant improvements in model convergence speed and generalization performance. These promising results emphasize the potential of our enhanced Adam algorithm in a wide range of deep learning tasks.



    Many studies have recommended that more environmental and sustainable cities primarily focus on green spaces, infrastructure, and environmental quality. The rapid growth of urbanization requires that the development of construction projects maintain the urban population's standard of living [1]. The construction industry in India has become more competitive and stringent; due to this, the project managers are forced to be more effective before commencing each aspect of the construction activities [2,3]. The scheduling and planning activities of a construction project will optimally balance the important factors of time, quality, scope, cost, and work estimates in a sequential network of activities [4,5]. Each scheduling method has its own benefits and shortcomings. In any case, the timetables are best introduced in the bar diagram structure to maximize comprehensibility [3]. These bar diagrams are enhanced by a suitable arrangement strategy for observing the advancement of the activities.

    The work scheduling in construction projects serves the following purposes:

    a) It improves the plan.

    The bar chart-type work plan gives the improved form of the work plan, which can, without much effort, be able to properly illustrate the activities of arranging, coordinating, executing, and control of the venture [5].

    b) It improves the timetable.

    The plan for getting work done shows the arranged timetable of exercises, and information astute while putting the work plan on a schedule premise; it thinks about the decreased effectiveness of assets to antagonistic climatic conditions and different variables. It confirms the cutoff times required for the undertaking and accomplishment of the achievements [6].

    c) It streamlines the assets utilized.

    A plan for getting work done assumes the most efficient utilization of the materials, labor, and hardware. It accommodates sudden changes, which may occur. Asset streamlining is accomplished through the deliberate usage of the buoys of non-basic exercises [1].

    d) It estimates the information assets and predicts the yield.

    A plan for getting work done optimizes the estimation of assets and provides an example of asset utilization [7].

    Undertaking network investigation is a conventional term covering all of the organization strategies utilized for the arrangement, planning, and control of ventures. The three most regularly utilized methods are as follows:

    1. Basic technique (CPM)

    2. Task assessment and audit procedure

    3. Priority network investigation (PNA), out of which CPM is carried out by Primavera [7].

    Their normal highlights are the utilization of the organization model for the portrayal of the timetable for the task, the application of a basic method to decide the project timeline, the distinction of basic activities, and the utilization of organizational analysis procedures to maintain the goals according to the timetable. Yet, every one of these methods has an unmistakable model with a shifting field of use [4].

    Planning is the first thing to initiate for any project. Planning aims at improved performance and to balance the golden triangles of time, quality, cost, and scope of the project for successful completion. Planning leads to providing clear instructions, resource utilization, risk analysis, and milestone achievements in advance before project initiation [3]. Planning specializes in the formulation of a time-based plan of action for coordinating various activities and resources to achieve the assigned goals [8]. Before scheduling, the preliminary planning involves the initial formulation of the project, where construction planning is divided into the following major steps which are explained in Table 1:

    Table 1.  Methodology for preliminary planning.
    Stages Planning Process Techniques/Methods
    Planning Time Separating project work,
    creating time network plans,
    and booking work.
    Work breakdown, Network examination, Gantt graph, Line of equilibrium strategy, Time-restricted planning, and Resource-restricted booking
    Planning Resources Determining asset necessities,
    Arranging labor necessities,
    Arranging material prerequisites,
    Arranging gear acquirement,
    Planning authoritative designs, Budgeting Costs, and
    Dispensing assignments and assets.
    Determining labor planning,
    Material Booking, Equipment choice and booking, Cost arranging, and planning, and Resource assignment
    Planning Implementation Forming checking philosophy Asset efficiency control, Time control, Contribution control, and Budgetary control

     | Show Table
    DownLoad: CSV

    a) Planning Time

    b) Planning Resources

    c) Planning Implementation

    a) To schedule the entire project sequences within the time frame using Primavera.

    b) To reduce the cost by monitoring and allocating resources accordingly.

    c) To measure the practical durations required to carry out construction activities.

    d) To track the project progress and analyze the reasons for delays and cost overruns.

    In order to understand the urban built environment, the methodology adopted in this research study is the construction scheduling technique application modeling which is used to improve and optimize the time delays, resource over-allocation, and budget overruns for the successful completion of the project [9]. This study aims to develop scheduling and planning on a commercial building for effective optimization over the traditional approach which was usually adapted to in-site methods in construction projects.

    The research question was analyzed in depth by adjusting quantitative project data using Primavera P6 project management software to compile detailed scheduling and planning.

    In order to showcase the major differences between modern software management and traditional methods, an ongoing project that uses Microsoft Excel for scheduling and planning a commercial building of G+3 in Nellore, India was selected for our study. The data collection required for this project like manpower, resources, machinery, and BOQ has been collected from the real-time data of the project. Due to client privacy reasons, the company profile details are protected and all other project-related specifications are underpinned. To demonstrate the major differences between modern software management and traditional methods, we chose an ongoing project using Microsoft Excel to schedule and plan a G+3 commercial building in Nellore, India. The data collection of manpower, resources, machinery and bill of quantities required by the project has been collected from the real-time data of the project. Company file details are protected for client privacy reasons and all other project related specifications are supported. Project data collected from these sources is then executed in Primavera P6 to differentiate and highlight the advantages of using Primavera versus traditional methods.

    Total Built-up Area: 30,000 Sq. ft

    a) Type of the Project: G+3, Commercial Building

    b) Floor Wise BUA: 6,000 Sq. ft

    c) Total Project Cost: 4.18 Crores; ($5, 58,356 USD)

    d) Contract Period: 5 Months

    e) Nature of Contract: Item Rate Tendered Contract

    f) Project Start Date: 31st Jan 2020

    g) Project Finish Date: 09th Jun 2020

    In data analysis and interpretation, the project data will be entered periodically and will figure out the work schedule, delay expectations, cost, and resource optimization, so as to achieve better project output. The final output of the study will give a clear understanding of the usage of project management software, thereby speeding up the construction process and making project execution more cost effective and efficient.

    The main data collected from the project are programmed in Primavera P6 to allow proper scheduling, planning and optimization of the construction projects. The data interpretation and data analysis have been carried out to address the problems faced by the traditional method. The key finding of this research study was based on the five most important functions in construction activities:

    ⅰ) Scheduling;

    ⅱ) Planning;

    ⅲ) Resourcing Allocation & Leveling;

    ⅳ) Monitoring;

    ⅴ) Controlling.

    It has been found that most of the traditional projects in India don't follow this hierarchy and simply repeat the ⅱ, ⅲ and ⅳ functions multiple times leading to poor planning and scheduling process [10]. Thus, by using project management software, Primavera will help design work breakdown structures (WBS), develop schedules, plan accordingly, allocate resources to activities, track progress, and manage and control work fronts.

    A venture work-breakdown system empowers parting of the undertaking work into progressive work breakdown levels of sub-projects, assignments, work bundles, and exercises. Every action in WBS addresses a recognizable lower-level undertaking structure that devours time, quality, and assets [8]. The project calendar covers the project's life cycle in a tabular or bar chart from the initiation stage to the project handover. This commercial project calendar details and its paths are invariably represented in a bar format of Figure 1.

    Figure 1.  Project Scheduling in Gantt chart using Primavera.

    In case of any delay in the commencement of a single activity due to many practical reasons, other activities are also affected. The consequences of these delays can be predicted using the Primavera. For instance, the activity of painting and fittings works in finishing is delayed due to the local Covid-19 lockdown. Since it was an activity that appeared on the critical path, the entire duration of the project was delayed by 21 days, this will be tedious to execute the rest of the tasks without knowing the seriousness of the problem.

    Figures 2 shows the network before the delay, which also shows the start and finish date of the activity enlightened in blue color. Here, the change in the activity network was happened due to the delay of a single activity (Finishing). The software itself envisages the start of the successor activities depending upon the defined relationships. The yellow circle in Figure 3 shows the change in the activity network due to the delay of a single activity (Finishing). The software itself envisages the start of the successor activities depending upon the defined relationship during the program.

    Figure 2.  Expected commencement of the activity (Fittings Finishing).
    Figure 3.  Change in network activity due to delay in finishing.

    Resource allocation helps to reschedule the project tasks so that limited resources are used efficiently and unavoidable project delays are minimized. The resources are allotted to each activity according to the necessity with their units of measure, the price per unit, and the total quantity of resources to be utilized. The maximum and the minimum quantity of resources available per unit of time should be mentioned so that the software allocates resources according to the schedule and duration of the activity. In Figure 4, we have defined the resources for the activities to instance the resources of the activity roof slab are enlisted below.

    Figure 4.  Resource utilization details of a randomly selected activity (Finishing).

    Resources for construction project activities are limited in the real-life world. So, resource optimization is very important to avoid waste and shortage of resources on a construction project. The resource usage profile is always inspected for the proper updates on resource availability according to its schedule. A sample resource usage chart is provided in Figure 5, where the ray mix is allocated for each day as per the schedule against the maximum units available per unit of time. The black line represents the maximum limit. Hence, the optimized resource should lie within the limit.

    Figure 5.  Activities list along with the date & duration of one particular resource.

    Resource leveling attempts to reduce the sharp variations in the resource demand histogram while maintaining the original project duration. In this project, few resources were over-allocated due to unavoidable circumstances like a delay of supply, weather conditions, etc., which extended the need for the reallocation of the resources without delaying the duration of the project. 'Level resource' is a tool in Primavera that is used to reallocate the resources along with the scheduling of activities giving a 25% error range [9]. Figures 6, 7, illustrate the scenario for ray mix concrete when there was a shortage in the supply due to the wrong approximation in the project quotation.

    Figure 6.  Allocations of resources due to wrong quotation.
    Figure 7.  Allocations after resource-leveling.

    Monitoring and controlling are performed during the project execution to identify problems promptly and to take corrective actions to avoid risks in the project performance. The monitoring and control process helps in providing feedback between the various project phases to implement preventive actions to keep the project on track, on time, and within the budget limit time[11].

    In this project, the project is tracked for the schedule-based tasks and the corresponding resource availability. The tracking can be done even away from the site using a special web options feature in the Primavera software. The resource assignment for each day of the scheduled activity is fixed, and the task progress is monitored and controlled using the Primavera on a daily basis which is presented in Figure 8.

    Figure 8.  Resource assignments for each day as per schedule.

    This study mainly focuses on the overall scheduling and planning of the commercial building using Primavera software and analyzing the speed of construction activities compared to the traditional ways. This modern method helps us to adopt minimal requirements in order to achieve an efficient urban built environment. Primavera P6 is an advanced project management tool that helps in the completion of the project within the planned duration and the budget over the traditional methods of planning and management.

    This enables project managers, engineers, quantity surveyors, and other associated professionals to have instant access to all the project information they require instantly. Construction companies that are handling multi projects can use this software effectively and efficiently for successful completion. Even though the software is expensive initially, it can be used in a wide range of companies involving repetitive projects. Team members can easily check and monitor the progress of a project when the client or the person in charge is away from the workplace.

    Primavera is used widely in the rescheduling of the project so that the resources are effectively used without any delay according to the schedule. From the above observations made on the site, it is been noticed that the entire work managed to be completed (21–07–20) on time with a delay of 21 d due to the local Covid-19 lockdown. However, the cost of the project is not affected economically and is completed within the planned cost of Rs 4.15 crores ($5, 58,356). In spite of the lockdown, the project is achieved within the time and budget frame due to the availability of resources & manpower along with the Primavera rescheduling technique; in which work hours are increased without affecting the next day's scheduled activities.

    This was the result of using Primavera software which envisages the start of the successor activities depending upon the defined relationships. Besides, the Primavera software aids in reducing 2–3% of the overall cost of the traditional project by reducing the resources based on the slack time. Finally, the project which uses Primavera project management software right from the initiation stages shows more success rate and transparency compared to the other traditional ways of planning and management. The decision-making process of urban built environment quality in construction delivers sustainable growth for future cities.

    The authors are thankful to the VFSTR for the support and lab facility for the smooth conduction of research work.

    There is no conflict of interest between the authors.



    [1] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature, 521 (2015), 436–444. https://doi.org/10.1038/nature14539 doi: 10.1038/nature14539
    [2] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, B. Recht, The marginal value of adaptive gradient methods in machine learning, in Advances in Neural Information Processing Systems, 30 (2017).
    [3] S. J. Reddi, S. Kale, S. Kumar, On the convergence of Adam and beyond, preprint, arXiv: 1904.09237.
    [4] I. Loshchilov, F. Hutter, Fixing weight decay regularization in Adam, 2018. Available from: https://openreview.net/forum?id = rk6qdGgCZ.
    [5] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, et al., Large batch optimization for deep learning: Training BERT in 76 minutes, preprint, arXiv: 1904.00962.
    [6] S. J. Reddi, S. Kale, S. Kumar, On the convergence of Adam and beyond, preprint arXiv: 1904.09237.
    [7] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, et al., On the variance of the adaptive learning rate and beyond, preprint, arXiv: 1908.03265.
    [8] J. Zhuang, T. Tang, Y. Ding, S. C. Tatikonda, N. Dvornek, X. Papademetris, et al., AdaBelief Optimizer: Adapting stepsizes by the belief in observed gradients, in Advances in Neural Information Processing System, 33 (2020), 18795–18806.
    [9] W. Ilboudo, T. Kobayashi, K. Sugimoto, Robust stochastic gradient descent with student-t distribution based first-order momentum, IEEE Trans. Neural Networks Learn. Syst., 33 (2020), 1324–1337. https://doi.org/10.1109/TNNLS.2020.3041755 doi: 10.1109/TNNLS.2020.3041755
    [10] T. Dozat, Incorporating Nesterov Momentum into Adam, 2016. Available from: https://openreview.net/forum?id = OM0jvwB8jIp57ZJjtNEZ.
    [11] L. Luo, Y. Xiong, Y. Liu, X. Sun, Adaptive gradient methods with dynamic bound of learning rate, preprint, arXiv: 1902.09843.
    [12] G. Mordido, P. Malviya, A. Baratin, S. Chandar, Lookbehind optimizer: k steps back, 1 step forward, preprint, arXiv: 2307.16704.
    [13] M. Reyad, A. M. Sarhan, M. Arafa, A modified Adam algorithm for deep neural network optimization, Neural Comput. Appl., 2023 (2023), 1–18. https://doi.org/10.1007/s00521-023-08568-z doi: 10.1007/s00521-023-08568-z
    [14] X. Chen, C. Liang, D. Huang, E. Real, K. Wang, Y. Liu, et al., Symbolic discovery of optimization algorithms, preprint, arXiv: 2302.06675.
    [15] R. Tian, A. P. Parikh, Amos: An Adam-style optimizer with adaptive weight decay towards model-oriented scale, preprint, arXiv: 2210.11693.
    [16] M. Liu, D. Yao, Z. Liu, J. Guo, J. Chen, An improved Adam optimization algorithm combining adaptive coefficients and composite gradients based on randomized block coordinate descent, Comput. Intell. Neurosci., 2023 (2023), 1–13. https://doi.org/10.1155/2023/4765891 doi: 10.1155/2023/4765891
    [17] X. Xie, P. Zhou, H. Li, Z. Lin, S. Yan, Adan: Adaptive Nesterov momentum algorithm for faster optimizing deep models, preprint, arXiv: 2208.06677.
    [18] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, preprint, arXiv: 1412.6980.
    [19] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2015), 770–778.
  • Reader Comments
  • © 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Metrics

Article views(2731) PDF downloads(83) Cited by(23)

Figures and Tables

Figures(7)  /  Tables(3)

/

DownLoad:  Full-Size Img  PowerPoint
Return
Return

Catalog