Stationary and non-stationary transition probabilities in decision making: Modeling COVID-19 dynamics

Romario Gildas Foko Tiomela; Samson Adekola Alagbe; Olawale Nasiru Lawal; Serges Love Teutu Talla; Isabella Kemajou-Brown; Romario Gildas Foko Tiomela; Samson Adekola Alagbe; Olawale Nasiru Lawal; Serges Love Teutu Talla; Isabella Kemajou-Brown

doi:10.3934/mbe.2025106

Mathematical Biosciences and Engineering

2025, Volume 22, Issue 11: 2870-2896. doi: 10.3934/mbe.2025106

Previous Article Next Article

Research article Special Issues

Stationary and non-stationary transition probabilities in decision making: Modeling COVID-19 dynamics

Department of Mathematics, EMERGE, Morgan State University, Baltimore, MD 21251, USA

Received: 14 July 2025 Revised: 22 August 2025 Accepted: 01 September 2025 Published: 15 September 2025

This study present a comparative modeling framework for COVID-19 dynamics using stationary and non-stationary transition probabilities within a Markov decision process (MDP). Stationary transitions assume constant rates, while non-stationary transitions capture time-dependent behaviors driven by policy interventions or behavioral changes. We develop a seven-compartmental epidemiological model, derive transition probabilities from binomial and multinomial processes, and implement time-dependent parameterizations to reflect real-world dynamics. Mathematical models for both stationary and non-stationary transition frameworks are developed and simulated over a 365-day period to emphasize dynamic variations in epidemic outcomes. Our findings highlight the significance of non-stationary modeling in accurately representing the dynamic characteristics of pandemic situations and provide recommendations for optimizing public health interventions under uncertainty. This comparative analysis offers useful information for epidemiological modeling and decision making in dynamic risk environments.
- Stationary,
- non-stationary,
- transition probability,
- COVID-19,
- MDP
Citation: Romario Gildas Foko Tiomela, Samson Adekola Alagbe, Olawale Nasiru Lawal, Serges Love Teutu Talla, Isabella Kemajou-Brown. Stationary and non-stationary transition probabilities in decision making: Modeling COVID-19 dynamics[J]. Mathematical Biosciences and Engineering, 2025, 22(11): 2870-2896. doi: 10.3934/mbe.2025106

Related Papers:

Abstract

This study present a comparative modeling framework for COVID-19 dynamics using stationary and non-stationary transition probabilities within a Markov decision process (MDP). Stationary transitions assume constant rates, while non-stationary transitions capture time-dependent behaviors driven by policy interventions or behavioral changes. We develop a seven-compartmental epidemiological model, derive transition probabilities from binomial and multinomial processes, and implement time-dependent parameterizations to reflect real-world dynamics. Mathematical models for both stationary and non-stationary transition frameworks are developed and simulated over a 365-day period to emphasize dynamic variations in epidemic outcomes. Our findings highlight the significance of non-stationary modeling in accurately representing the dynamic characteristics of pandemic situations and provide recommendations for optimizing public health interventions under uncertainty. This comparative analysis offers useful information for epidemiological modeling and decision making in dynamic risk environments.

References

[1]	M. Biggerstaff, B. J. Cowling, Z. M. Cucunubá, L. Dinh, N. M. Ferguson, H. Gao, et al., Early insights from statistical and mathematical modeling of key epidemiologic parameters of COVID-19, Emerg. Infect. Diseases, 26 (2020), e1–e14. https://doi.org/10.3201/eid2611.201074 doi: 10.3201/eid2611.201074
[2]	M. Chinazzi, J. T. Davis, M. Ajelli, C. Gioannini, M. Litvinova, S. Merler, et al., The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak, Science, 368 (2020), 395–400. https://doi.org/10.1126/science.aba9757 doi: 10.1126/science.aba9757
[3]	F. Crunfli, V. C. Carregari, F. P. Veras, P. H. Vendramini, A. G. F. Valença, A. S. L. M. Antunes, et al., SARS-CoV-2 infects brain astrocytes of COVID-19 patients and impairs neuronal viability, MedRxiv, (2021), 2020–10. https://doi.org/10.1101/2020.10.09.20207464
[4]	A. D. Barbour, S. Utev, Approximating the Reed-Frost epidemic process, Stoch. Proc. Appl., 113 (2004), 173–197. https://doi.org/10.1016/j.spa.2004.03.013 doi: 10.1016/j.spa.2004.03.013
[5]	C. Lefevre, P. Picard, A non-standard family of polynomials and the final size distribution of Reed-Frost epidemic processes, Adv. Appl. Probab., 22 (1990), 25–48. https://doi.org/10.2307/1427595 doi: 10.2307/1427595
[6]	J. Ng, E. J. Orav, A generalized chain binomial model with application to HIV infection, Math. Biosci., 101 (1990), 99–119. https://doi.org/10.1016/0025-5564(90)90104-7 doi: 10.1016/0025-5564(90)90104-7
[7]	A. M. Perez, M. P. Ward, A. Charmandarián, V. Ritacco, Simulation model of within-herd transmission of bovine tuberculosis in Argentine dairy herds, Prevent. Veter. Med., 54 (2002), 361–372. https://doi.org/10.1016/S0167-5877(02)00043-0 doi: 10.1016/S0167-5877(02)00043-0
[8]	J. Ranta, P. H. Mäkelä, A. Takala, E. Arjas, Predicting the course of meningococcal disease outbreaks in closed subpopulations, Epidemiol. Infect., 123 (1999), 359–371. https://doi.org/10.1017/S0950268899003039 doi: 10.1017/S0950268899003039
[9]	T. Tsutsui, N. Minami, M. Koiwai, T. Hamaoka, I. Yamane, K. Shimura, A stochastic-modeling evaluation of the foot-and-mouth-disease survey conducted after the outbreak in Miyazaki, Japan in 2000, Prevent. Veter. Med., 61 (2003), 45–58. https://doi.org/10.1016/S0167-5877(03)00160-0 doi: 10.1016/S0167-5877(03)00160-0
[10]	R. M. Anderson, R. M. May, Infectious diseases of humans: Dynamics and control, Oxford university press, 1991.
[11]	H. W. Hethcote, The mathematics of infectious diseases, SIAM Rev., 42 (2000), 599–653. https://doi.org/10.1137/S0036144500371907 doi: 10.1137/S0036144500371907
[12]	G. Giordano, F. Blanchini, R. Bruno, P. Colaneri, A. Di Filippo, A. Di Matteo, et al., Modelling the COVID-19 epidemic and implementation of population-wide interventions in Italy, Nat. Med., 26 (2020), 855–860. https://doi.org/10.1038/s41591-020-0883-7 doi: 10.1038/s41591-020-0883-7
[13]	Q. Li, H. Chen, Y. Li, M. Feng, J. Kurths, Network spreading among areas: A dynamical complex network modeling approach, Chaos Interdiscipl. J. Nonlinear Sci., 32 (2022), 103102. https://doi.org/10.1063/5.0102390 doi: 10.1063/5.0102390
[14]	S. Purkayastha, R. Bhattacharyya, R. Bhaduri, R. Kundu, X. Gu, M. Salvatore, et al., A comparison of five epidemiological models for transmission of SARS-CoV-2 in India, BMC Infect. Diseases, 21 (2021), 1–23. https://doi.org/10.1186/s12879-021-06077-9 doi: 10.1186/s12879-021-06077-9
[15]	H. C. Tuckwell, R. J. Williams, Some properties of a simple stochastic epidemic model of SIR type, Math. Biosci., 208 (2007), 76–97. https://doi.org/10.1016/j.mbs.2006.09.018 doi: 10.1016/j.mbs.2006.09.018
[16]	L. Palopoli, D. Fontanelli, M. Frego, M. Roveri, A Markovian model for the spread of the SARS-CoV-2 virus, Automatica, 151 (2023), 110921. https://doi.org/10.1016/j.automatica.2023.110921 doi: 10.1016/j.automatica.2023.110921
[17]	M. Ahmadi, N. Jansen, B. Wu, U. Topcu, Control theory meets POMDPs: A hybrid systems approach, IEEE Transact. Autom. Control, 66 (2020), 5191–5204. https://doi.org/10.1109/TAC.2020.3035755 doi: 10.1109/TAC.2020.3035755
[18]	M. C. Browne, E. M. Clarke, O. Grümberg, Characterizing finite Kripke structures in propositional temporal logic, Theor. Computer Sci., 59 (1988), 115–131. https://doi.org/10.1016/0304-3975(88)90098-9 doi: 10.1016/0304-3975(88)90098-9
[19]	Y. Li, Z. Zeng, M. Feng, J. Kurths, Protection degree and migration in the stochastic SIRS model: A queueing system perspective, IEEE Transact. Circuits Syst. I Regular Papers, 69 (2021), 771–783. https://doi.org/10.1109/TCSI.2021.3119978 doi: 10.1109/TCSI.2021.3119978
[20]	J. C. Miller, A. C. Slim, E. M. Volz, Edge-based compartmental modelling for infectious disease spread, J. Royal Soc. Interface, 9 (2012), 890–906. https://doi.org/10.1098/rsif.2011.0403 doi: 10.1098/rsif.2011.0403
[21]	F. Llopis-Cardona, C. Armero, G. Sanfélix-Gimeno, Estimating disease incidence rates and transition probabilities in elderly patients using multi-state models: a case study in fragility fracture using a Bayesian approach, BMC Med. Res. Methodol., 23 (2023), 103102-1–40. https://doi.org/10.1186/s12874-023-01859-y doi: 10.1186/s12874-023-01859-y
[22]	A. J. Black, J. V. Ross, Computation of epidemic final size distributions, J. Theoret. Biol., 367 (2015), 159–165. https://doi.org/10.1016/j.jtbi.2014.11.029 doi: 10.1016/j.jtbi.2014.11.029
[23]	C. Barril, P. A. Bliman, S. Cuadrado, Final Size for Epidemic Models with Asymptomatic Transmission, Bullet. Math. Biol., 85 (2023), 52. https://doi.org/10.1007/s11538-023-01159-y doi: 10.1007/s11538-023-01159-y
[24]	P. Giles, The mathematical theory of infectious diseases and its applications, J. Operat. Res. Soc., 28 (1977), 479–480. https://doi.org/10.1057/jors.1977.92 doi: 10.1057/jors.1977.92
[25]	X. Yuan, Y. Yao, X. Li, M. Feng, Impact of time-dependent infection rate and self-isolation awareness on networked epidemic propagation, Nonlinear Dynam., 112 (2024), 15653–15669. https://doi.org/10.1007/s11071-024-09832-0 doi: 10.1007/s11071-024-09832-0
[26]	X. Yuan, Y. Yao, H. Wu, M. Feng, Impacts of Physical-Layer Information on Epidemic Spreading in Cyber-Physical Networked Systems, IEEE Transact. Circuits Syst. I Regular Papers, (2025), 1–13. https://doi.org/10.1109/TCSI.2025.3550386
[27]	A. Zardini, M. Galli, M. Tirani, D. Cereda, M. Manica, F. Trentini, et al., A quantitative assessment of epidemiological parameters required to investigate COVID-19 burden, Epidemics, 37 (2021), 100530. https://doi.org/10.1016/j.epidem.2021.100530 doi: 10.1016/j.epidem.2021.100530
[28]	F. Riccardo, M. Ajelli, X. D. Andrianou, A. Bella, M. Del Manso, M. Fabiani, et al., Epidemiological characteristics of COVID-19 cases and estimates of the reproductive numbers 1 month into the epidemic, Italy, 28 January to 31 March 2020, Eurosurveillance, 25 (2020), 2000790.
[29]	J. Tang, Y. Yao, M. Xie, M. Feng, SIS Epidemic Modelling on Homogeneous Networked System: General Recovering Process and Mean-Field Perspective, Appl. Math. Model., (2025), 116188. https://doi.org/10.1016/j.apm.2025.116188
[30]	S. M. Ross, Introduction to probability models, Academic press, 2014.
[31]	J. A. Jacquez, A note on chain-binomial models of epidemic spread: what is wrong with the Reed-Frost formulation?, Math. Biosci., 87 (1987), 73–82. https://doi.org/10.1016/0025-5564(87)90034-4 doi: 10.1016/0025-5564(87)90034-4
[32]	D. Silver, Lectures on Reinforcement Learning, howpublished, 2015. Available from: https://www.davidsilver.uk/teaching/

Reader Comments

Your name:*

Email:*
© 2025 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)