Developing a Deep Neural Network model for COVID-19 diagnosis based on CT scan images

1.   Introduction

Optogenetics is a recent and innovative technique which allows to induce or prevent electric shocks in living tissues, by means of light stimulation. Successfully demonstrated in mammalian neurons in 2005 ([8]), the technique relies on the genetic modification of cells to make them express particular ionic channels, called rhodopsins, whose opening and closing are directly triggered by light stimulation. One of these rhodopsins comes from an unicellular flagellate algae, Chlamydomonas reinhardtii, and has been baptized Channelrodhopsins-2 (ChR2). It is a cation channel that opens when illuminated with blue light.

Since the field of Optogenetics is young, the mathematical modeling of the phenomenon is quite scarce. Some models have been proposed, based on the study of the photocycles initiated by the absorption of a photon. In 2009, Nikolic and al. [33] proposed two models for the ChR2 that are able to reproduce the photocurrents generated by the light stimulation of the channel. Those models are constituted of several states that can be either conductive (the channel is open) or non-conductive (the channel is closed). Transitions between those states are spontaneous, depend on the membrane potential or are triggered by the absorption of a photon. For example, the four-states model of Nikolic and al. [33] has two open states ($o_1$ and $o_2$) and two closed states ($c_1$ and $c_2$). Its transitions are represented on Figure 1.

The purpose of this paper is to extend to infinite dimension the optimal control of Piecewise Deterministic Markov Processes (PDMPs) and to define an infinite-dimensional controlled Hodgkin-Huxley model, containing ChR2 channels, as an infinite-dimensional controlled PDMP and prove existence of optimal ordinary controls. We now give the definition of the model.

We consider an axon, described as a 1-dimensional cable and we set $I=[0,1]$ (the more physical case $I=[-l,l]$ with $2l>0$ the length of the axon is included here by a scaling argument). Let $D_{ChR2} := \{o_1,o_2,c_1,c_2\}$. Individually, a ChR2 features a stochastic evolution which can be properly described by a Markov Chain on the finite space constituted of the different states that the ChR2 can occupy. In the four-states model above, two of the transitions are triggered by light stimulation, in the form of a parameter $u$ that can evolve in time. Here $u(t)$ is physically proportional to the intensity of the light with which the protein is illuminated. For now, we will consider that when the control is on (i.e., when the light is on), the entire axon is uniformly illuminated. Hence for all $t\geq 0$, $u(t)$ features no spatial dependency.

The deterministic Hodgkin-Huxley model was introduced in [30]. A stochastic infinite-dimensional model was studied in [4], [10], [27] and [39]. The Sodium ($Na^+$) channels and Potassium ($K^+$) channels are described by two pure jump processes with state spaces $D_1:=\{n_0,n_1,n_2,n_3,n_4\}$ and

$D_2:=\{m_0h_1,m_1h_1,m_2h_1,m_3h_1,m_0h_0,m_1h_0,m_2h_0,m_3h_0\}.$

For a given scale $N\in \mathbb{N^*}$, we consider that the axon is populated by $N_{hh}=N-1$ channels of type $Na^+$, $K^+$ or $ChR2$, at positions $\frac{1}{N}(\mathbb{Z}\cap N \mathring{I})$. In the sequel we will use the notation $I_N:=\mathbb{Z}\cap N \mathring{I}$. We consider the Gelfand triple ($V,H,V^*$) with $V := H_0^1(I)$ and $H :=L^2(I)$. The process we study is defined as a controlled infinite-dimensional Piecewise Deterministic Markov Process (PDMP). All constants and auxiliary functions in the next definition will be defined further in the paper.

Definition 1.1. Stochastic controlled infinite-dimensional Hodgkin-HuxleyChR2 model. Let $N\in \mathbb{N^*}$. We call $N^{\mathrm{th}}$ stochastic controlled infinite-dimensional Hodgkin-Huxley-ChR2 model the controlled PDMP $(v(t),d(t))\in V\times D_N$ defined by the following characteristics:

• A state space $V\times D_N$ with $D_N = D^{I_N}$ and $D= D_1\cup D_2 \cup D_{ChR2}$.

• A control space $U=[0,u_{max}]$, $u_{max}>0$.

• A set of uncontrolled PDEs: For every $d\in D_N$,

with

with $\delta_z\in V^*$ the Dirac mass at $z\in I$, $\rho >0$ a constant and $C_m>0$ the membrane capacitance. For $x\in \{K, Na, L, ChR2\}$, $g_x > 0$ is the normalized conductance of the channel of type $x$ and $V_x\in \mathbb{R}$ is the driving potential of the channel. See Section 5 for more details.

• A jump rate function $\lambda : V\times D_N\times U\rightarrow \mathbb{R}_+$ defined for all $(v,d,u)\in H\times D_N\times U$ by

with $\sigma_{x,y} :\mathbb{R}\times U\rightarrow \mathbb{R}_+^{*}$ smooth functions for all $(x,y)\in D^2$. See Table 1 in Section 5.1 for the expression of those functions.

• A discrete transition measure $\mathcal{Q}:V\times D_N\times U\rightarrow \mathcal{P}(D_N)$ defined for all $(v,d,u)\in E\times D_N\times U$ and $y\in D$ by

where $d^{i:y}$ is obtained from $d$ by putting its $i^{\mathrm{th}}$ component equal to $y$.

This model can be physically understood as follows. Between jumps of the stochastic component, the membrane potential evolves according to the Hodgkin-Huxley dynamics with fixed conductances given by the state of the piecewise constant stochastic component (equation (2)). The propagation of the potential along the membrane is governed by the Laplacian term $\frac{1}{C_m}\Delta$ and we obtain equation (1). The piecewise stochastic component gives the configuration of the ion channels, taking values in $D$, along the axon. A jump of this component represent a change of the state of one ion channel along the axon (basically the opening or the closing of a channel). The intensity of this jumping process is given by the rate functions defining the dynamics of the ion channel variables in the deterministic Hodgkin-Huxley model (see Table 1 in Section 5.1 for the equations of the deterministic finite-dimensional Hodgkin-Huxley model).

Remark 1. If the PDEs in Definition 1.1 do not depend on the control variable, the theory developed in this papers addresses optimal control problems where the function $f_d$ can be controlled (see Section 5.2).

The optimal control problem consists in mimicking an output signal that encodes a given biological behavior, while minimizing the intensity of the light applied to the neuron. For example, it can be a time-constant signal and in this case, we want to change the resting potential of the neuron to study its role on its general behavior. We can also think of pathological behaviors that would be fixed in this way. The minimization of light intensity is crucial because the range of intensity experimentally reachable is quite small and is always a matter of preoccupation for experimenters. These considerations lead us to formulate the following mathematical optimal control problem.

Suppose we are given a reference signal $V_{ref}\in V$. The control problem is then to find $\alpha \in \mathcal{A}$ that minimizes the following expected cost

where $\mathcal{A}$ is the space of control strategies, $\Upsilon$ an auxiliary state space that comprises $V\times D_N$, $X_{\cdot}^{\alpha}$ is the controlled PDMP and $X_{\cdot}^{\alpha}(\phi)$ its continuous component.

We will prove the following result.

Theorem 1.2. Under the assumptions of Section 2.1, there exists an optimal control strategy $\alpha^*\in\mathcal{A}$ such that for all $z\in\Upsilon$,

and the value function $z\rightarrow \inf_{\alpha\in\mathcal{A}} J_z(\alpha)$ is continuous on $\Upsilon$.

Piecewise Deterministic Markov Processes constitute a large class of Markov processes suited to describe a tremendous variety of phenomena such as the behavior of excitable cells ([4], [10], [36]), the evolution of stocks in financial markets ([11]) or the congestion of communication networks ([22]), among many others. PDMPs can basically describe any non diffusive Markovian system. The general theory of PDMPs, and the tools to study them, were introduced by Davis ([18]) in 1984, at a time when the theory of diffusion was already amply developed. Since then, they have been widely investigated in terms of asymptotic behavior, control, limit theorems and CLT, numerical methods, among others (see for instance [9], [14], [15], [17] and references therein). PDMPs are jump processes coupled with a deterministic evolution between the jumps. They are fully described by three local characteristics: the deterministic flow $\phi$, the jump rate $\lambda$, and the transition measure $\mathcal{Q}$. In [18], the temporal evolution of a PDMP between jumps (i.e., the flow $\phi$) is governed by an Ordinary Differential Equation (ODE). For that matter, this kind of PDMPs will be referred to as finite-dimensional in the sequel.

Optimal control of such processes have been introduced by Vermes ([40]) in finite dimension. In [40], the class of piecewise open-loop controls is introduced as the proper class to consider to obtain strongly Markovian processes. A Hamilton-Jabobi-Bellman equation is formulated and necessary and sufficient conditions are given for the existence of optimal controls. The standard broader class of so-called relaxed controls is considered and it plays a crucial role in getting the existence of optimal controls when no convexity assumption is imposed. This class of controls has been studied, in the finite-dimensional case, by Gamkrelidze ([26]), Warga ([42] and [41]) and Young ([45]). Relaxed controls provide a compact class that is adequate for studying optimization problems. Still in finite dimension, many control problems have been formulated and studied such as optimal control ([25]), optimal stopping ([16]) or controllability ([28]). In infinite dimension, relaxed controls were introduced by Ahmed ([1], [2], [3]). They were also studied by Papageorgiou in [37] where the author shows the strong continuity of relaxed trajectories with respect to the relaxed control. This continuity result will be of great interest in this paper.

A formal infinite-dimensional PDMP was defined in [10] for the first time, the set of ODEs being replaced by a special set of Partial Differential Equations (PDE). The extended generator and its domain are provided and the model is used to define a stochastic spatial Hodgkin-Huxley model of neuron dynamics. The optimal control problem we have in mind here regards those Hodgkin-Huxley type models. Seminal work on an uncontrolled infinite-dimensional Hodgkin-Huxley model was conducted in [4] where the trajectory of the infinite-dimensional stochastic system is shown to converge to the deterministic one, in probability. This type of model has then been studied in [39] in terms of limit theorems and in [27] in terms of averaging. The extension to infinite dimension heavily relies on the fact that semilinear parabolic equations can be interpreted as ODEs in Hilbert spaces.

To give a sense to Definition 1.1 and to Theorem 1.2, we will define a controlled infinite-dimensional PDMP for which the control acts on the three local characteristics. We consider controlled semilinear parabolic PDEs, jump rates $\lambda$ and transition measures $\mathcal{Q}$ depending on the control. This kind of PDE takes the form

where $L$ is the infinitesimal generator of a strongly continuous semigroup and $f$ is some function (possibly nonlinear). The optimal control problem we address is the finite-time minimization of an unbounded expected cost functional along the trajectory of the form

where $x(\cdot)$ is the continuous component of the PDMP, $u(\cdot)$ the control and $T>0$ the finite time horizon, the cost function $c(\cdot,\cdot)$ being potentially unbounded.

To address this optimal control problem, we use the fairly widespread approach that consists in studying the imbedded discrete-time Markov chain composed of the times and the locations of the jumps. Since the evolution between jumps is deterministic, there exists a one-to-one correspondence between the PDMP and a pure jump process that enable to define the imbedded Markov chain. The discrete-time Markov chain belongs to the class of Markov Decision Processes (MDPs). This kind of approach has been used in [25] and [12] (see also the book [31] for a self-contained presentation of MDPs). In these articles, the authors apply dynamic programming to the MDP derived from a PDMP, to prove the existence of optimal relaxed strategies. Some sufficient conditions are also given to get non-relaxed, also called ordinary, optimal strategies. However, in both articles, the PDMP is finite dimensional. To the best of our knowledge, the optimal control of infinite-dimensional PDMPs has not yet been treated and this is one of our main objectives here, along with its motivation, derived from the Optogenetics, to formulate and study infinite-dimensional controlled neuron models.

The paper is structured as follows. In Section 2 we adapt the definition of a standard infinite-dimensional PDMP given in [10] in order to address control problems of such processes. To obtain a strongly Markovian process, we enlarge the state space and we prove an extension to controlled PDMPs of [10,Theorem 4]. We also define in this section the MDP associated to our controlled PDMP and that we study later on. In Section 3 we use the results of [37] to define relaxed controlled PDMPs and relaxed MDPs in infinite dimension. Section 4 gathers the main results of the paper. We show that the optimal control problems of PDMPs and of MDPs are equivalent. We build up a general framework in which the MDP is contracting. The value function is then shown to be continuous and existence of optimal relaxed control strategies is proved. We finally give in this section, some convexity assumptions under which an ordinary optimal control strategy can be retrieved.

The final Section 5 is devoted to showing that the previous theoretical results apply to the model of Optogenetics previously introduced. Several variants of the model are discussed, the scope of the theoretical results being much larger than the model of Definition 1.1.

2.   Theoretical framework for the control of infinite-dimensional PDMPs

2.1.   The enlarged process and assumptions

In the present section we define the infinite-dimensional controlled PDMPs that we consider in this paper in a way that enables us to formulate control problems in which the three characteristics of the PDMP depend on an additional variable that we call the control parameter. In particular we introduce the enlarged process which enable us to address optimization problems in the subsequent sections.

Let $(\Omega,\mathcal{F},(\mathcal{F}_t)_{t\geq 0},\mathbb{P})$ be a filtered probability space satisfying the usual conditions. We consider a Gelfand triple ($V \subset H \subset V^*$) such that $H$ is a separable Hilbert space and $V$ a separable, reflexive Banach space continuously and densely embedded in $H$. The pivot space $H$ is identified with its dual $H^*$, $V^*$ is the topological dual of $V$. $H$ is then continuously and densely embedded in $V^*$. We will denote by $||\cdot||_V$, $||\cdot||_H$, and $||\cdot||_{V^*}$ the norms on $V$, $H$, and $V^*$, by $(\cdot,\cdot)$ the inner product in $H$ and by $\langle\cdot,\cdot \rangle$ the duality pairing of ($V,V^*$). Note that for $v\in V$ and $h\in H$, $\langle h, v \rangle = (h,v)$.

Let $D$ be a finite set, the state space of the discrete variable and $Z$ a compact Polish space, the control space. Let $T>0$ be the finite time horizon. Intuitively a controlled PDMP $(v_t,d_t)_{t\in[0,T]}$ should be constructed on $H\times D$ from the space of ordinary control rules defined as

where $U$, the action space, is a closed subset of $Z$. Elements of $A$ are defined up to a set in $[0,T]$ of Lebesgue measure 0. The control rules introduced above are called ordinary in contrast with the relaxed ones that we will introduce and use in order to prove existence of optimal strategies. When endowed with the coarsest $\sigma$-algebra such that

is measurable for all bounded and measurable functions $w:\mathbb{R}_+ \times U \rightarrow \mathbb{R}$, the set of control rules $A$ becomes a Borel space (see [46,Lemma 1]). This will be crucial for the discrete-time control problem that we consider later. Conditionally to the continuous component $v_t$ and the control $a(t)$, the discrete component $d_t$ is a continuous-time Markov chain given by a jump rate function $\lambda : H\times D\times U\rightarrow \mathbb{R}_+$ and a transition measure $\mathcal{Q} : H\times D\times U\rightarrow \mathcal{P}(D)$.

Between two consecutive jumps of the discrete component, the continuous component $v_t$ solves a controlled semilinear parabolic PDE

For $(v,d,a)\in H\times D\times A$ we will denote by $\phi^a(v,d)$ the flow of (6). Let $T_n,n\in\mathbb{N}$ be the jump times of the PDMP. Their distribution is then given by

for $t\in[T_n;T_{n+1})$. When a jump occurs, the distribution of the post jump state is given by

The triple ($\lambda, \mathcal{Q}, \phi$) fully describes the process and is referred to as the local characteristics of the PDMP.

We will make the following assumptions on the local characteristics of the PDMP.

(H($\lambda$)): For every $d\in D$, $\lambda_d: H\times Z \rightarrow \mathbb{R}_+$ is a function such that:

1. There exist $M_\lambda, \delta >0$ such that:

2. $z\rightarrow \lambda_d(x,z)$ is continuous on $Z$, for all $x\in H$.

3. $x\rightarrow \lambda_d(x,z)$ is locally Lipschitz continuous, uniformly in $Z$, that is, for every compact set $K\subset H$, there exists $l_{\lambda}(K)>0$ such that

(H($\mathcal{Q}$)): The function $\mathcal{Q} : H\times D\times Z\times \mathcal{B}(D) \rightarrow [0,1]$ is a transition probability such that: $(x,z)\rightarrow \mathcal{Q}(\{p\} | x,d,z)$ is continuous for all $(d,p)\in D^2$ (weak continuity) and $\mathcal{Q}(\{d\}|x,d,z)=0$ for all $(x,z)\in H\times Z$.

(H($L$)): $L:V\rightarrow V^*$ is such that:

1. $L$ is linear, monotone;

2. $||Lx||_{V^*}\leq c + c_1 ||x||_V$ with $c>0$ and $c_1\geq 0$;

3. $\langle Lx,x \rangle \geq c_2 ||x||_{V}^2, c_2 > 0$;

4. $-L$ generates a strongly continuous semigroup $(S(t))_{t\geq 0}$ on $H$ such that $S(t):H\to H$ is compact for every $t>0$. We will denote by $M_S$ a bound, for the operator norm, of the semigroup on $[0,T]$.

(H($f$)): For every $d\in D$, $f_d:H\times Z \rightarrow H$ is a function such that:

1. $x\rightarrow f_d(x,z)$ is Lipschitz continuous, uniformly in $Z$, that is,

2. $(x,z) \rightarrow f_d(x,z)$ is continuous from $H\times Z$ to $H_w$, where $H_w$ denotes the space $H$ endowed with the topology of weak convergence.

Let us make some comments on the assumptions above. Assumption (H($\lambda$))1. will ensure that the process is regular, i.e. the number of jumps of $d_t$ is almost surely finite in every finite time interval. Assumption (H($\lambda$))2. will enable us to construct relaxed trajectories. Assumptions (H($\lambda$))3. and (H($\mathcal{Q}$)) will be necessary to obtain the existence of optimal relaxed controls for the associated MDP. Assumptions (H($L$))1.2.3. (H($f$)) will ensure the existence and uniqueness of the solution of (6). Note that all the results of this paper are unchanged if assumption (H($f$))1 is replaced by

(H(f))': For every $d\in D$, $f_d:H\times Z \rightarrow H$ is a function such that:

1. $x\rightarrow -f_d(x,z)$ is continuous monotone, for all $z\in Z$.

2. $||f_d(x,z)||_H \leq b_1 + b_2||x||_H, b_1\geq 0, b_2>0$, for all $z\in Z$.

In particular, assumption (H($f$)) implies (H(f))'2. and we will use the constants $b_1$ and $b_2$ further in this paper. Note that they can be chosen uniformly in $D$ since it is a finite set. To see this, note that $z\to f_d(0,z)$ is a weakly continuous on the compact space $Z$ and thus weakly bounded. It is then strongly bounded by the Uniform Boundedness Principle.

Finally, assumptions (H($f$))3. and (H($L$))4. will respectively ensure the existence of relaxed solutions of (6) and the strong continuity of these solutions with regards to the relaxed control. For that last matter, the compactness of $Z$ is also required. The following theorem is a reminder that the assumption on the semigroup does not make the problem trivial since it implies that $L$ is unbounded when $H$ is infinite-dimensional.

Theorem 2.1. (see [24,Theorem 4.29])

1. For a strongly continuous semigroup $(T(t))_{t\geq 0}$ the following properties are equivalent

(a) $(T(t))_{t\geq 0}$ is immediately compact.

(b) $(T(t))_{t\geq 0}$ is immediately norm continuous, and its generator has compact resolvent.

2. Let $X$ be a Banach space. A bounded operator $A\in\mathcal{L}(X)$ has compact resolvent if and only if $X$ is finite-dimensional.

We define $\mathcal{U}_{ad}((0,T),U) := \{a \in L^1((0,T), Z) | a(t) \in U \text{ a.e.}\} = A$ ($Z$ is compact and so is $U$) the space of admissible rules. Because of (H($L$)) and (H($f$)), for all $a\in \mathcal{U}_{ad}((0,T),U)$, (6) has a unique solution belonging to $L^2((0,T),V)\cap H^1((0,T),V^*)$ and moreover, the solution belongs to $C([0,T],H)$ (see [37] for the construction of such a solution). We will make an extensive use of the mild formulation of the solution of (6), given by

with $\phi^{a}_0(v,d)= v$. One of the keys in the construction of a controlled PDMP in finite or infinite dimension is to ensure that $\phi^{a}$ enjoys the flow property $\phi^{a}_{t+s}(v,d) = \phi^{a}_s(\phi^{a}_t(v,d),d)$ for all $(v,d,a)\in H\times D\times \mathcal{U}_{ad}((0,T),U)$ and $(t,s)\in\mathbb{R}_+$. It is the flow property that guarantees the Markov property for the process. Under the formulation (9), it is easy to see that the solution $\phi^{a}$ cannot feature the flow property for any reasonable set of admissible rules. In particular, the jump process $(d_t,t\geq 0)$ given by (7) and (8) is not Markovian. Moreover in control problems, and especially in Markovian control problems, we are generally looking for feedback controls which depend only on the current state variable so that at any time, the controller needs only to observe the current state to be able to take an action. Feedback controls would ensure the flow property. However they impose a huge restriction on the class of admissible controls. Indeed, feedback controls would be functions $u: H\times D\rightarrow U$ and for the solution of (6) to be uniquely determined, the function $x\rightarrow f_d(x,u(x,d))$ needs to be Lipschitz continuous. It would automatically exclude discontinuous controls and therefore would not be adapted to control problems. To avoid this issue, Vermes introduced piecewise open-loop controls (see [40]): after a jump of the discrete component, the controller observes the location of the jump, say $(v,d)\in H\times D$ and chooses a control rule $a \in \mathcal{U}_{ad}((0,T),U)$ to be applied until the next jump. The time elapsed since the last jump must then be added to the state variable in order to see a control rule as a feedback control. While Vermes [40] and Davis [19] only add the last post jump location we also want to keep track of the time of the last jump in order to define proper controls for the Markov Decision Processes that we introduce in the next section, and to eventually obtain optimal feedback policies. According to these remarks, we now enlarge the state space and define control strategies for the enlarged process. We introduce first several sets that will be useful later on.

Definition 2.2. We define the sets $D(T,2) := \{(t,s)\in[0,T]^2 \mid t+s \leq T\}$, $\Xi := H\times D\times D(T,2)\times H$ and $\Upsilon :=H\times D\times [0,T]$.

Definition 2.3. Control strategies. Enlarged controlled PDMP. Survival function.

a) The set $\mathcal{A}$ of admissible control strategies is defined by

b) On $\Xi$ we define the enlarged controlled PDMP $(X^{\alpha}_t)_{t\geq 0}=(v_t,d_t,\tau_t,h_t, \nu_t)_{t\geq 0}$ with strategy $\alpha\in\mathcal{A}$ as follows:

• $(v_t,d_t)_{t\geq 0}$ is the process defined by (6), (7) and (8),

• $\tau_t$ is the time elapsed since the last jump at time $t$,

• $h_t$ is the time of the last jump before time $t$,

• $\nu_t$ is the post jump location right after the jump at time $h_t$.

c) Let $z:=(v,d,h)\in\Upsilon$. For $a\in\mathcal{U}_{ad}([0,T];U)$ we will denote by $\chi_.^{a}(z)$ the solution of

and its immediate extension $\chi_.^{\alpha}(z)$ to $\mathcal{A}$ such that the process $(X_t^{\alpha})_{t\geq 0}$ starting at $(v,d,0,h,v)\in \Xi,$ admits $\chi_.^{\alpha}$ as survival function:

The notation $\phi_t^{a}(z)$ means here

and $\phi_t^{\alpha}(z)$ is to be understood as $\phi_t^{\alpha(z)}(z)$.

Remark 2. ⅰ) Thanks to [46,Lemma 3], the set of admissible control strategies is in bijection with

and thus can be seen as a set of measurable feedback controls acting on $\Xi$ (but not depending on the first component) and with values in $U$. The formulation of Definition 2.3 is adequate to address the associated discrete-time control problem in Section 2.3.

ⅱ) In view of Definition 2.3, given $\alpha\in \mathcal{A}$, the deterministic dynamics of the process $(X^{\alpha}_t)_{t\geq 0} = (v_t,d_t,\tau_t,,h_t,\nu_t)_{t\geq 0}$ between two consecutive jumps obeys the initial value problem

with $s$ the last time of jump. The jump rate function and transition measure of the enlarged PDMP are straightforwardly given by the ones of the original process and will be denoted the same (see Appendix A for their expression).

ⅲ) If the relation $t=h_t+\tau_t$ indicates that the variable $h_t$ might be redundant, recall that we keep track of it on purpose. Indeed, the optimal control will appear as a function of the jump times so that keeping them as a variable will make the control feedback.

ⅳ) Because of the special definition of the enlarged process, for every control strategy in $\mathcal{A}$, the initial point of the process $(X_t^{\alpha})_{t\geq 0}$ cannot be any point of the enlarged state space $\Xi$. More precisely we introduce in Definition 2.4 below the space of coherent initial points.

Definition 2.4. Space of coherent initial points.

Take $\alpha\in\mathcal{A}$ and $x:=(v_0,d_0,0,h_0,v_0)\in \Xi$ and extend the notation $\phi_t^{\alpha}(x)$ of Definition 2.3 to $\Xi$ by

The set $\Xi^{\alpha}\subset\Xi$ of coherent initial points is defined as follows

Then we have for all $x:=(v_0,d_0,\tau_0,h_0,\nu_0)\in \Xi^{\alpha}$,

Note that $(X^{\alpha}_t)$ can be constructed like any PDMP by a classical iteration that we recall in Appendix A for the sake of completeness.

Proposition 1. The flow property.

Take $\alpha\in\mathcal{A}$ and $x:=(v_0,d_0,\tau_0,h_0,\nu_0)\in \Xi^{\alpha}$. Then $\phi_{t+s}^{\alpha}(x) = \phi_{t}^{\alpha}(\phi_{s}^{\alpha}(x),d_s,\tau_{s}, h_s,\nu_s)$ for all $(t,s)\in\mathbb{R}_+^2$.

Based on equation (12) and the definition of $\Xi^{\alpha}$, the proof of Proposition 1 is straightforward.

Notation. Let $\alpha\in\mathcal{A}$. For $z\in\Upsilon$, we will use the notation $\alpha_s(z):=\alpha(z)(s)$. Furthermore, we will sometimes denote by $\mathcal{Q}_{\alpha}(\cdot|v,d)$ instead of $\mathcal{Q}(\cdot|v,d,\alpha_{\tau}(\nu,d,h))$ for all $(v,d,\tau,h,\nu)\in \mathcal{A}\times \Xi^{\alpha}$.

2.2.   A probability space common to all strategies

Up to now, thanks to Definition 2.3, we can formally associate the PDMP $(X^{\alpha}_t)_{t\in \mathbb{R}_+}$ to a given strategy $\alpha\in\mathcal{A}$. However, we need to show that there exists a filtered probabily space satisfying the usual conditions under which, for every control strategy $\alpha \in \mathcal{A}$, the controlled PDMP $(X_t^{\alpha})_{t\geq 0}$ is a homogeneous strong Markov process. This is what we do in the next theorem which provides an extension of [10,Theorem 4] to controlled infinite-dimensional PDMPs and some estimates on the continuous component of the PDMP.

Theorem 2.5. Assume that assumptions (H(λ)), (H($\mathcal{Q}$)), (H($L$)) and (H($f$)) (or (H(f))') are satisfied.

a) There exists a filtered probability space satisfying the usual conditions such that for every control strategy $\alpha\in \mathcal{A}$ the process $(X_t^{\alpha})_{t\geq0}$ introduced in Definition 2.3 is a homogeneous strong Markov process on $\Xi$ with extended generator $\mathcal{G}^{\alpha}$ given in Appendix B.

b) For every compact set $K\subset H$, there exists a deterministic constant $c_K > 0$ such that for all control strategy $\alpha\in\mathcal{A}$ and initial point $x:=(v,d,\tau,h,\nu)\in \Xi^{\alpha}$, with $v \in K$, the first component $v_t^{\alpha}$ of the control PDMP $(X_t^{\alpha})_{t\geq 0}$ starting at $x$ is such that

The proof of Theorem 2.5 is given in Appendix B. In the next section, we introduce the MDP that will allow us to prove the existence of optimal strategies.

2.3.   A Markov Decision Process (MDP)

Because of the particular definition of the state space $\Xi$, the state of the PDMP just after a jump is in fact fully determined by a point in $\Upsilon$. In Appendix B we recall the one-to-one correspondence between the PDMP on $\Xi$ and the included pure jump process $(Z_n)_{n\in\mathbb{N}}$ with values in $\Upsilon$. This pure jump process allows to define a Markov Decision Process $(Z'_n)_{n\in\mathbb{N}}$ with values in $\Upsilon\cup\{\Delta_{\infty}\}$, where $\Delta_{\infty}$ is a cemetery state added to $\Upsilon$ to define a proper MDP. In order to lighten the notations, the dependence on a control strategy $\alpha\in\mathcal{A}$ of both jump processes is implicit. The stochastic kernel $\mathcal{Q}'$ of the MDP satisfies

for any $z:=(v,d,h)\in \Upsilon$, Borel sets $B\subset H$, $C\subset D$, $E\subset [0,T]$, and $a \in \mathcal{U}_{ad}([0,T],U)$, where

with $\phi_{t}^{a}(z)$ given by (9) and $\mathcal{Q}'(\{\Delta_{\infty}\}|z ,a) = \chi_{T-h}^{a}(z)$, and $\mathcal{Q}'(\{\Delta_{\infty}\}|\Delta_{\infty},a) = 1$. The conditional jumps of the MDP $(Z'_n)_{n\in\mathbb{N}}$ are then given by the kernel $\mathcal{Q}'(\cdot | z, \alpha(z))$ for $(z,\alpha)\in\Upsilon\times\mathcal{A}$. Note that $Z'_n= Z_n$ as long as $T_n \leq T$, where $T_n$ is the last component of $Z_n$. Since we work with Borel state and control spaces, we will be able to apply techniques of [6] for discrete-time stochastic control problems, without being concerned by measurability matters. See [6,Section 1.2] for an illuminating discussion on these measurability questions.

3.   Relaxed controls

Relaxed controls are constructed by enlarging the set of ordinary ones, in order to convexify the original system, and in such a way that it is possible to approximate relaxed strategies by ordinary ones. The difficulty in doing so is twofold. First, the set of relaxed trajectories should not be much larger than the original one. Second, the topology considered on the set of relaxed controls should make it a compact set and, at the same time, make the flow of the associated PDE continuous. Compactness and continuity are two notions in conflict so being able to achieve such a construction is crucial. Intuitively a relaxed control strategy on the action space $U$ corresponds to randomizing the control action: at time $t$, instead of taking a predetermined action, the controller will take an action with some probability, making the control a transition probability. This has to be formalized mathematically.

Notation and reminder. $Z$ is a compact Polish space, $C(Z)$ denotes the set of all real-valued continuous, necessarily bounded, functions on $Z$, endowed with the supremum norm. Because $Z$ is compact, by the Riesz Representation Theorem, the dual space $[C(Z)]^*$ of $C(Z)$ is identified with the space $M(Z)$ of Radon measures on $\mathcal{B}(Z)$, the Borel $\sigma$-field of $Z$. We will denote by $M_+^1(Z)$ the space of probability measures on $Z$. The action space $U$ is a closed subset of $Z$. We will use the notations $L^1(C(Z)):=L^1((0,T),C(Z))$ and $L^{\infty}(M(Z)):=L^{\infty}((0,T),M(Z))$.

3.1.   Relaxed controls for a PDE

Let $\mathcal{B}([0,T])$ denote the Borel $\sigma$ -field of $[0,T]$ and $Leb$ the Lebesgue measure. A transition probability from $([0,T],\mathcal{B}([0,T]), Leb)$ into $(Z,\mathcal{B}(Z))$ is a function $\gamma : [0,T]\times \mathcal{B}(Z) \rightarrow [0,1]$ such that

We will denote by $\mathcal{R}([0,T],Z)$ the set of all transition probability measures from $([0,T],\mathcal{B}([0,T]), Leb)$ into $(Z,\mathcal{B}(Z))$.

Recall that we consider the PDE (6):

The relaxed PDE is then of the form

where $\mathcal{R}([0,T],U) := \{\gamma\in \mathcal{R}([0,T],Z) | \gamma(t)(U) = 1 \text{ a.e. in }[0,T] \}$ is the set of transition probabilities from $([0,T],\mathcal{B}([0,T]), Leb)$ into $(Z,\mathcal{B}(Z))$ with support in $U$. The integral part of (15) is to be understood in the sense of Bochner-Lebesgue as we show now. The topology we consider on $\mathcal{R}([0,T],U)$ follows from [5] and because $Z$ is a compact metric space, it coincides with the usual topology of relaxed control theory of [43]. It is the coarsest topology that makes continuous all mappings

for every Carathodory integrand $f:[0,T]\times Z\rightarrow \mathbb{R}$, a Carathodory integrand being such that

This topology is called the weak topology on $\mathcal{R}([0,T],Z)$ but we show now that it is in fact metrizable. Indeed, Carathodory integrands $f$ on $[0,T]\times Z$ can be identified with the Lebesgue-Bochner space $L^1(C(Z))$ via the application $t\rightarrow f(t,\cdot) \in L^1(C(Z))$. Now, since $M(Z)$ is a separable ($Z$ is compact), dual space (dual of $C(Z)$), it enjoys the Radon-Nikodym property. Using [21,Theorem 1 p. 98], it follows that $[L^1(C(Z))]^* = L^{\infty}( M(Z))$. Hence, the weak topology on $\mathcal{R}([0,T],Z)$ can be identified with the $w^*$-topology in $(L^{\infty}(M(Z)),L^1(C(Z)))$, the latter being metrizable since $L^1(C(Z))$ is a separable space (see [23,Theorem 1 p. 426]). This crucial property allows to work with sequences when dealing with continuity matters with regards to relaxed controls.

Finally, by Alaoglu's Theorem, $\mathcal{R}([0,T],U)$ is $w^*$-compact in $L^{\infty}(M(Z))$, and the set of original admissible controls $\mathcal{U}_{ad}([0,T],U)$ is dense in $\mathcal{R}([0,T],U)$ (see [5,Corollary 3 p. 469]).

For the same reasons why (14) admits a unique solution, by setting $\bar{f}_d(v,\gamma):=\int_Zf_d(v,u)\gamma(\mathrm{d}u)$, it is straightforward to see that $(15)$ admits a unique solution. The following theorem gathers the results of [37,Theorems 3.2 and 4.1] and will be of paramount importance in the sequel.

Theorem 3.1. If assumptions (H($L$)) and (H($f$)) (or (H(f))') hold, then

a) the space of relaxed trajectories (i.e. solutions of 15) is a convex, compact set of $C([0,T],H)$. It is the closure in $C([0,T],H)$ of the space of original trajectories (i.e. solutions of 14).

b) The mapping that maps a relaxed control to the solution of (15) is continuous from $\mathcal{R}([0,T],U)$ into $C([0,T],H)$.

3.2.   Relaxed controls for infinite-dimensional PDMPs

First of all, note that since the control acts on all three characteristics of the PDMP, convexity assumptions on the fields $f_d(v,U)$ would not necessarily ensure existence of optimal controls as it does for partial differential equations. Such assumptions should also be imposed on the rate function and the transition measure of the PDMP. For this reason, relaxed controls are even more important to prove existence of optimal controls for PDMP. For what has been done for PDE above, we are now able to define relaxed PDMPs. The next definition is the relaxed analogue of Definition 2.3.

Definition 3.2. Relaxed control strategies, relaxed local characteristics.

a) The set $\mathcal{A}^{\mathcal{R}}$ of relaxed admissible control strategies for the PDMP is defined by

Given a relaxed control strategy $\mu\in\mathcal{A}^{\mathcal{R}}$ and $z\in \Upsilon$, we will denote by $\mu^z:=\mu(z)\in \mathcal{R}([0,T];U)$ and $\mu^z_t$ the corresponding probability measure on $(Z,\mathcal{B}(Z))$.

b) For $\gamma\in M_+^1(Z)$, $(v,d)\in H\times D$ and $C\in\mathcal{B}(D)$, we extend the jump rate function and transition measure as follows

the expression for the enlarged process being straightforward. This allows us to give the relaxed survival function of the PDMP and the relaxed mild formulation of the solution of (15)

for $\mu\in\mathcal{A}^{\mathcal{R}}$ and $z:=(v,d,h)\in\Upsilon$. For $\gamma\in\mathcal{R}([0,T],U)$, we will also use the following notation

The following proposition is a direct consequence of Theorem 2.5b).

Proposition 2. For every compact set $K\subset H$, there exists a deterministic constant $c_K > 0$ such that for all control strategy $\mu\in\mathcal{A}^{\mathcal{R}}$ and initial point $x:=(v,d,\tau,h,\nu)\in \Xi^{\alpha}$, with $v \in K$, the first component $v_t^{\mu}$ of the control PDMP $(X_t^{\mu})_{t\geq 0}$ starting at $x$ is such that

The relaxed transition measure is given in the next section through the relaxed stochastic kernel of the MDP associated to our relaxed PDMP.

3.3.   Relaxed associated MDP

Let $z:=(v,d,h)\in\Upsilon$ and $\gamma\in\mathcal{R}([0,T],U)$. The relaxed stochastic kernel of the relaxed MDP satisfies

for Borel sets $B\subset H$, $C\subset D$, $E\subset [0,T]$, where

and $\mathcal{Q}'(\{\Delta_{\infty}\}|z ,\gamma) = \chi_{T-h}^{\gamma}(z)$, and $\mathcal{Q}'(\{\Delta_{\infty}\}|\Delta_{\infty},\gamma) = 1$, with, as before, the conditional jumps of the MDP $(Z'_n)_{n\in\mathbb{N}}$ given by the kernel $\mathcal{Q}'(\cdot | z, \mu(z))$ for $(z,\mu)\in\Upsilon\times\mathcal{A}^{\mathcal{R}}$.

4.   Main results

Here, we are interested in finding optimal controls for optimization problems involving infinite-dimensional PDMPs. For instance, we may want to track a targeted "signal" (as a solution of a given PDE, see Section 5). To do so, we are going to study the optimal control problem of the imbedded MDP defined in Section 2.3. This strategy has been for example used in [12] in the particular setting of a decoupled finite-dimensional PDMP, the rate function being constant.

4.1.   The optimal control problem

Thanks to the preceding sections we can consider ordinary or relaxed costs for the PDMP $X^\alpha$ or the MDP and their corresponding value functions. For $z:=(v,d,h)\in\Upsilon$ and $\alpha\in\mathcal{A}$ we denote by $\mathbb{E}_{z}^{\alpha}$ the conditional expectation given that $X_{h}^{\alpha}=(v,d,0,h,v)$ and by $X_s^{\alpha}(\phi)$ the first component of $X_s^{\alpha}$. Furthermore, we denote by $X_s^{\alpha}:=(v_s,d_s,\tau_s,h_s,\nu_s)$, then the shortened notation $\alpha(X_s^{\alpha})$ will refer to $\alpha_{\tau_s}(\nu_s,d_s,h_s)$. These notations are straightforwardly extended to $\mathcal{A}^{\mathcal{R}}$. We introduce a running cost $c : H\times Z\rightarrow \mathbb{R}_+$ and a terminal cost $g: H \rightarrow \mathbb{R}_+$ satisfying

(H($c$)): $(v,z)\rightarrow c(v,z)$ and $v\rightarrow g(v)$ are nonnegative quadratic functions, that is there exists $(a,b,c,d,e,f,g,h,i,j)\in \mathbb{R}^9$ such that for $v,z\in H\times Z$,

with $\bar{d}(\cdot,\cdot)$ the distance on $Z$.

Remark 3. This assumption might seem a bit restrictive, but it falls within the framework of all the applications we have in mind. More importantly, it can be widely loosened if we slightly change the assumptions of Theorem 4.3. In particular, all the following results, up to Lemma 4.14, are true and proved for continuous functions $c : H\times Z\rightarrow \mathbb{R}_+$ and $g: H \rightarrow \mathbb{R}_+$. See Remark 6 below.

Definition 4.1. Ordinary value function for the PDMP $\, X^\alpha$.

For $\alpha\in\mathcal{A} \, ,$ we define the ordinary expected total cost function $V_{\alpha}:\Upsilon\rightarrow \mathbb{R}$ and the corresponding value function $V$ as follows:

Assumption (H(c)) ensures that $V_{\alpha}$ and $V$ are properly defined.

Definition 4.2. Relaxed value function for the PDMP $X^{\mu}$.

For $\mu\in\mathcal{A}^{\mathcal{R}}$ we define the relaxed expected cost function $V_{\mu} : \Upsilon \rightarrow \mathbb{R}$ and the corresponding relaxed value function $\tilde{V}$ as follows:

We can now state the main result of this section.

Theorem 4.3. Under assumptions (H($\lambda$)), (H($\mathcal{Q}$)), (H($L$)), (H($f$)) and (H(c)), the value function $\tilde{V}$ of the relaxed optimal control problem on the PDMP is continuous on $\Upsilon$ and there exists an optimal relaxed control strategy $\mu^*\in\mathcal{A}^{\mathcal{R}}$ such that

Remark 4. All the subsequent results that lead to Theorem 4.3 would be easily transposable to the case of a lower semicontinuous cost function. We would then obtain a lower semicontinuous value function.

The next section is dedicated to proving Theorem 4.3 via the optimal control of the MDP introduced before. Let us briefly sum up what we are going to do. We first show that the optimal control problem of the PDMP is equivalent to the optimal control problem of the MDP and that an optimal control for the latter gives an optimal control strategy for the original PDMP. We will then build up a framework, based on so called bounding functions (see [12]), in which the value function of the MDP is the fixed point of a contracting operator. Finally, we show that under the assumptions of Theorem 4.3, the relaxed PDMP $X^\mu$ belongs to this framework.

4.2.   Optimal control of the MDP

Let us define the ordinary cost $c'$ on $\Upsilon\cup \{\Delta_{\infty}\}\times\mathcal{U}_{ad}([0,T];U)$ for the MDP defined in Section 2.3. For $z:=(v,d,h)\in \Upsilon$ and $a\in\mathcal{U}_{ad}([0,T];U)$,

and $c'(\Delta_{\infty},a):=0$.

Assumption (H(c)) allows $c'$ to be properly extended to $\mathcal{R}([0,T],U)$ by the formula

and $c'(\Delta_{\infty},\gamma) = 0$ for $(z,\gamma)\in \Upsilon\times \mathcal{R}([0,T],U)$. We can now define the expected cost function and value function for the MDP.

Definition 4.4. Cost and value functions for the MDP $\, (Z'_n)$.

For $\alpha \in \mathcal{A}\,$ (resp. $\mu\in \mathcal{A}^{\mathcal{R}}$), we define the total expected cost $J_{\alpha}$ (resp. $J_{\mu}$) and the value function $J$ (resp. $J'$)

for $z\in\Upsilon$ and with $\alpha(Z'_n)$ (resp. $\mu(Z'_n)$) being elements of $\mathcal{U}_{ad}([0,T],U)$ (resp. $\mathcal{R}([0,T],U)$).

The finiteness of these sums will by justified later by Lemma 4.9.

4.2.1.   The equivalence Theorem

In the following theorem we prove that the relaxed expected cost function of the PDMP equals the one of the associated MDP. Thus, the value functions also coincide. For the finite-dimensional case we refer the reader to [19] or [12] where the discrete component of the PDMP is a Poisson process and therefore the PDMP is entirely decoupled. The PDMPs that we consider are fully coupled.

Theorem 4.5. The relaxed expected costs for the PDMP and the MDP coincide: $V_{\mu}(z)=J_{\mu}(z)$ for all $z \in \Upsilon$ and relaxed control $\mu \in\mathcal{A}^{\mathcal{R}}$. Thus, the value functions $\tilde{V}$ and $J'$ coincide on $\Upsilon$.

Remark 5. Since we have $\mathcal{A}\subset \mathcal{A}^{\mathcal{R}}$, the expected costs $V_{\alpha}(z)$ and $J_{\alpha}(z)$ also coincide for all $z \in \Upsilon$ and ordinary control strategy $\alpha \in\mathcal{A}$

Proof. Let $\mu\in\mathcal{A}^{\mathcal{R}}$ and $z=(v,d,h)\in \Upsilon$ and consider the PDMP $X^{\mu}$ starting at $(v,d,0,h,v)\in\Xi^{\mu}$. We drop the dependence in the control in the notation and denote by $(T_n)_{n\in\mathbb{N}}$ the jump times, and $Z_n:=(v_{T_n},d_{T_n},T_n)\in \Upsilon$ the point in $\Upsilon$ corresponding to $X_{T_n}^{\mu}$. Let $H_n = (Z_0,\dots,Z_n)$, $T_n\leq T$. For a purpose of concision we will rewrite $\mu^n := \mu(Z_n) \in \mathcal{R}([0,T],U)$ for all $n\in\mathbb{N}$.

all quantities being non-negative. We want now to examine the two terms that we call $\mathcal{I}_1$ and $\mathcal{I}_2$ separately. For $n\in \mathbb{N},$ we start with

that we split according to $T_n\leq T <T_n+1$ or $T_{n+1}\leq T\,$ (if $\, T\leq T_n$, the corresponding term vanishes). Then

By the strong Markov property and the flow property, the first term on the RHS is equal to

Using the same arguments, the second term on the RHS of $\mathcal{I}_1$ can be written as

An integration by parts yields

Moreover

By definition of the Markov chain $(Z'_n)_{n\in\mathbb{N}}$ and the function $c'$, we then obtain for the total expected cost of the PDMP,

4.2.2.   Existence of optimal controls for the MDP

We now show existence of optimal relaxed controls under a contraction assumption. We use the notation $\mathcal{R}:=\mathcal{R}([0,T];U)$ in the sequel. Let us also recall some notations regarding the different control sets we consider.

• $u$ is an element of the control set $U$.

• $a:[0,T]\rightarrow U$ is an element of the space of admissible control rules $\mathcal{U}_{ad}([0,T],U)$

• $\alpha : \Upsilon\rightarrow \mathcal{U}_{ad}([0,T],U)$ is an element of the space of admissible strategies for the original PDMP.

• $\gamma : [0,T] \rightarrow M_+^1(Z)$ is an element of the space of relaxed admissible control rules $\mathcal{R}$.

• $\mu : \Upsilon \rightarrow \mathcal{R}$ is an element of the space of relaxed admissible strategies for the relaxed PDMP.

The classical way to address the discrete-time stochastic control problem that we introduced in Definition 4.4 is to consider an additional control space that we will call the space of Markovian policies and denote by $\Pi$. Formally $\Pi := \left(\mathcal{A}^\mathcal{R}\right)^{\mathbb{N}}$ and a Markovian control policy for the MDP is a sequence of relaxed admissible strategies to be applied at each stage. The optimal control problem is to find $\pi:=(\mu_n)_{n\in\mathbb{N}}\in\Pi$ that minimizes

Now denote by $J^*(z)$ this infimum. We will in fact prove the existence of a stationary optimal control policy that will validate the equality

Let us now define some operators that will be useful for our study and state the first theorem of this section. Let $w:\Upsilon \rightarrow \mathbb{R}$ a continuous function, $(z,\gamma,\mu)\in \Upsilon\times \mathcal{R}\times\mathcal{A}^{\mathcal{R}}$ and define

where $(\mathcal{Q}'w)(z,\gamma):=\int_{\Upsilon} w(x)\mathcal{Q}'(\mathrm{d}x|z,\gamma)$ which admits also the expression

Theorem 4.6. Assume that there exists a subspace $\mathbb{C}$ of the space of continuous bounded functions from $\Upsilon$ to $\mathbb{R}$ such that the operator $\mathcal{T}: \mathbb{C}\rightarrow \mathbb{C}$ is contracting and the zero function belongs to $\mathbb{C}$. Assume furthermore that $\mathbb{C}$ is a Banach space. Then $J'$ is the unique fixed point of $\mathcal{T}$ and there exists an optimal control $\mu^*\in\mathcal{A}^{\mathcal{R}}$ such that

All the results needed to prove this Theorem can be found in [6]. We break down the proof into the two following elementary propositions, suited to our specific problem. Before that, recall that from [6,Proposition 9.1 p.216], $\Pi$ is the adequate control space to consider since history-dependent policies do not improve the value function.

Let us now consider the $n$-stages expected cost function and value function defined by

for $n\in\mathbb{N}$ and $\pi:=(\mu_n)_{n\in\mathbb{N}}\in\Pi$. We also set $J_{\infty} := \lim_{n\to \infty} J_n$.

Proposition 3. Let assumptions of Theorem 4.3 hold. Let $v,w: \Upsilon\to \mathbb{R}$ such that $v\leq w$ on $\Upsilon$, and let $\mu\in\mathcal{A}^{\mathcal{R}}$. Then $\mathcal{T}_{\mu}v \leq \mathcal{T}_{\mu}w$. Moreover

with $\pi:=(\mu_n)_{n\in\mathbb{N}}$ and $J_{\infty}$ is the unique fixed point of $\mathcal{T}$ in $\mathbb{C}$.

Proof. The first relation is straightforward since all quantities defining $\mathcal{Q}'$ are nonnegative. The equality $J_n = \inf_{\pi\in\Pi} \mathcal{T}_{\mu_0}\mathcal{T}_{\mu_1}\dots\mathcal{T}_{\mu_{n-1}} 0$ is also immediate since $\mathcal{T}_{\mu}$ just shifts the process of one stage (see also [6,Lemma 8.1,p194]).

Let $I\in\mathbb{C}$, $\varepsilon > 0$ and $n\in\mathbb{N}$. For every $k\in \{1..n-1\}$, $\mathcal{T}^kI\in\mathbb{C}$ and so there exist $\mu_0,\mu_1,\dots,\mu_{n-1}\in\left(\mathcal{A}^{\mathcal{R}}\right)^n$ such that

We then get

Since this last inequality is true for any $\varepsilon>0$ we get

and by definition of $\mathcal{T}$, $\mathcal{T}I \leq \mathcal{T}_{\mu_{n-1}} I$. Using the first relation of the proposition we get

Finally, $\mathcal{T}^nI = \inf_{\pi\in\Pi} \mathcal{T}_{\mu_{0}}\mathcal{T}_{\mu_{1}}\dots\mathcal{T}_{\mu_{n-1}} I$ for all $I\in \mathbb{C}$ and $n\in \mathbb{N}$. We deduce from the Banach fixed point theorem that $J_{\infty} = \lim_{n\to \infty} \mathcal{T}^n0$ belongs to $\mathbb{C}$ and is the only fixed point of $\mathcal{T}$.

Proposition 4. There exists $\mu^*\in\mathcal{A}^{\mathcal{R}}$ such that $J_{\infty} = J_{\mu^*} = J'$.

Proof. By definition, for every $\pi\in\Pi$, $J_n \leq J_{n\pi}$, so that $J_{\infty} \leq J^*$. Now from the previous proposition, $J_{\infty} = \inf_{\gamma\in\mathcal{R}} RJ_{\infty}(\cdot,\gamma)$, $\mathcal{R}$ is a compact space and $RJ_{\infty}$ is a continuous function. We can thus find a measurable mapping $\mu^* : \Upsilon \to \mathcal{R}$ such that $J_{\infty} = \mathcal{T}_{\mu*}J_{\infty}$. $J_{\infty}\geq 0$ so from the first relation of the previous proposition, for all $n\in\mathbb{N}$, $J_{\infty} = \mathcal{T}_{\mu*}^nJ_{\infty} \geq \mathcal{T}_{\mu*}^n 0$ and by taking the limit $J_{\infty} \geq J_{\mu^*}$. Since $J_{\mu^*} \geq J^*$ we get $J_{\infty} = J_{\mu^*} = J^*$. We conclude the proof by remarking that $J^* \leq J' \leq J_{\mu^*}$.

The next section is devoted to proving that the assumptions of Theorem 4.6 are satisfied for the MDP.

4.2.3.   Bounding functions and contracting MDP

The concept of bounding function that we define below will ensure that the operator $\mathcal{T}$ is a contraction. The existence of the space $\mathbb{C}$ of Theorem 4.6 will mostly result from Theorem 3.1 and again from the concept of bounding function.

Definition 4.7. Bounding functions for a PDMP.

Let $c$ (resp. $g$) be a running (resp. terminal) cost as in Section 4.1. A measurable function $b:H\rightarrow \mathbb{R}_+$ is called a bounding function for the PDMP if there exist constants $c_c,c_g,c_{\phi}\in \mathbb{R}_+$ such that

(ⅰ) $c(v,u)\leq c_cb(v)$ for all $(v,u)\in H\times Z$,

(ⅱ) $g(v) \leq c_gb(v)$ for all $v\in H$,

(ⅲ) $b(\phi_t^{\gamma}(z))\leq c_{\phi}b(v)$ for all $(t,z,\gamma)\in [0,T]\times\Upsilon\times \mathcal{R}$, $z=(v,d,h)$.

Given a bounding function for the PDMP we can construct one for the MDP with or without relaxed controls, as shown in the next lemma (cf. [13,Definition 7.1.2 p.195]).

Lemma 4.8. Let $b$ is a bounding function for the PDMP. We keep the notations of Definition 4.7. Let $\zeta>0$. The function $B_{\zeta} : \Upsilon \longmapsto \mathbb{R}_+$ defined by $B_{\zeta}(z):=b(v)e^{\zeta(T-h)}$ for $z=(v,d,h)$ is an upper bounding function for the MDP. The two inequalities below are satisfied for all $(z,\gamma)\in \Upsilon\times \mathcal{R}$,

Proof. Take $(z,\gamma)\in \Upsilon\times \mathcal{R}$, $z=(v,d,h)$. On the one hand from (24) and Definition 4.7 we obtain

which immediately implies (25). On the other hand

which implies (26).

Let $b$ be a bounding function for the PDMP. Consider $\zeta^*$ such that $\,C:= c_{\phi}\frac{M_{\lambda}}{\zeta^* + \delta}\; < 1$. Denote by $B^*$ the associated bounding function for the MDP. We introduce the Banach space

The following two lemmas give an estimate on the expected cost of the MDP that justifies manipulations of infinite sums.

Lemma 4.9. The inequality $\mathbb{E}_{z}^{\gamma}\left[B^*(Z'_k)\right] \leq C^k B^*(z)$ holds for any $(z,\gamma,k)\in \Upsilon\times \mathcal{R}\times \mathbb{N}$.

Proof. We proceed by induction on $k$. Let $z\in\Upsilon$. The desired inequality holds for $k=0$ since $\mathbb{E}_{z}^{\gamma}\left[B^*(Z'_0)\right]=B^*(z)$. Suppose now that it holds for $k\in \mathbb{N}$. Then

Using (26) and the definition of $C$, we conclude that $\mathbb{E}_{z}^{\gamma}\left[B^*(Z'_{k+1})\right]\leq C \mathbb{E}_{z}^{\gamma}\left[ B^*(Z'_k)\right]$ and by the assumption on $k$ $\mathbb{E}_{z}^{\gamma}\left[B^*(Z'_{k+1})\right]\leq C^{k+1}B^*(z)$.

Lemma 4.10. There exists $\kappa>0$ such that for any $(z,\mu)\in \Upsilon \times \mathcal{A}^\mathcal{R}$,

Proof. The results follows from Lemma 4.9 and from the fact that

for any $k\in \mathbb{N}$.

We now state the result on the operator $\mathcal{T}$.

Lemma 4.11. $\mathcal {T}$ is a contraction on $\mathcal{L}^*$: for any $(v,w)\in {\mathcal{L}^*}\times {\mathcal{L}^*}$,

where $C=c_{\phi}\frac{M_{\lambda}}{\zeta^* + \delta}$.

Proof. We prove here the contraction property. The fact $\mathcal{T}: \mathcal{L}^*\to\mathcal{L}^*$ is less straightforward and is addressed in the next section. Let $z:=(v,d,h)\in\Upsilon$. Let us recall that for functions $f,g:\mathcal{R}\rightarrow \mathbb{R}$

Moreover since $\inf_{\gamma\in \mathcal{R}} f(\gamma) - \inf_{\gamma\in \mathcal{R}} g(\gamma) = \sup_{\gamma\in \mathcal{R}} (-g(\gamma)) - \sup_{\gamma\in \mathcal{R}} (-f(\gamma))$, we have

where

so that

where

We then conclude that

4.2.4.   Continuity properties

Here we prove that the trajectories of the relaxed PDMP are continuous w.r.t. the control and that the operator $R$ transforms continuous functions in continuous functions.

Lemma 4.12. Assume that (H($L$)) and (H($f$)) are satisfied. Then the mapping

is continuous from $\Upsilon\times \mathcal{R}$ in $C([0,T];H)$.

Proof. This proof is based on the result of Theorem 3.1. Here we add the joint continuity on $\Upsilon\times \mathcal{R}$ whereas the continuity is just on $\mathcal{R}$ in [37]. Let $t\in[0,T]$ and let $(z,\gamma) \in \Upsilon\times \mathcal{R}$. Assume that $(z_n,\gamma_n)\rightarrow (z,\gamma)$. Since $D$ is a finite set, we take the discrete topology on it and if we denote by $z^n=(v_n,d_n,h_n)$ and $z=(v,d,h)$, we have the equality $d_n=d$ for $n$ large enough. So for $n$ large enough we have

From(H($f$))1. we get

where $\ell_n(t) := \int_0^t \int_Z S(t-s)[f_d(\phi_t^{\gamma}(z),u)\gamma_n(s)(\mathrm{d}u)-f_d(\phi_t^{\gamma}(z),u)\gamma(s)(\mathrm{d}u)]\mathrm{d}s.$ By the Gronwall lemma we obtain a constant $C>0$ such that

Since $\lim_{n\rightarrow +\infty}||v_n-v||_H=0$, the proof is complete if we show that the sequence of functions $(||\ell_n||_H)$ uniformly converges to 0.

Let us denote by $x_n(t):= \int_0^t \int_Z (h, S(t-s)f_d(\phi_t^{\gamma}(z),u)))_H\gamma_n(s)(\mathrm{d}u)\mathrm{d}s$. Using the same argument as the proof of [37,Theorem 3.1], there is no difficulty in proving that $(x_n)_{n\in\mathbb{N}}$ is compact in $C([0,T],H)$ so that, passing to a subsequence if necessary, we may assume that $x_n\to x$ in $C([0,T],H)$. Now let $h\in H$.

since $(t,u) \to ( h, S(t-s)f_d(\phi_t^{\gamma}(z),u)))_H \in L^1(C(Z))$ and $\gamma_n\to \gamma$ weakly* in $L^{\infty}(M(Z)) = [L^1(C(Z))]^*$. Thus, $x(t) = \int_0^t \int_Z S(t-s)f_d(\phi_t^{\gamma}(z),u)\gamma(s)(\mathrm{d}u)\mathrm{d}s$ and $\ell_n(t) = x_n(t)- x(t)$ for all $t\in[0,T]$, proving the uniform convergence of $||\ell_n||_H$ on $[0,T]$.

The next lemma establishes the continuity property of the operator $R$.

Lemma 4.13. Suppose that assumptions (H($L$)), (H($f$)), (H($\lambda$)), (H($\mathcal{Q}$)), (H(c)) are satisfied. Let $b$ be a continuous bounding function for the PDMP. Let $w:\Upsilon\times U \rightarrow \mathbb{R}$ be continuous with $|w(z,u)|\leq c_wB^*(z)$ for some $c_w\geq0$. Then

is continuous on $\Upsilon\times \mathcal{R}$, with $z:=(v,d,h)$. Quite straightforwardly,

is continuous on $\Upsilon\times \mathcal{R}$.

Proof. See Appendix C.

It now remains to show that there exists a bounding function for the PDMP. This is the result of the next lemma.

Lemma 4.14. Suppose assumptions (H($L$)), (H($f$)) and (H(c)) are satisfied. Now define $\tilde{c}$ and $\tilde{g}$ from $c$ and $g$ by taking the absolute value of the coefficients of these quadratic functions. Let $M_2>0$. Define $M_3 := (M_2+b_1T)M_Se^{M_Sb_2T}$ and $b:H\rightarrow \mathbb{R}_+$ by

is a continuous bounding function for the PDMP.

Proof. For all $(v,u)\in H\times U$, $c(v,u)\leq b(v)$ and $g(v)\leq b(v)$. Now let $(t,z,\gamma)\in [0,T]\times \Upsilon\times \mathcal{R}$, $z=(v,d,h)$.

• If $||\phi_t^{\gamma}(z)||_H \leq M_3$, $b(\phi_t^{\gamma}(z)) = b(M_3)$. If $||v||_H \leq M_3$ then $b(v) = b(M_3) = b(\phi_t^{\gamma}(z))$. Otherwise, $||v||_H > M_3$ and $b(v) > b(M_3) = b(\phi_t^{\gamma}(z))$.

• If $||\phi_t^{\gamma}(z)||_H > M_3$ then $||v||_H > M_2$ and $||\phi_t^{\gamma}(z)||_H \leq ||v||_HM_3/M_2$ (See 42 in Appendix B). So,

since $M_3/M_2 > 1$.

Remark 6. Lemma 4.14 ensures the existence of a bounding function for the PDMP. To broaden the class of cost functions considered, we could just assume the existence of a bounding for the PDMP in Theorem 4.3 and then, the assumption on $c$ and $g$ should just be the continuity.

4.3.   Existence of an optimal ordinary strategy

Ordinary strategies are of crucial importance because they are the ones that the controller can implement in practice. Here we give convexity assumptions that ensure the existence of an ordinary optimal control strategy for the PDMP.

(A) (a) For all $d\in D$, the function $f_d : (y,u)\in H\times U \rightarrow E$ is linear in the control variable $u$.

(b) For all $d\in D$, $y\in H$ and $E\in \mathcal{B}(D)$, the function $\lambda_d(y,\cdot) : U \rightarrow \mathbb{R}_+$ is concave and the function $\lambda_d(y,\cdot)\mathcal{Q}(E|y,d,\cdot) : u\in H\times U \rightarrow \mathbb{R}_+$ is convex.

(c) The cost function $c:(y,u)\in E\times U \rightarrow \mathbb{R}_+$ is convex in the control variable $u$.

Theorem 4.15. Suppose that assumptions (H($L$)), (H($f$)), (H($\lambda$)), (H($\mathcal{Q}$)), (H(c)) and (A) are satisfied. If we consider $\mu^*\in \mathcal{A}^\mathcal{R}$ an optimal relaxed strategy for the PDMP, then the ordinary strategy $\bar{\mu}_t := \int_Z u\mu^*_t(du)\in\mathcal{A}$ is optimal, i.e. $V_{\bar{\mu}}(z) = \tilde{V}_{\mu^*}(z) = V(z), \forall z\in \Upsilon.$

Proof. This result is based on the fact that for all $(z,\gamma)\in \Upsilon\times \mathcal{R}, (Lw)(z,\gamma) \geq (Lw)(z,\bar{\gamma})$, with $\bar{\gamma} = \int_Z u\gamma(\mathrm{d}u).$ Indeed, the fact that the function $f_d$ is linear in the control variable implies that for all $(t,z,\gamma)\in [0,T]\times\Upsilon\times \mathcal{R},\phi_t^{\gamma}(z) =\phi_t^{\bar{\gamma}}(z)$. The convexity assumptions (A) give the following inequalities

for all $(s,z,\gamma,E)\in [0,T]\times\Upsilon\times \mathcal{R}\times \mathcal{B}(D)$, so that in particular $\chi_t^{\gamma}(z)\geq \chi_t^{\bar{\gamma}}(z)$. We can now denote for all $(z,\gamma)\in \Upsilon\times \mathcal{R}$ and $w: \Upsilon\rightarrow \mathbb{R}_+$,

Furthermore,

so that

4.4.   An elementary example

Here we treat an elementary example that satisfies the assumptions made in the previous two sections.

Let $V = H_0^1([0,1])$, $H=L^2([0,1])$, $D=\{-1,1\}$, $U=[-1,1]$. $V$ is a Hilbert space with inner product

We consider the following PDE for the deterministic evolution between jumps

with Dirichlet boundary conditions. We define the jump rate function for $(v,u)\in H\times U$ by

and the transition measure by $\mathcal{Q}(\{-1\}|v,1,u) = 1$, and $\mathcal{Q}(\{1\}|v,-1,u) = 1.$

Finally, we consider a quadratic cost function $c(v,u) = K||V_{\mathrm{ref}}-v||^2 + u^2$, where $V_{\mathrm{ref}}\in D(\Delta)$ is a reference signal that we want to approach.

Lemma 4.16. The PDMP defined above admits the continuous bounding function

Furthermore, the value function of the optimal control problem is continuous and there exists an optimal ordinary control strategy.

Proof. The proof consists in verifying that all assumptions of Theorem 4.15 are satisfied. Assumptions (H($\mathcal{Q}$)), (H(c)) and (A) are straightforward. For $(v,u)\in H\times U$, $1/2\leq\lambda_1(v,u) \leq 2$ and $e^{-1} \leq \lambda_1(v,u) \leq 2$. The continuity in the variable $u$ is straightforward and the locally Lipschitz continuity comes from the fact that the functions $v\to 1/({e^{-||v||^2}+1})$, and $v \to e^{-\beta(v)}$, with $\beta(v):=1/(||v||^2+1)$, are Fréchet differentiable with derivatives $v\to 2(v,\cdot)_H/(e^{-||v||^2}+1)^2$, and $v \to 2(v,\cdot)_H\beta^2(v)e^{-\beta(v)}.$

$-\Delta v : w\in V \rightarrow \int_0^1 v'(x)w'(x)\mathrm{d}x$ so that $-\Delta : V\to V^*$ is linear. Let $(v,w)\in V^2$.

and so $||-\Delta v||_{V^*} \leq ||v||_V$. $\langle -\Delta v , v \rangle = \int_0^1 (v'(x))^2\mathrm{d}x \geq C'||v||^2_V$, for some constant $C'>0$, by the Poincaré inequality.

Now, define for $k\in\mathbb{N}^*$, $f_k(\cdot) := \sqrt{2}\sin(k\pi\cdot)$, a Hilbert base of $H$.

On $H$, $S(t)$ is the diagonal operator

For $t>0$, $S(t)$ is a contracting Hilbert-Schmidt operator.

For $(v,w,u)\in H^2\times U$, $f_d(v,u) = (d+u)v$ and

This means that for every $z=(v,d,h)\in\Upsilon$, $\gamma\in\mathcal{R}([0,T],U)$ and $t\in [0,T]$, $||\phi_t^{\gamma}(z) ||_H \leq e^{2T}||v||_H$.

5.   Application to the model in Optogenetics

5.1.   Proof of Theorem 1.2

We begin this section by making some comments on Definition 1.1.

In (1), $C_m>0$ is the membrane capacitance and $V_-$ and $V_+$ are constants defined by $V_-:=\min\{V_{Na},V_K,V_L, V_{ChR2}\}$ and $V_+:=\max\{V_{Na},V_K,V_L,V_{ChR2}\}$. They represent the physiological domain of our process. In (2), the constants $g_x > 0$ are the normalized conductances of the channels of type $x$ and $V_x\in\mathbb{R}$ are the driving potentials of the channels. The constant $\rho>0$ is the relative conductance between the open states of the $ChR2$. For a matter of coherence with the theoretical framework presented in the paper, we will prove Theorem 1.2 for the mollification of the model that we define now. This model is very close to the one of Definition 1.1. It is obtained by replacing the Dirac masses $\delta_z$ by their mollifications $\xi^N_z$ that are defined as follows. Let $\varphi$ be the function defined on $\mathbb{R}$ by

with

such that $\int_{\mathbb{R}}\varphi(x)\mathrm{d}x = 1$.

Now, let $U_N:=\left(\frac{1}{2N},1-\frac{1}{2N}\right)$ and $\varphi_N(x) := 2N \varphi(2Nx)$ for $x\in\mathbb{R}$. For $z\in I_N$, the $N^{\mathrm{th}}$ mollified Dirac mass $\xi^N_z$ at $z$ is defined for $x\in [0,1]$ by

For all $z\in I_N, \xi^N_z \in C^{\infty}([0,1])$ and $\xi^N_z \rightarrow \delta_z$ almost everywhere in $[0,1]$ as $N\rightarrow +\infty$, so that $(\xi^N_z,\phi)_H \to \phi(z)$, as $N\to \infty$ for every $\phi\in C(I,\mathbb{R})$. The expressions $v(i/N)$ in Definition 1.1 are also replaced by $(\xi^N_{i/N},v)_H$. The decision to use the mollified Dirac mass over the Dirac mass can be motivated by two main reasons. First of all, as mentioned in [10], the concentration of ions is homogeneous in a spatially extended domain around an open channel so the current is modeled as being present not only at the point of a channel, but in a neighborhood of it. Second, the smooth mollified Dirac mass leads to smooth solutions of the PDE and we need at least continuity of the flow. Nevertheless, the results of Theorem 1.2 remain valid with the Dirac masses and we refer the reader to Section 5.2.

The following lemma is a direct consequence of [10,Proposition 7] and will be very important for the model to fall within the theoretical framework of the previous sections.

Lemma 5.1. For every $y_0\in V$ with $y_0(x) \in [V_-,V_+]$ for all $x\in I$, the solution $y$ of (1) is such that for $t\in [0,T]$,

Physiologically speaking, we are only interested in the domain $[V_-,V_+]$. Since Lemma 5.1 shows that this domain is invariant for the controlled PDMP, we can modify the characteristics of the PDMP outside the domain $[V_-,V_+]$ without changing its dynamics. We will do so for the rate functions $\sigma_{x,y}$ of Table 1. From now on, consider a compact set $K$ containing the closed ball of $H$, centered in zero and with radius $\max(V_-,V_+)$. We will rewrite $\sigma_{x,y}$ the quantities modified outside $K$ such that they all become bounded functions. This modification will enable assumption (H($\lambda$))1. to be verified.

The next lemma shows that the stochastic controlled infinite-dimensional Hodgkin-Huxley-ChR2 model defines a controlled infinite-dimensional PDMP as defined in Definition 2.3 and that Theorem 2.5 applies.

Lemma 5.2. For $N\in N^*$, the $N^{\mathrm{th}}$ stochastic controlled infinite-dimensional Hodgkin-Huxley-ChR2 model satisfies assumptions (H($\lambda$)), (H($\mathcal{Q}$)), (H($L$)) and (H($f$)). Moreover, for any control strategy $\alpha\in\mathcal{A}$, the membrane potential $v^\alpha$ satisfies

Proof. The local Lipschitz continuity of $\lambda_d$ from $H\times Z$ in $\mathbb{R}^+$ comes from the local Lipschitz continuity of all the functions $\sigma_{x,y}$ of Table 2.5 and the inequality $|(\xi^N_z,v)_H-(\xi^N_z,w)_H|\leq 2N||v-w||_H$. By Lemma 5.1, the modified jump rates are bounded. Since they are positive, they are bounded away from zero, and then, Assumption (H($\lambda$)) is satisfied. Assumption (H($\mathcal{Q}$)) is also easily satisfied. We showed in Section 4.4that (H($L$)) is satisfied. As for $f_d$, the function does not depend on the control variable and is continuous from $H$ to $H$. For $d\in D$ and $(y_1,y_2)\in H^2$,

We then get

Finally, since the continuous component $v_t^{\alpha}$ of the PDMP does not jump, the bounds are a direct consequence of Lemma 5.1.

Proof of Theorem 1.2. In Lemma 5.2 we already showed that assumptions (H($\lambda$)), (H($\mathcal{Q}$)), (H($L$)) and (H($f$)) are satisfied. The cost function $c$ is convex in the control variable and norm quadratic on $H\times Z$. The flow does not depend on the control. The rate function $\lambda$ is linear in the control. the function $\lambda \mathcal{Q}$ is also linear in the control. We conclude that all the assumptions of Theorem 4.3 are satisfied and that an optimal ordinary strategy can be retrieved.

We end this section with an important remark that significantly extends the scope of this example. Up to now, we only considered stationary reference signals but nonautonomous ones can be studied as well, as long as they feature some properties. Indeed, it is only a matter of incorporating the signal reference $V_{\mathrm{ref}}\in C([0,T],H)$ in the process by adding a variable to the PDMP. Instead of considering $H$ as the initial state space for the continuous component, we consider $\tilde{H} := H\times H$.

This way, the part on the control problem is not impacted at all and we consider the continuous cost function $\tilde{c}$ defined for $(v,\bar{v},u)\in \tilde{H}\times U$ by

the result and proof of lemma 1.2 remaining unchanged with the continuous bounding function defined for $v\in H$ by

In the next section, we present some variants of the model and the corresponding results in terms of optimal control.

5.2.   Variants of the model and perspectives

We begin this section by giving arguments showing that the results of Theorem 4.3 remain valid for the model of Definition 1.1, which does not exactly fits into our theoretical framework. Then, the variations we present concern the model of ChR2, the addition of other light-sensitive ionic channels, the way the control acts on the three local characteristics and the control space. The optimal control problem itself will remain unchanged. First of all, let us mention that since the model of Definition 1.1 satisfies the convexity conditions (A), the theoretical part on relaxed controls is not necessary for this model. Nevertheless, the model of ChR2 presented on Figure 1 is only one among several others, some of which do not enjoy a linear, or even concave, rate function $\lambda$. For those models, that we present next, assumption (A) fails and the relaxed controls are essential.

We will not present them here, but the previous results for the Hodgkin-Huxley model remain straightforwardly unchanged for other neuron models such as the FitzHugh-Nagumo model or the Morris-Lecar model.

Optimal control for the original model.

In the original model, the function $f_d$ is defined from $V$ to $V^*$. Nevertheless, the semigroup of the Laplacian regularizes Dirac masses (see [4,Lemma 3.1]) and the uniform bound in Theorem 2.5 is in fact valid in $V$, the solution belonging to $C([0,T],V)$. This is all we need since the control does not act on the PDE. This is why the domain of our process is $V\times D_N$ and not just $H\times D_N$, and all computations of the proofs of the previous sections can be done in the Hilbert space $V$. From this consideration, and using the continuous embedding of $H_0^1(I)$ in $C_0(I)$ we can justify the local Lipschitz continuity of $\lambda_d$ from $V\times Z$ in $\mathbb{R}^+$. Indeed, it comes from the local Lipschitz continuity of all functions $\sigma_{x,y}$ of Table 1 and from the inequality

Finally, [10,Proposition 5] states that the bounds of Lemma 5.2 remain valid with Dirac masses.

Modifications of the ChR2 model.

We already mentioned the paper of Nikolic and al. [33] in which a three states model is presented. It is a somehow simpler model that the four states model of Figure 1 but it gives good qualitative results on the photocurrents produced by the ChR2. In first approximation the model can be considered to depend linearly in the control as seen on Figure 2.

This model features one open state $o$ and two closed states, one light-adapted $d$ and one dark-adapted $c$. This model would lead to the same type of model as in the previous Section. In fact, the time constants $1/K_d$ and $1/K_r$ are also light dependent with a dependence in $\log(u)$. The corresponding model is represented on Figure 3 below

Some mathematical comments are needed here. On Figure 3, the control $u$ represents the light intensity and $c_1$, $c_2$, $K_r$ and $\tau_d$ are positive constants. This model of ChR2 is experimentally accurate for intensities between $10^8$ and $10^{10}$ $\mu\mathrm{m}^2\cdot\mathrm{s}^{-1}$ approximately. We would then consider $U:=[0,u_{max}]$ with $u_{max}\simeq 10^{10}$ $\mu\mathrm{m}^2\cdot\mathrm{s}^{-1}$. Furthermore,

The first limit is not physical since rate jumps between states are positive numbers. The second limit is not physical either because it would mean that, in the dark, the proteins are trapped in the open state $o$, which is not the case. In the dark, when $u=0$, the jump rates corresponding to the transition $o\to d$ and $d\to c$ are positive constants. For this reason, the functions $\sigma_{o,d}$ and $\sigma_{d,c}$ should be smooth functions such that they are equal to the rates of Figure 3 for large intensities, but still with $\tau_d-\log(u) >0$, and converge to $K_d^{\mathrm{dark}}>0$ and $K_r^{\mathrm{dark}}>0$ respectively, when $u$ goes to $0$. The resulting rate function $\lambda$ is not concave and thus does not satisfy assumption (A) anymore. We can only affirm the existence of optimal relaxed strategies.

The four states model of Figure 1 is also an approximation of a more accurate model that we represent on Figure 4 below. The transition rates can depend on either the membrane potential $v$ or the irradiance $u$, which is the control variable. The details of the model and the numerical constants can be found in [44]. Note that the model of Figure 4 is already an approximation of the model in [44] because the full model in [44] would not lead to a Markovian behavior for the ChR2 (the transition rates would depend on the time elapsed since the light was switched on).

with $K_{d1}^{(1)}$, $K_{d1}^{(2)}$, $e_{12d}$, $e_{21d}$, $c$, $c_1$ and $c_2$ positive constants. As for the model of Figure 3, the mathematical definition of the function $\sigma_{o_1,c_1}$ should be such that it is a positive smooth function and equals $K_{d1}(v)$ in some subset of the physiological domain $[V_-,V_+]$. The resulting rate function $\lambda$ will be concave but the function $\lambda\mathcal{Q}$ will not be convex (it will be concave as well). Hence, Assumption (A) is not satisfied.

Addition of other light-sensitive ion channels.

Channelrhodopsin-2 has a promoting role in eliciting action potentials. There also exists a chlorine pump, called Halorhodopsin (NpHR), that has an inhibitory action. NpHR can be used along with ChR2 to obtain a control in both directions. Its modelisation as a multistate model was considered in [34]. The transition rates between the different states have the same shape that the ones of the ChR2 and the same simplifications are possible. This new light-sensitive channel can be easily incorporated in our stochastic model and we can state existence of optimal relaxed and/or ordinary control strategies depending on the level of complexity of the NpHR model we consider. It is here important to remark that since the two ionic channels do not react to the same wavelength of the light, the resulting control variable would be two-dimensional with values in $[0,u_{max}]^2$. This would not change the qualitative results of the previous sections.

Modification of the way the control acts on the local characteristics.

Up to now, the control acts only on the rate function, and also on the measure transition via its special definition from the rate function. Nevertheless, we can present here a modification of the model where the control acts linearly on the PDE. This modification amounts to considering that the control variable is directly the gating variable of the ChR2. Indeed, we show in [38] that the optimal control of the deterministic counterpart of the stochastic Hodgkin-Huxley-ChR2 model, in finite dimension and with the three states ChR2 model of Figure 2, is closely linked to the optimal control of

where the control variable is the former gating variable $o$. Now the stochastic counterpart of the last model is such that the function $f_d$ is now linear in the control and the rate function $\lambda$ and the transition measure function $\mathcal{Q}$ do not depend on the control any more. Finally, by adding NpHR channels to this model, we would obtain a fully controlled infinite-dimensional PDMP in the sense that the control would then act on the three local characteristics of the PDMP. Depending on the model of NpHR chosen, we would obtain relaxed or ordinary optimal control strategy.

Modification of the control space.

In all models discussed previously, the control has no spatial dependence. Any light-stimulation device, such as a laser, has a spatial resolution and it is possible that we do not want or cannot stimulate the entire axon. For this reason, spatial dependence of the control should be considered. Now, as long as the control space remains a compact Polish space, spatial dependence of the control could be considered. We propose here a control space defined as a subspace of the Skorohod space $\mathbb{D}$, constituted of the càdlàg functions from $[0,1]$ to $\mathbb{R}$. This control space represents the aggregation of multiple laser beams that can be switched on and off. Suppose that each of these beams produces on the axon a disc of light of diameter $r > 0$ that we call spatial resolution of the light. For an axon represented by the segment $[0,1]$, $r$ is exactly the spatial domain illuminated. We consider now two possibilities for the control space. Suppose first that the spatial resolution is fixed and define $p:=\lfloor\frac{1}{r}\rfloor$ and

Lemma 5.3. $\mathcal{U}$ is a compact subset of $\mathbb{D}$.

Proof. We tackle this proof by remarking that $\mathcal{U}$ is in bijection with the finite dimensional compact space $[0,u_{max}]^p$.

In this case, the introduction of the space $\mathbb{D}$ was quite artificial since the control space remains finite-dimensional. Nevertheless, the Skorohod space will be very useful for the other control space. Suppose now that the spatial resolution of the laser can evolve in $[r_{min},r_{max}]$ with $r_{min},r_{max} > 0$. Let $p\in\mathbb{N}^*$ the number of lasers used and define

Now $\tilde{\mathcal{U}}$ is infinite-dimensional and the Skorohod space allows us to use the characterization of compact subsets of $\mathbb{D}$.

Lemma 5.4. $\tilde{\mathcal{U}}$ is a compact subset of $\mathbb{D}$.

Proof. For this proof, we need to introduce some notation and a critera of compactness in $\mathbb{D}$. A complete treatment of the space $\mathbb{D}$ can be found in [7].

Let $u\in \mathbb{D}$ and $\{x_i\}_{0\leq i\leq n}$ a subdivision of $[0,1]$, $n\in \mathbb{N}^*$. We define, for $i\in\{0,..,n-1\}$,

and for $\delta >0$,

the infimum being taken on all the subdivisions $\{x_i\}_{0\leq i\leq n}$ of $[0,1]$ such that $x_{i+1}-x_i > \delta$ for all $i\in\{0,..,n-1\}$. Now since $\tilde{\mathcal{U}}$ is obviously bounded in $\mathbb{D}$, from [7,Theorem 14.3], we need to show that

Let $\delta > 0$ with $\delta < r_{min}$ and $u\in \tilde{\mathcal{U}}$. There exists as subdivision $\{x_i\}_{0\leq i\leq p}$ of $[0,1]$ such that for every $i\in \{0,..,p-1\}$, $u$ is constant on $[x_i,x_{i+1})$ and $x_{i+1}-x_i > \delta$. Thus $w'_u(\delta)=0$ which ends the proof.

With either $\mathcal{U}$ or $\tilde{\mathcal{U}}$ as the control space, the stochastic controlled infinite-dimensional Hodgkin-Huxley-ChR2 model admits an optimal ordinary control strategy.

Perspectives.

Theorem 4.3 proves the existence of optimal controls for a wide class of infinite-dimensional PDMPs and Theorem 4.15 gives sufficient conditions to retrieve ordinary optimal controls. Nevertheless, these theorems do not indicate how to compute optimal controls or approximations of optimal controls. Thus, it would be very interesting, in a further work, to implement numerical methods in order to compute at least approximation of optimal controls for the control problems defined in this paper. One efficient way to address numerical optimal control problems for PDMPs is to use quantization methods that consist in replacing the state and control spaces by discrete spaces and work with approximations of the processes on these discrete spaces ([29], [35], [20]).

Appendix A.   Construction of $X^\alpha$ by iteration

Let $\alpha\in\mathcal{A}$ and let $x:=(v,d,\tau,h,\nu)\in\Xi^{\alpha}$ with $z:=(\nu,d,h)\in\Upsilon$. The existence of the probability $\mathbb{P}_x^{\alpha}$ below is the object of the next section where Theorem 2.5 is proved.

• Let $T_1$ be the time of the first jump of $(X^{\alpha}_t)$. With the notations of Proposition 1, the law of $T_1$ is defined by its survival function given for all $t>0$ by

• For $t<T_1$, $X_t^{\alpha}$ solves (10) starting from $x$ namely $(v_t, d_t , \tau_t,h_t, \nu_t)= ( \phi_t^{\alpha}(x), d, \tau+t,h,\nu).$

•When a jump occurs at time $T_1$, conditionally to $T_1$, $X_{T_1}^{\alpha}$ is a random variable distributed according to a measure $\hat{\mathcal{Q}}$ on $(\Xi,\mathcal{B}(\Xi))$, itself defined by a measure $\mathcal{Q}$ on $(D,\mathcal{B}(D))$. The target state $d_1$ of the discrete variable is a random variable distributed according to the measure

$\mathcal{Q}(\cdot|\phi_{T_1}^{\alpha}(x),d_{T_1^-},\alpha(\nu_{T_1^-},d_{T_1^-},h_{T_1^-})(\tau_{T_1^-}))$ such that for all $B\in\mathcal{B}(D)$,

where we use the notation $\alpha(T_1^-)=\alpha(d_{T_1^-},\tau_{T_1^-},h_{T_1^-},\nu_{T_1^-})$. This equality means that the variables $v$ and $\nu$ do not jump at time $T_1$, and the variables $\tau$ and $h$ jump in a deterministic way to $\{0\}$ and $\{h+\tau_{T_1^-}\}$ respectively.

• The construction iterates after time $T_1$ with the new starting point $(v_{T_1},d_{T_1},0, h+T_1,v_{T_1}$).

Formally the expressions of the jump rate and the transition measures on $\Xi$ are

with $F\times B \times E \times G \times J\in\mathcal{B}(\Xi)$, $u\in U$ and $x:=(v,d,\tau,h,\nu)\in\Xi$.

Appendix B.   Proof of Theorem 2.5

There are two filtered spaces on which we can define the enlarged process $(X^\alpha)$ of Definition 2.3. They are linked by the one-to-one correspondence between the PDMP $(X^\alpha)$ and the included jump process $(Z^\alpha)$ that we define now. We then introduce both spaces since each one of them is relevant to prove useful properties.

Given the sample path $(X_s^{\alpha},s\leq T)$ such that $X_0^{\alpha}:=(v,d,\tau,h,\nu)\in\Xi^{\alpha}$, the jump times $T_k$ of $X^{\alpha}$ can be retrieved by the formula

Moreover we can associate to $X^{\alpha}$ a pure jump process $(Z_t^{\alpha})_{t\geq 0}$ taking values in $\Upsilon$ in a one-to-one correspondence as follows,

Conversely, given the sample path of $Z^{\alpha}$ on $[0,T]$ starting from $Z_0^{\alpha}=(\nu_0^{Z},d_0^{Z},T_0^{Z})$, we can recover the path of $X^{\alpha}$ on $[0,T]$. Denote $Z_t^{\alpha}$ as $(\nu_t^{Z},d_t^{Z},T_t^{Z})$ and define $T_0: = T_0^Z$ and $T_k := \inf\{t>T_{k-1}|T_t^Z\neq T_{t^-}^Z\}$. Then

Let us note that $T_{T_k}^Z = T_k$ for all $k\in\mathbb{N}$, and that by construction of the PDMP all jumps are detected since $\mathbb{P}^{\alpha}[T_{k+1}=T_k]=0$. When no confusion is possible, we write, for $\alpha \in\mathcal{A}$ and $n\in\mathbb{N}$, $Z_n=Z^{\alpha}_{T_n}$.

Part 1. The canonical space of jump processes with values in $\Upsilon$. The following construction is very classical, see for instance Davis [19] Appendix A1. We adapt it here to our peculiar process and to the framework of control. Remember that a jump process is defined by a sequence of inter-arrival times and jump locations

where $\gamma_0\in \Upsilon$ is the initial position, and for $i\in\mathbb{N}^*$, $s_i$ is the time elapsed between the $(i-1)^{\mathrm{th}}$ and the $i^{\mathrm{th}}$ jump while $\gamma_i$ is the location right after the $i^{\mathrm{th}}$ jump. The jump times $(t_i)_{i\in\mathbb{N}}$ are deduced from the sequence $(s_i)_{i\in\mathbb{N}^*}$ by $t_0=0$ and $t_i=t_{i-1}+s_i$ for $i\in\mathbb{N}^*$ and the jump process $(J_t)_{t\geq0}$ is given by $J_t := \gamma_i$ for $t\in[t_i,t_{i+1})$ and $J_t=\Delta$ for $t\geq t_{\infty}:=\lim_{i\rightarrow \infty} t_i$, $\Delta$ being an extra state, called cemetery.

Accordingly we introduce $Y^{\Upsilon} := (\mathbb{R}_+\times\Upsilon)\cup\{(\mathbb{R}_+\cup\infty,\Delta)\}$. Let $(Y_i^{\Upsilon})_{i\in\mathbb{N}^*}$ be a sequence of copies of the space $Y^{\Upsilon}$. We define $\Omega^{\Upsilon} := \Upsilon\times\Pi_{i=1}^{\infty} Y_i^{\Upsilon}$ the canonical space of jump processes with values in $\Upsilon$, endowed with its Borel $\sigma$-algebra $\mathcal{F}^{\Upsilon}$ and the coordinate mappings on $\Omega^{\Upsilon}$ as follows

We also introduce $\, \omega_i : \Omega^{\Upsilon}\rightarrow \Omega^{\Upsilon}_i$ for $i\in\mathbb{N}^*$, defined by

for $\omega\in\Omega^{\Upsilon}$. Now for $\omega\in\Omega^{\Upsilon}$ and $i\in\mathbb{N}^*$, let

and the sample path $(x_t(\omega))_{t\geq0}$ be defined by

A relevant filtration for our problem is the natural filtration of the coordinate process $(x_t)_{t\geq 0}$ on $\Omega^{\Upsilon}$

for all $t\in\mathbb{R}_+$. For given starting point $\gamma_0\in\Upsilon$ and control strategy $\alpha\in\mathcal{A}$, a controlled probability measure, denoted $\mathbb{P}^{\alpha}_{\gamma_0}$, is defined on $\Omega^{\Upsilon}$ by the specification of a family of controlled conditional distribution functions as follows: $\mu_1$ is a controlled probability measure on $(Y^{\Upsilon},\mathcal{B}(Y^{\Upsilon}))$ or equivalently a measurable mapping from $\mathcal{U}_{ad}([0,T];U)$ to the set of probability measures on $(Y^{\Upsilon},\mathcal{B}(Y^{\Upsilon}))$, such that for all $\alpha\in\mathcal{A}$,

For $i\in\mathbb{N}\setminus\{0,1\}$, $\mu_i : \Omega^{\Upsilon}_i\times \mathcal{U}_{ad}([0,T];U)\times \mathcal{B}(Y^{\Upsilon}) \rightarrow [0,1]$ are controlled transition measures satisfying:

1. $\mu_i(\cdot;\Sigma)$ is measurable for each $\Sigma \in \mathcal{B}(Y^{\Upsilon})$,

2. $\mu_i(\omega_{i-1}(\omega),\alpha(\Gamma_{i-1}(\omega));\cdot)$ is a probability measure for every $\omega\in\Omega^{\Upsilon}$ and $\alpha\in\mathcal{A}$,

3. $\mu_i(\omega_{i-1}(\omega),\alpha(\Gamma_{i-1}(\omega));(\{0\}\times\Upsilon)\cup(\mathbb{R}_+\times\{\Gamma_{i-1}(\omega)\}))=0$ for every $\omega\in\Omega^{\Upsilon}$ and $\alpha\in\mathcal{A}$,

4. $\mu_i(\omega_{i-1}(\omega),\alpha(\Gamma_{i-1}(\omega));\{(\infty,\Delta)\})=1$ if $S_k(\omega) = \infty$ or $\Gamma_k(\omega) = \Delta$ for some $k\in \{1,\dots,i-1\}$, for every $\alpha\in\mathcal{A}$.

We need to extend the definition of $\alpha\in\mathcal{A}$ to the state $(\infty,\Delta)$ by setting $\alpha(\Delta) := u_\Delta$ where $u_\Delta$ is itself an isolated cemetery state and $\alpha$ takes in fact values in $\mathcal{U}_{ad}([0,T];U\cup\{u_\Delta\})$.

Now for a given control strategy $\alpha\in\mathcal{A}$, $\mathbb{P}_{\gamma_0}^{\alpha}$ is the unique probability measure on $(\Omega^{\Upsilon},\mathcal{T}^{\Upsilon})$ such that for each $i\in\mathbb{N}^*$ and bounded function $f$ on $\Omega^{\Upsilon}_i$

with $\alpha$ depending only on the variable in $\Upsilon$ when writing "$\alpha(y_{i-1})$", $y_{i-1}=(s_{i-1},\gamma_{i-1})$. Let's now denote by $\mathcal{F}^{\Upsilon}_{\gamma,\alpha}$ and $(\mathcal{F}_t^{\Upsilon,\gamma,\alpha})_{t\geq0}$ the completed $\sigma$-fields of $\mathcal{F}^{\Upsilon}$ and $(\mathcal{F}_t^{\Upsilon})_{t\geq0}$ with all the $\mathbb{P}^{\alpha}_{\gamma}$-null sets of $\mathcal{F}^{\Upsilon}$. We then rename the intersection of these $\sigma$-fields redefine $\mathcal{F}^{\Upsilon}$ and $(\mathcal{F}_t^{\Upsilon})_{t\geq0}$ so that we have

Then ($\Omega^{\Upsilon},\mathcal{F}^{\Upsilon},(\mathcal{F}_t^{\Upsilon})_{t\geq0}$) is the natural filtered space of controlled jump processes.

Part 2. The canonical space of cdlg functions with values in $\Xi$. Let $\Omega_{\Xi}$ be the set of right-continuous functions with left limits (cdlg functions), defined on $\mathbb{R}_+$ with values in $\Xi$. Analogously to what we have done in Part 1, we can construct a filtered space ($\Omega^{\Xi},\mathcal{F}^{\Xi},(\mathcal{F}_t^{\Xi})_{t\geq0}$) with coordinate process $(x_t^{\Xi})_{t\geq0}$ and a probability $\mathbb{P}^{\alpha}$ on ($\Omega^{\Xi},\mathcal{F}^{\Xi}$) for every control strategy $\alpha\in\mathcal{A}$ such that the infinite-dimensional PDMP is a $\mathbb{P}^{\alpha}$-strong Markov process. For $(t,y)\in\mathbb{R}_+\times\Omega_{\Xi}, x_t^{\Xi}(y)=y(t)$.

We start with the definition of $\mathcal{F}_t^{\Xi,0} := \sigma\{x_s^{\Xi} | s\leq t\}$ for $t\in\mathbb{R}_+$ and $\mathcal{F}^{\Xi,0}:=\vee_{t\geq 0} \mathcal{F}_t^{\Xi,0}$. In Davis [19] p 59, the construction of the PDMP is conducted on the Hilbert cube, the space of sequences of independent and uniformly distributed random variables in $[0,1]$. In the case of controlled PDMP, the survival function $F(t,x)$ in [19] is replaced by the extension to $\xi^{\alpha}$ of $\chi^{\alpha}$ defined in Definition 2.3 and the construction depends on the chosen control. This extension is defined for $x:=(v,d,\tau,h,\nu)\in\Xi^{\alpha}$ by

such that for $z:=(v,d,h)\in\Upsilon$, $\chi_t^{\alpha}(z) = \chi_t^{\alpha}(v,d,0,h,v)$.

This procedure thus provides for each control $\alpha\in\mathcal{A}$ and starting point $x\in\Xi^{\alpha}$ a measurable mapping $\psi_x^{\alpha}$ from the Hilbert cube to $\Omega_{\Xi}$. Let $\mathbb{P}_x^{\alpha}:=\mathbb{P}\left[(\psi_x^{\alpha})^{-1}\right]$ denote the image measure of the Hilbert cube probability $\mathbb{P}$ under $\psi_x^{\alpha}$. Now for $x\in\Xi^\alpha$, let $\mathcal{F}_t^{x,\alpha}$ be the completion of $\mathcal{F}_t^{\Xi,0}$ with all $\mathbb{P}_x^{\alpha}$-null sets of $\mathcal{F}^{\Xi,0}$, and define

The right-continuity of $(\mathcal{F}^{\Xi}_t)_{t\geq0}$ follows from the right-continuity of $(\mathcal{F}_t^{\Upsilon})_{t\geq 0}$ and the one-to-one correspondence. The right-continuity of $(\mathcal{F}_t^{\Upsilon})_{t\geq 0}$ is a classical result on right-constant processes. For these reasons, we lose the superscripts $\Xi$ and $\Upsilon$ consider the natural filtration $(\mathcal{F}_t)_{t\geq 0}$ in the sequel.

Now that we have a filtered probability space that satisfies the usual conditions, let us show that the simple Markov property holds for $(X_t^\alpha)$. Let $\alpha\in\mathcal{A}$ be a control strategy, $s>0$ and $k\in\mathbb{N^*}$. By construction of the process $(X_t^{\alpha})_{t\geq0}$,

Now for $x\in\Xi^\alpha$, $(t,s)\in\mathbb{R}_+^2$ and $k\in\mathbb{N^*}$,

The equality (*) is the classical formula for jump processes (see Jacod [32]). On the other hand,

because $X_t^{\alpha} = \Big(\phi_{t-T_k}^{\alpha}(X_{T_k}^{\alpha}),d_{T_k},t-T_k,h_{T_k},\nu_{T_k}\Big)$ and by the flow property $\phi_u^{\alpha}(X_{t}^{\alpha})=\phi_{u+t-T_k}^{\alpha}(X_{T_k}^{\alpha})$ on $\mathbf{1}_{\{T_k\leq t<T_{k+1}\}}$.

Thus we showed that for all $x\in\Xi^{\alpha}$, $(t,s)\in\mathbb{R}_+^2$ and $k\in\mathbb{N^*}$,

Now if we write $T_t^{\alpha} := \inf\{s>t : X_s^{\alpha} \neq X_{s^-}^{\alpha}\}$ the next jump time of the process after $t$, we get

which means that, conditionally to $\mathcal{F}_t$, the next jump has the same distribution as the first jump of the process started at $X_t^{\alpha}$. Since the location of the jump only depends on the position at the jump time, and not before, equality (40) is just what we need to prove our process verifies the simple Markov property.

To extend the proof to the strong Markov property, the application of Theorem (25.5) (Davis [19]) on the characterization of jump process stopping times on Borel spaces is straightforward.

From the results of [10], there is no difficulty in finding the expression of the extended generator $\mathcal{G}^{\alpha}$ and its domain:

• Let $\alpha\in\mathcal{A}$. The domain $D(\mathcal{G}^{\alpha})$ of $\mathcal{G}^{\alpha}$ is the set of all measurable $f: \Xi\rightarrow \mathbb{R}$ such that $t\mapsto f(\phi_t^{\alpha}(x),d,\tau+t,h,\nu)$ (resp. $(v_0,d_0,\tau_0,h_0,\nu_0,t,\omega)\mapsto f(v_0,d_0,\tau_0,h_0,\nu_0)-f(v(t^-,\omega),d(t^-,\omega), \tau(t^-,\omega), h(t^-,\omega),\nu(t^-,\omega))$) is absolutely continuous on $\mathbb{R}_+$ for all $x=(v,d,\tau,h,\nu)\in \Xi^{\alpha}$ (resp. a valid integrand for the associated random jump measure).

• Let $f$ be continuously differentiable w.r.t. $v\in V$ and $\tau\in\mathbb{R}_+$. Define $h_v$ as the unique element of $V^*$ such that

where $\frac{\mathrm{d}f}{\mathrm{d}v} [v,d,\tau,h,\nu]$ denotes the Frchet-derivative of $f$ w.r.t $v\in E$ evaluated at $(v,d,\tau,h,\nu)$. If $h_v(v,d,\tau,h,\nu)\in V^*$ whenever $v\in V$ and is bounded in $V$ for bounded arguments then for almost every $t\in [0,T]$,

The bound on the continuous component of the PDMP comes from the following estimation. Let $\alpha\in\mathcal{A}$ and $x:=(v,d,\tau,h,\nu)\in\Xi^{\alpha}$ and denote by $v^{\alpha}$ the first component of $X^{\alpha}$. Then for $t\in[0,T]$,

by Gronwall's inequality.

Appendix C.   Proof of Lemma 4.13

Part 1. Let's first look at the case when $w$ is bounded by a constant $w_{\infty}$ and define for $(z,\gamma) \in \Upsilon\times \mathcal{R}$

Now take $(z,\gamma) \in \Upsilon\times \mathcal{R}$ and suppose $(z_n,\gamma_n)\rightarrow (z,\gamma)$. Let's write $z=(v,d,h)$ and $z_n = (v_n,d_n,h_n)$ for $n\in\mathbb{N}$. For $s\in[0,T]$, let $w_n(s,u):=w(\phi_s^{\gamma_n}(z_n),d_n,h_n+s,u)$ and $w(s,u):=w(\phi_s^{\gamma}(z),d,h+s,u)$. Let also $a_n = \min(T-h,T-h_n)$ and $b_n = \max(T-h,T-h_n)$. Then

The first term on the right-hand side converges to zero for $n\rightarrow \infty$ since the integrand is bounded.

by dominated convergence and the continuity of $w$ and of $\phi$ proved in Lemma 4.12.

again by dominated convergence, provided that for $s\in [0,T]$, the convergence $\chi_s^{\gamma_n}(z_n) \xrightarrow[n\rightarrow \infty]{} \chi_s^{\gamma}(z)$ holds. For this convergence to hold it is enough that for $t\in [0,T],$

It is enough to take $n$ large enough so that $d_n=d$ and to write

By the local Lipschitz property of $\lambda_{d}$,

and $\int_0^t ||\phi_s^{\gamma_n}(z_n) - \phi_s^{\gamma}(z)||_E\mathrm{d}s \leq t\sup_{s\in[0,T]} || \phi_s^{\gamma_n}(z_n) - \phi_s^{\gamma}(z)||_H \xrightarrow[n\rightarrow \infty]{} 0$ by Lemma 4.12. The second term converges to zero by the definition of the weakly* convergence in $L^{\infty}(M(Z))$.

Part 2. In the general case where $|w|\leq w_c B^{*}$, let $w^B(z,u) = w(z,u)-c_wB^*(z)\leq 0$ for $(z,u)\in \Upsilon\times U$. $w^B$ is a continuous function and there exists a nonincreasing sequence $(w_n^B)$ of bounded continuous functions such that $w_n^B \xrightarrow[n\rightarrow \infty]{} w^B$. By the first part of the proof we know that

is bounded, continuous, decreasing and converges to

which is thus upper semicontinuous. Since $b$ is a continuous bounding function it is easy to show that

is continuous so that in fact $W$ is upper semicontinuous. Now considering the function $w_B(z,u) = -w(z,u)-c_wB^*(z)\leq 0$ we easily show that $W$ is also lower semicontinuous so that finally $W$ is continuous.

Now the continuity of the applications $(z,\gamma)\rightarrow c'(z,\gamma)$ and $(z,\gamma)\rightarrow (\mathcal{Q}'w)(z,\gamma)$ comes from the previous result applied to the continuous functions defined for $(z,u) \in \Upsilon\times U$ by $w_1(z,u) := c(v,u)$ and $w_2(z,u):=\lambda_d(v,u)\int_D w(v,r,h)\mathcal{Q}(\mathrm{d}r|v,d,u)$ with $z=(v,d,h)$. Here the different assumptions of continuity (H($\lambda$))2.3., (H(c))1. and (H($\mathcal{Q}$)) are needed.