An autonomous agent for negotiation with multiple communication channels using parametrized deep Q-network

1.   Introduction

Negotiation provides a fundamental and powerful mechanism for managing interaction among computational agents ^[3], and it plays a central role in the field of distributed artificial intelligence. Effective communication is crucial in a negotiation ^[4,5,6,7], as the negotiators need to exchange information about their desires, infer their opponent's desires from communication, and balance between the two. To reach a successful negotiation, negotiation agents are thus required to employ a comprehensive strategy that integrates linguistic communication skills and bidding strategies, especially when interacting with humans. However, most work in the field of automated negotiation ^[8,9,10,11] typically uses offers/counter-offers to simplify the process, thereby ignoring linguistic communication in negotiation. While some work ^{[12,13,14,15]} considers negotiation with linguistic communication, those studies focus on the dialogue system that generates negotiation dialogues using natural language processing and incorporates the offer generation as a low-level strategy. This kind of work regards linguistic communication as a medium for conveying offer information and ignores the ability of linguistic communication to express other aspects, such as emotions and intentions. As a matter of fact, a high degree of linguistic communication skills can bring good negotiation experience and help express intentions better, thus speeding up the negotiation process.

Therefore, to model a negotiation that both supports offer and linguistic communication channels, this work follows a variant of the alternative offer protocol ^[16,17] where the negotiators can simultaneously exchange offers and linguistic information (in terms of cheaptalk). Based on this protocol, we propose a novel framework called multiple channel automated negotiation (MCAN) that leverages parametrized deep Q-networks (P-DQNs) to learn a comprehensive negotiation strategy which integrates linguistic communication skills and bidding strategies at the same time. Combining the advantage of deep Q-networks (DQNs) and a deep deterministic policy gradient (DDPG), P-DQNs, which learns the optimal bidding strategy on each kind of linguistic communication skills and then chooses the co-optimal one, is suitable for this dual optimization problem. Unlike other algorithms that learn linguistic communication skills and bidding strategies separately, P-DQNs learns these two at the same time, which enables linguistic communication skills learning and bidding strategy learning to share all useful information without the need to exchange information between separate networks.

The framework is shown in Figure 1, which models the negotiation process as a Markov decision process (MDP). The MCAN agent takes the opponent's proposal (e.g., cheaptalk and offer) as input and sends it to the proposal resolver. After updating the state, the deep reinforcement learning (DRL) policy outputs an action to the proposal generator guiding the optimal proposal. To train this agent, we designed several rule-based negotiation agents depending on classic bidding strategies and different linguistic communication skills. The experimental results show that the MCAN agent outperforms those handcrafted agents according to the averaged utility. Furthermore, user studies are conducted in a GUI human-agent negotiation environment. We evaluate the MCAN agents and other rule-based negotiation agents based on the metrics introduced in ^[2]. The experimental results of negotiation with human negotiators show that the MCAN agent has successfully learned a comprehensive negotiation strategy that integrates linguistic communication skills and bidding strategies, even trained with simple rule-based agents.

The main contributions of the work are as follows:

● To the best of our knowledge, the first DRL-based negotiation agent MCAN is proposed which allows learning linguistic communication skills and bidding strategy simultaneously.

● We implements a negotiation environment where the human or agent negotiators can negotiate through both offer and linguistic channels.

● The extensive experiments show that even under the training of only simple rule-based agents, the MCAN agent can outperform human players, and the human perceptions of the MCAN agent are significantly higher than those of other agents.

2.   Related work

Learning is important for automated negotiation and can significantly increase the performance of negotiation agents and their decision-making model. For the past decade, a variety of learning methods has been deployed in a range of negotiation scenarios. For example, Chen et al. ^[18] employed Gaussian process regression to approximate the opponent's model given no prior knowledge about their opponents' preferences and strategies. To tackle the problem of limited experience available in every single session, Chen et al. ^[19] developed a strategy that transfers knowledge efficiently from previous tasks on the basis of factored conditional restricted Boltzmann machines.

Reinforcement learning (RL) techniques, among others, have received increasingly more attention and been deployed in many negotiation tasks, like Q-learning ^[8]. More recently, deep learning has shown impressive performance in a number of areas ^{[20,21,22,23]}. RL has also been applied successfully in conjunction with deep learning, negotiation agents based on deep reinforcement learning (DRL) have gradually emerged to further improve agents' decision power. The authors of ^[9] have used an actor-critic algorithm for training both bidding and acceptance strategies in continuous state and action spaces. Sengupta et al. ^[11] use deep deterministic policy gradient (DDPG) ^[24] and soft actor-critic (SAC) ^[25] to learn a target utility for bidding strategy, which restricts the problem in continuous state and action spaces. These efforts, although successful, have considered negotiations in the form that information exchange between parties is only dependent on offers. As many real-world settings also involve linguistic channels to express intentions, ask questions, discuss plans and so on, the information bandwidth is therefore restricted and grounded in the action space of negotiation.

There exists some research on linguistic communication in negotiation. Mell et al. ^[26] used effective conversation tactics to create value over repeated negotiations, and Oudah et al. ^[2] studied the impact of different talking styles on human opponents. These two methods used rule-based communication strategies which need to be pre-programmed by experts beforehand. Cao et al. ^[27] employed a deep neural network to negotiate on both proposal and linguistic channels, but the communication is implicit and cannot have support for human-machine interaction. Other studies focused on negotiation dialogues generation based on natural language processing ^[12,13,15]. He et al. ^[12] decoupled strategy and generation in negotiation dialogues using a two-layer architecture. Zhou ^[15] created a dynamic strategy coach to give tips and analyze during negotiation. Joshi et al. ^[13] incorporated graph neural networks as a strategy-graph network to predict the next optimal strategies. This kind of work only regards linguistic communication as a medium for conveying an offer and ignores the ability of linguistic communication to express important and useful information like emotions and intentions. The MACN agent can influence opponents and express its intentions by linguistic communication in terms of cheap talk, and learns linguistic communication skills and bidding strategy explicitly and it has generalization ability when facing different negotiators in different domains. To achieve that, the proposed agent must make decisions in a hybrid discrete-continuous action space, requiring a new algorithm for this problem. Therefore, this work employs parametrized deep Q-networks (P-DQNs) ^[28] to learn a target utility for a bidding strategy, and linguistic communication skills with the opponent. The P-DQNs is able to support hybrid discrete-continuous action space where every discrete action followed by a continuous parameter, by combining the advantages of DQN and DDPG.

3.   Preliminaries

3.1.   Negotiation settings

A negotiation setting contains a protocol, domain and agents. The negotiation protocol determines the rules of how agents interact with each other. The negotiation domain determines an outcome space $\Omega$, where the agents negotiate for a win-win offer $\omega \in \Omega$. The negotiation domain can have single or multiple issues, where issues refer to the resources under contention, such as the price of an object or the level of the quality. The agents are the negotiators participating in the negotiation. The negotiation between two negotiators is called bilateral negotiation.

A large number of works ^[3,29,30] adopt the alternative offer protocal ^[17] for bilateral negotiation. During negotiation, the two negotiators in turn make offers and counter-offers until one accepts the other or the negotiation is terminated due to time-out. Another key aspect of a negotiation model is how an agent evaluates the utility of an offer. Formally, given an offer $\omega$, which contains the value of each issue, let $i$ denotes agent $i$, $v_j$ denotes the value of issue $j$ in the $\omega$, and $w_j^i$ denotes the weighting preference which agent $i$ assigns to issue $j$ (i.e., $\sum_{j = 1}^{n}w_{j}^{i} = 1, 0 \leq w_{j}^{i}\leq1$ where $n$ denotes issue numbers). Therefore, the utility of the offer $\omega$ for agent $i$ is defined as:

where $V_{j}^{i}$ is the evaluation function of agent $i$, mapping the value of issue $j$ to utility.

In order to get a good deal, agents are equipped with various bidding strategies ^[31]. At every round, the agent calculates its target expected utility, which determines whether it accepts an offer or not using these strategies. Among them, two strategies are commonly used: time-dependent strategies and behavior-dependent strategies ^[32]. Time-dependent strategies produce offers solely based on time. As time goes by, these strategies decrease the target expected utility gradually. Behavior-dependent strategies make offers depending on how opponent makes offers. The most well-known behavior-dependent strategy is tit-for-tat, which treats the opponent the same way it is treated.

3.2.   Reinforcement learning

The reinforcement learning ^[33,34,35] problem can be formulated by a Markov decision process (MDP) defined as a 5-tuple $(\mathcal{S, A, P, R}, \gamma)$, where $\mathcal{S}$ and $\mathcal{A}$ denote the sets of states and actions, respectively, $\mathcal{P}$ denotes the transition function: $\mathcal{S \times A \times S}\rightarrow \left[0, 1 \right]$, $\mathcal{R}$ denotes the reward function: $\mathcal{S \times A \times S} \rightarrow \mathbb{R}$, and $\gamma \in \left[ 0, 1\right)$ denotes the discount factor. At the step $t$, the agent observes state $s_t \in \mathcal{S}$ and selects action $a_t$ from the action space $\mathcal{A}$ following a policy $\pi$, which is a mapping from state space to action space. After taking the action, the agent receives reward $r_t$. The goal of the agent is to find a policy $\pi$ that maximizes the expected discounted cumulative return,

where $T$ is the length of the whole episode.

Deep learning has been used widely recently ^{[23,36,37,38]}. Combined with deep learning, deep Q-networks (DQNs) ^[39,40] use the Bellman equation to approximates the optimal $Q^{*}$ for discrete action space problems. For continuous action space problems, deep deterministic policy gradients (DDPGs) ^[24] consider the action-value function $Q_\pi(s_t, a_t)$ as a critic denotes how good it is when taking action $a_t$ in $s_t$, where $a_t$ is generated by deterministic policy $\mu_{\theta}: \mathcal{S} \rightarrow \mathcal{A}$. However, to deal with negotiation with offers and linguistic communication channels, the hybrid discrete-continuous action space is required to build the negotiation agent. In the following, the parametrized deep Q-networks (P-DQNs) is thus introduced.

3.3.   Parametrized deep Q-networks

As for the hybrid discrete-continuous action space where every discrete action is followed by a continuous parameter, parametrized deep Q-networks (P-DQNs) combines the advantages of DQN and DDPG and gives a general solution to this problem. We focus on a discrete-continuous hybrid action space

where $k$ is the $k$-th discrete action from discrete action set $K$, and $x_k$ is the continuous parameter corresponding to discrete action $k$ from $k$-th continuous action set $\mathcal{X}_k$. Therefore, the action value function is denoted as $Q(s, k, x_k)$, where $s \in \mathcal{S}, k \in K$, and $x_k \in \mathcal{X}_k$. Let $k_t$ be the discrete action selected at time $t$ and let $x_t$ be the associated continuous parameter. Then, the Bellman equation becomes

P-DQNs first solve $x_k^{*} = \operatorname{argsup}_{x_{k} \in \mathcal{X}_{k}} Q\left(s_{t+1}, k, x_{k}\right)$ using a deterministic policy network $x_k(s; \theta_x): \mathcal{S} \rightarrow \mathcal{X}_k$, where $\theta_x$ denotes the network weights of the policy network, and then take the largest $Q(s_{t+1}, k, x_k^{*})$ using a deep neural network $Q(s, k, x_k; \theta_Q)$, where $\theta_Q$ denotes the network weights. With this formulation, it is easy to apply the standard DQN approach of minimising the mean-square Bellman error to update the Q-network using mini-batches sampled from the replay buffer.

where $y = r+\gamma \max _{k^{\prime} \in[K]} Q\left(s^{\prime}, k^{\prime}, x_{k^{\prime}}\left(s^{\prime}; \theta_{x}\right); \theta_{Q}\right)$ is the update target derived from Eq (3.4). Then the loss for the actor network in P-DQNs is given by the negative sum of Q-values:

Through these two loss functions, we can calculate the policy gradient as

Using Eq (3.7), the policy network weight can be updated, and then the hybrid discrete-continuous action space problem can be solved. The P-DQN framework given in ^[28] can be found below.

4.   The design of the MCAN agent

4.1.   Negotiation environment

We consider the bilateral multi-issue negotiation scenario where a buyer and a seller bargain for a piece of goods with communication. We follow the process of the alternative offer protocol except that we use both offer and cheaptalk while the origin protocol only uses offer. This protocol assumes an e-market environment where agents can communicate with opponents using cheaptalk while offering an offer. A buyer $b$ always starts the negotiation by sending a proposal to seller $s$. During the negotiation both sides make proposals and counter-proposals in turn until one accepts the other, or the negotiation is terminated due to time-out. We are using the utility function defined in Eq (3.1). The cheaptalk refers to the views on negotiation that the agent wants to express to the opponent. In actual negotiations, linguistic communication can better convey the willingness to negotiate and speed up the negotiation process. Here we present a summary of the nine events that usually occur in nogotiation, as shown in Table 1.

Combining these events with different personas (nice, tough), we defines 18 different cheaptalk template for negotiators to choose. The nice persona seeks to avoid criticizing, complaining, or condemning its opponents, while respectfully building them up. On the contrary, the tough persona seeks to pull others down rather than build them up and advocates to be mean to others. The examples are given in Table 2. Besides, according to whether cheaptalk and offer are consistent, the negotiators can be honest or dishonest. For example, a dishonest negotiator will choose the event promise to the opponent to make concessions while giving an offer without concessions.

4.2.   Proposed negotiation framework

We model this negotiation problem as an MDP, and the overall architecture is shown in Figure 1. To simplify and highlight the core of our work, cheaptalk is presented by its cheaptalk index in the pre-defined templates in Table 2. The MCAN agent receives the proposal from the opponent and uses the proposal resolver, which resolves the offer and cheaptalk index from the proposal. After calculating the utility of the offer, the information is added to the utility history memory and cheaptalk history memory, respectively. The state is then generated by concatenating the current time, utility received history, utility proposed history, cheaptalk received history and cheaptalk proposed history. After that, the MCAN agent uses the DRL policy to generate an action which contains the best responding cheaptalk index and the corresponding target expected utility. After checking the acceptance condition, the counter-proposal is made by proposal generator.

4.2.1.   Proposal resolver and proposal generator

The proposal resolver and the proposal generator are similar, since they're the reverse process of each other. The proposal resolver uses a sentence from the opponent as input and outputs the core of the proposal in the form of a dictionary while the proposal generator uses the proposal in dictionary form as input and outputs the responding sentence to the opponent. Using the pre-defined cheaptalk template, we can easily classify the sentence paradigm and extract the offer and cheaptalk index. The offer generator in the proposal generator is defined as

where $u_{target}$ denotes the target expected utility the agent wants to attain, and $\omega$ denotes any outcome in the outcome space $\Omega$.

4.2.2.   Utility history and cheaptalk history

Utility history and cheaptalk history are used to record the proposals proposed and received in the negotiation history. After resolving the proposal from the opponent, the utility of the offer will be calculated using Eq (3.1). It is then added to the utility received history. Meanwhile, the cheaptalk index will be added to the talk received history. After the MCAN agent makes a proposal, the utility proposed history and cheaptalk proposed history will be updated.

4.2.3.   DRL policy

DRL policy is aiming to give the best cheaptalk index and the corresponding target expected utility for an offer at a certain state. We adopt the P-DQN algorithm here, as P-DQN gives a general solution to the hybrid discrete-continuous action space problem. It adapts well to our negotiation settings because it is able to calculate the target utility and cheaptalk index at the same time, which solves the problem of mutual influence between cheaptalk index action and target expected utility action.

The state is concatenated by the current time, utility received history, utility proposed history, cheaptalk received history and cheaptalk proposed history. For a negotiation session with time limit $T$, we use $u_{propose}[t-i, t]$ to denote the utility proposed history from $t - i$ to $t$, where $i \in \left[0, t - 1\right]$. Similarly, $u_{receive}[t-i, t]$ denotes the utility received vector. $c_{propose}[t-i, t]$ denotes the cheaptalk index proposed from $t - i$ to $t$, and $c_{receive}[t-i, t]$ denotes the cheaptalk index received from $t - i$ to $t$. After defining these, the state and action can be defined as

where $t_r$ denotes the relative time ($t_r = t / T$). The action outputs the cheaptalk index and the target expected utility. By checking the acceptance condition as (Eq (4.2)), the agent decides to use the proposal generator or accept the opponent's proposal.

The goal of the DRL agent is to maximize the utility of the agreement offer, so the reward function is defined as

where $U(\omega)$ means the self utility of the offer $\omega$.

The main algorithm is shown in Algorithm 2. Before training, an opponent pool is prepared for the MCAN agent to negotiate with. In every episode, the algorithm randomly chooses an opponent from the opponent pool $\Pi$. Then, the MCAN agent and the chosen opponent are asked to start a negotiation. During the negotiation, the MCAN agent collects the history information and updates the state $s$. Using the $\epsilon$-greedy policy, the MCAN agent generates the current target expected utility and corresponding cheaptalk index. After checking the acceptance condition, the proposal is sent to the opponent. Through the alternative proposal, the MCAN collects the experience in the replay buffer and uses them to update the Q actor network and parameter actor network weights. After several rounds of training, the policy converged.

5.   Experiments

In this section, we present the experimental settings and results of the MCAN agent against opponents. We begin with the experimental technicalities, which are given in Section 5.1. Section 5.2 compares the performance of our agent against the rule based negotiation agents. Then, a user study is conducted to obtain human evaluation in Section 5.3. A GUI human-agent negotiation environment is developed to facilitate interaction between agents with human players. Section 5.4 reports a comparative experiment to show how the P-DQN algorithm promotes the performance of the MCAN agent.

5.1.   Experimental setup

We first design several rule-based negotiation agents and use them to train our MCAN agent. The rule-based negotiation agents are designed by integrating the linguistic communication skills of humans and classic bidding strategies. Here, we named the nice persona as friendly and the tough persona as unfriendly. Besides, according to whether the negotiator is honest, we divide linguistic communication skills into four types. The classic bidding strategies we used here are time-dependent Boulware agent and behavior-dependent tit-for-tat agent. We combine the linguistic communication skills and bidding strategy by using the target expected utility. Each round, the bidding strategy outputs a target expected utility based on utility history information and time, and the target expected utility is modified based on different linguistic communication skills. Besides, friendly agents will choose the left column of Table 2, and unfriendly agents choose the right side. The honesty will decide whether the offer and cheaptalk have consistent performance. We named these 8 rule-based negotiation agents as friendly honest time-dependent Boulware (fht), friendly dishonest time-dependent Boulware (fdt), unfriendly honest time-dependent Boulware (uht), unfriendly dishonest time-dependent Boulware (udt), friendly honest behavior-dependent (fhb), friendly dishonest behavior (fdb), unfriendly honest behavior-dependent (uhb) and unfriendly dishonest behavior-dependent (udb), respectively. For the MCAN agent, we use the implement of the P-DQN from ^[28]. The only difference in the network architecture is that we added a layer activation function of tanh at the end of the actor parameter network to limit the output to between –1 and 1. The hyperparameters are set as follows: batch size = 128, discount = 0.99, Q-networks update parameter = 0.1, actor parameter networks update parameter = 0.001, replay buffer size = 100000. The optimizers for Q-networks and the actor parameter network are both Adam, with learning rates 0.001 and 0.0001, respectively.

All the experiments were conducted using our self-developed Python environment. Among the negotiation settings, the reservation price is set as 0.1, and the maximum number of round in one episode is 28–32 rounds to avoid possible exploitation. We used min-max normalisation for making the utility values between 0 and 1.

5.2.   Performance against rule-based negotiation agents

We trained the MCAN agent for 1,000,000 episodes, and the learning curve and evaluation curve of the first 500,000 episodes are shown in Figure 2. In the training phase, in each episode the MCAN agent will face an opponent randomly sampled from 8 rule-based negotiation agents. In the evaluation phase, the MCAN agent will have negotiations with all 8 opponents one by one. We can find that the MCAN agent converged at around 200,000 episodes with a reward of 0.68. Through the evaluation result graph, we can find that there is an obvious advantage in the negotiation with fht and fhb. These two agents are more likely to make concessions first, which makes the MCAN agent achieve a good deal easily. For negotiation with ufht and ufhb, the MCAN agent can gain a slight advantage by reasonable using of cheaptalk signals since they are honest. For the remaining four agents, the MCAN agent will try to reach a tie as much as possible. With the addition of the MCAN agent, there are a total of 9 agents. We let these 9 agents negotiate in pairs, each pair carries out 1000 evaluations. Then, we calculated the averaged agreement utility over 1000 evaluations as the evaluation metric for one pair. Fht and fub are equipped with the friendly and honest talk strategy. When facing other agents, they are likely to concede first. This leads them to reach negotiations with all agents, but the agreement utility of the negotiation is mostly below 0.5. The other six agents are either dishonest or tough, which leads them to be unable to reach a deal. We averaged the results of one agent against 8 other agents, and the result is shown in Figure 3.

It can be seen that the MCAN agent reached the highest average utility of 0.59, followed by fht and fhb, with 0.46 and 0.41, respectively. The remaining four agents are relatively close, all around 0.2. Through the above experiments, we can find that the MCAN agent can obtain a relatively good advantage when negotiating with the rule-based negotiation agents.

5.3.   User study

To evaluate the negotiating performance against human participants, a GUI human-agent negotiation with talk environment was developed to enable agents to negotiate with humans. In this environment, we invited 200 volunteers from the College of Intelligence and Computing, Tianjin University, to conduct a human-agent negotiation test. Human volunteers are invited to be a party to the negotiation (buyer or seller), and their tasks are to get the highest possible agreement utility of negotiation. For the convenience of the volunteers, we used single-issue negotiation instead. Each volunteer participated 30 tests. For every 10 tests, they faced one agent randomly sampled from those 9 agents to fill out a questionnaire. We used the two evaluation metrics introduced by Oudah ^[2]. The first one is the average utility made between a volunteer and an agent, and the second one is the degree to winning friends, which is quantified through the questionnaire. The volunteers are asked (using a 5-point Likert scale) the degree to which they understand their opponent's intentions to what degree it is likable, intelligent, cooperative, friendy and willing to negotiate again. To compare the relative performances of the agents, we used the standardized z-score for each metric.

The results are shown in Figure 4. From Figure 4(a) we can find that the averaged utility obtained by the MCAN is the highest among the 9 agents, and it even slightly surpassed the performance of humans. We have found that maintaining honesty is an important reason for obtaining higher averaged utility since the fht, ufht, fhb and ufhb will maintain good credit in the negotiation. Besides, the performances of fht and fhb far exceeds other agents, which shows that keeping a nice persona will better than a tough persona. The results of Figure 4(b) are quite similar to Figure 4(a). The MCAN obtained the highest score among all agents, which shows that it is more capable of winning friends compared to the other rule-based agents. In the Figure 4(b), an interesting result is that the score of ufht and ufhb is not as positive as in Figure 4(a), which implies that the friendliness and honesty are both important. The Figure 5 summarizes the results of the study by showing the relative performance of the 9 agents in terms of averaged utility and score of winning friends.

According to Figure 5, it can be found that the MCAN agent learned the linguistic communication skills and bidding strategy successfully. To further analyze the behaviors learned by the MCAN agent, we visualized the cheaptalk MCAN has used in the negotiation with human volunteers, which is shown in Figure 6. The horizontal axis denotes the cheaptalk classes. Among them, the first 9 are the cheaptalk under the nice persona in the order of Table 2, while the last 9 are based on the tough persona. The results shows that the MCAN agent learns to behavior based on the the nice persona and is more likely to use indices 1, 2 and 3. This phenomenon shows that MCAN agent knows to show sincerity by expressing concessions to opponents in order to build trust between both parties and express its needs politely. These three cheaptalk actions are also worthy of attention, and they can be used to provide references for future linguistic design.

5.4.   Analysis of P-DQN

In this section we will study the effect of the P-DQN by conducting a comparative experiment. The P-DQN takes the utility history and cheaptalk history as input and outputs the best cheaptalk index and the corresponding target expected utility. To evaluate the P-DQN's output, we made two variants of the P-DQN by replacing either the cheaptalk index side or the target expected utility side with random output, called the TP-DQN and OP-DQN, respectively. Then we carried out two other tournaments like we did in experiment 1 by replacing the P-DQN with our two variants. Then, we calculated the self utility attained against the 8 rule-based agents and the opponent average utility attained. The results are shown in Figure 7, and it can be found that the performances of the TP-DQN and OP-DQN plummet, which implies that the P-DQN's output is indivisible. The combined effect of cheaptalk index and target expected utility gives the P-DQN a great advantage.

6.   Conclusion and future work

This work proposes a novel framework MCAN that leverages P-DQNs to learn a comprehensive negotiation strategy which integrates linguistic communication skills and bidding strategies. The P-DQNs algorithm, which learns the optimal bidding strategy on each kind of linguistic communication skills and then chooses the co-optimal one, is suitable for this dual optimization problem. The experimental result shows that the MCAN agent outperforms the rule-based agent according to the averaged utility of negotiation. According to the user study, we also evaluated the MCAN agent and rule-based agents based on the metrics of winning friends in addition to utility metrics. The user study on negotiation with humans depicts that the MCAN agent has learned a comprehensive negotiation strategy which integrates linguistic communication skills and bidding strategies.

The exceptional results justify investment in further research efforts into this approach. In the future, we plan to explore how to learn more sophisticated linguistic communication skills to achieve a more application-oriented architecture. Second, the extension of the framework to other negotiation settings, such as concurrent negotiation or multi-lateral negotiation, is another interesting avenue to exploit. It is also of great interest to investigate how to train negotiation skills of human participants based on the framework. Last but not least, to further aid the learning performance in a different negotiation domain, incorporating transfer learning techniques into the proposed approach is believed to lead to improvement in negotiation power.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 61602391).

Conflict of interest

The authors declare there is no conflict of interest.