Simulation and Experimental on Coordination Control of Dual-Valve Electrohydraulic Servo Systems Based on Integration of Reinforcement Learning and Adaptive Robust Control Algorithm

doi:10.3969/j.issn.1004-132X.2026.02.005

Abstract

Abstract:

To enhance the control performance of a dual-valve electro-hydraulic servo system，this study conducts simulation analysis and experimental validation on a proposed coordinated control strategy， SAC-ARC， which integrates reinforcement learning with adaptive robust control.First， a co-simulation model of the hydraulic system was established using the AMESim and Simulink software platforms， and the tracking performance of the dual-valve electro-hydraulic servo system was analyzed under various proportional valve control signal compensation strategies. Subsequently， comparative simulations were performed to evaluate the tracking performance and robustness of the SAC-ARC strategy. The tracking errors of SAC-ARC were compared with those of PID， ARC， and RBF-ARC control strategies under complex working conditions， including various composite signals and the presence of internal and external disturbances. Finally， experimental validation was carried out on an established test platform. The simulation and experimental results demonstrate that the SAC-ARC control strategy exhibits superior tracking performance under all tested working conditions. Its maximum transient error and cumulative tracking error are both significantly lower than those of the comparative control strategies， thus validating the effectiveness and superiority of the proposed strategy for the dual-valve electro-hydraulic servo system.

Key words: electrohydraulic servo system, dual-valve parallel control, flow allocation, co-simulation

摘要：

为提高异构双阀电液伺服系统的控制性能，在融合强化学习与自适应鲁棒控制算法的异构双阀协调控制策略SAC-ARC的基础上，开展SAC-ARC控制策略仿真分析及实验验证。首先利用AMESim和Simulink软件平台建立了液压系统联合仿真模型，分析了不同比例阀控制信号补偿策略下异构双阀电液伺服系统的跟踪性能。然后对比仿真了SAC-ARC与PID、ARC及RBF-ARC控制策略在多种复合信号及系统受到内外扰动等复杂工况下的跟踪误差，以验证其跟踪性能与鲁棒性。最后在搭建的实验平台上进行了实验验证。仿真与实验结果表明：SAC-ARC控制策略在各工况下均表现出优异的跟踪性能，其最大瞬态误差和累计跟踪误差均显著低于其他对比控制策略，验证了该控制策略在异构双阀电液伺服系统中的有效性与优越性。

关键词: 电液伺服系统, 双阀并联控制, 流量分配, 联合仿真

CLC Number:

TP273

SU Shijie, CHENG Yongqin, HU Yi, HE Jianhui, YANG Shuji. Simulation and Experimental on Coordination Control of Dual-Valve Electrohydraulic Servo Systems Based on Integration of Reinforcement Learning and Adaptive Robust Control Algorithm[J]. China Mechanical Engineering, 2026, 37(2): 295-303.

苏世杰, 程泳钦, 胡毅, 何建辉, 杨书吉. 融合强化学习自适应鲁棒控制算法的异构双阀协调控制策略仿真及实验[J]. 中国机械工程, 2026, 37(2): 295-303.

Figures/Tables 23

Fig.1 The control model of dual-valve electro-hydraulic servo system

Fig.2 The co-simulation model

Tab.1 Co-simulation parameters

参数	数值
液压泵的排量/（mL·r^-1）	10
电机的转速/（r·min^-1）	1440
活塞行程/m	0.2
活塞杆直径/m	0.05
活塞直径/m	0.1
溢流阀的启动压力/MPa	14
负载质量/kg	100
比例阀的死区/%	15
比例阀的额定压降/MPa	7
比例阀的最大流量/（L·min^-1）	12
比例阀的响应频率/Hz	15
伺服阀的额定压降/MPa	7
伺服阀的最大流量/（L·min^-1）	4
伺服阀的响应频率/Hz	60

Tab.2 Training hyperparameters of SAC network

参数	数值
激活函数	ReLU
学习率（ $λ Q$ 和 $λ π$ ）	0.001
衰减因子（ $γ$ ）	0.99
记忆缓冲区的数量	$1 × 106$
隐藏层的单元数（所有网络）	128
隐藏层数目	3

Tab.2 Training hyperparameters of SAC network

参数	数值
激活函数	ReLU
学习率（ $λ Q$ 和 $λ π$ ）	0.001
衰减因子（ $γ$ ）	0.99
记忆缓冲区的数量	$1 × 106$
隐藏层的单元数（所有网络）	128
隐藏层数目	3

Tab 3 Training sample set

样本分类		样本模型
复合信号S₀		$y = 6 t t ≤ 1 6 1 < t ≤ 2 6 s i n (π t + π / 2) 2 < t ≤ 3.5 6 s i n (1.5 π t + 7 π / 4) 3.5 < t ≤ 4.5 - 6 t > 4.5$
内部扰动 $P$ 与外部负载 $F$ 变化		y=｛10 mm/s（滑块速度），2 s（周期）， 10 mm（振幅）｝ $F = 0 t ≤ t 1 Z 1 t > t 1$ $Z 1 ∈ (5,15), t 1 ∈ (1,3.5)$ $P = 14 t ∉ (t 2, t 2 + 0.1) Z 2 t ∈ (t 2, t 2 + 0.1)$ $Z 2 ∈ (4,7), t 2 ∈ (1,3.5)$
不确定性信号	复合斜坡信号S₁	$y = 0 t < 0.5 a t - 0.5 a 0.5 ≤ t < 1.5 a 1.5 ≤ t < 2.5 - a t + 3.5 a 2.5 ≤ t < 3.5 0 t ≥ 3.5$ $a ∈ [3,8]$
	阶梯信号S₂	y=｛10 mm/s（滑块速度），2 s（周期）， k mm（振幅）｝， $k ∈ [5,10]$
	方波信号S₃	$y = k t < 1 0 1 ≤ t < 2 k 2 ≤ t < 3 0 t ≥ 3$ $k ∈ [2,6]$ ， $k ∈ N$
	时变正弦信号S₄	$y = a s i n (b π t) (1 - 10 e 3)$ ， $a ∈ [2,6]$ ， $b ∈ [0.5,1.5]$

Tab 3 Training sample set

样本分类		样本模型
复合信号S₀		$y = 6 t t ≤ 1 6 1 < t ≤ 2 6 s i n (π t + π / 2) 2 < t ≤ 3.5 6 s i n (1.5 π t + 7 π / 4) 3.5 < t ≤ 4.5 - 6 t > 4.5$
内部扰动 $P$ 与外部负载 $F$ 变化		y=｛10 mm/s（滑块速度），2 s（周期）， 10 mm（振幅）｝ $F = 0 t ≤ t 1 Z 1 t > t 1$ $Z 1 ∈ (5,15), t 1 ∈ (1,3.5)$ $P = 14 t ∉ (t 2, t 2 + 0.1) Z 2 t ∈ (t 2, t 2 + 0.1)$ $Z 2 ∈ (4,7), t 2 ∈ (1,3.5)$
不确定性信号	复合斜坡信号S₁	$y = 0 t < 0.5 a t - 0.5 a 0.5 ≤ t < 1.5 a 1.5 ≤ t < 2.5 - a t + 3.5 a 2.5 ≤ t < 3.5 0 t ≥ 3.5$ $a ∈ [3,8]$
	阶梯信号S₂	y=｛10 mm/s（滑块速度），2 s（周期）， k mm（振幅）｝， $k ∈ [5,10]$
	方波信号S₃	$y = k t < 1 0 1 ≤ t < 2 k 2 ≤ t < 3 0 t ≥ 3$ $k ∈ [2,6]$ ， $k ∈ N$
	时变正弦信号S₄	$y = a s i n (b π t) (1 - 10 e 3)$ ， $a ∈ [2,6]$ ， $b ∈ [0.5,1.5]$

Fig.3 Proportional valve control signals with different compensation strategies

Fig.4 Responses and tracking errors of the proportional valve under different control signal compensation strategies

Fig.5 Maximum tracking error of the proportional valve under different control signal compensation strategies

Fig.6 The training process of the SAC-ARC control strategy

Fig.7 Responses and tracking errors of the composite signal input for various control strategies

Fig.8 ARC parameters of RBF-ARC and SAC-ARC control strategies when tracking composite signals

Fig.9 The IAE value of the composite signal input for various control strategies

Tab.4 Parameter values of randomly disturbance conditions

工况/参数	P/MPa	$t 1$ /s	Z/kN	$t 2$ /s
W₁	6	1.2	10	2.4
W₂	6	1.2	15	3.4
W₃	5	2.2	5	1.4
W₄	5	2.2	15	3.4
W₅	4	3.2	5	1.4
W₆	4	3.2	10	2.4

Tab.4 Parameter values of randomly disturbance conditions

工况/参数	P/MPa	$t 1$ /s	Z/kN	$t 2$ /s
W₁	6	1.2	10	2.4
W₂	6	1.2	15	3.4
W₃	5	2.2	5	1.4
W₄	5	2.2	15	3.4
W₅	4	3.2	5	1.4
W₆	4	3.2	10	2.4

Fig.10 Responses and tracking errors of the disturbance condition （W6） for various control strategies

Tab.5 IAE values of the stair-step signal input under different disturbance conditions for various control strategies

工况/控制策略	PID	ARC	RBF-ARC	SAC-ARC
W₁	705.4	41.4	57.2	24.6
W₂	702.7	44.1	60.8	24.9
W₃	706.5	42.8	64.9	19.6
W₄	702.4	51.2	73.5	24.0
W₅	707.3	35.8	68.6	22.8
W₆	705.8	44.4	73.1	27.7

Fig.11 Responses and tracking errors of composite ramp signal 1 for various control strategies

Fig.12 Responses and tracking errors of stair-step signal 1 input for various control strategies

Fig.13 Responses and tracking errors of the time-varying sinusoidal signals input for various control strategies

Fig.14 Responses and tracking errors of square signals input for various control methods

Tab 6 The IAE value of the different signal inputs for various control strategies

输入信号	控制策略	信号1	信号2
复合斜坡信号	PID	235.7	373.7
	ARC	15.8	17.7
	RBF-ARC	25.2	31.0
	SAC-ARC	4.1	8.8
阶梯信号	PID	445.3	888.7
	ARC	23.5	31.6
	RBF-ARC	48.1	61.1
	SAC-ARC	12.2	19.7
时变正弦信号	PID	937.6	983.2
	ARC	38.9	32.9
	RBF-ARC	85.6	72.4
	SAC-ARC	24.7	21.3
方波信号	PID	803.6	1954.5
	ARC	742.8	1776.6
	RBF-ARC	615.4	1589.6
	SAC-ARC	609.9	1570.5

Fig.15 The experimental platform of heterogeneous dual-valve control system

Fig.16 Responses and tracking errors of composite signal S0 input for various control strategies

Fig.17 The IAE value of composite signal S0 input for various control strategies

References 15

[1]	苏世杰，程泳钦，胡毅，等.基于强化学习自适应鲁棒控制的异构双阀协调控制策略研究［J］. 中国机械工程，2025， 36（10）： 2335-2342.
	SU Shijie， CHENG Yongqin， HU Yi， et al. Coordination Control of Dual-Valve Electrohydraulic Servo Systems Based on Integration of Reinforcement Learning and Adaptive Robust Control［J］. China Mechanical Engineering， 2025， 36（10）： 2335-2342.
[2]	Litong LYU， CHEN Zheng， YAO Bin， et al. Development of Parallel-connected Pump-valve-coordinated Control Unit with Improved Performance and Efficiency［J］. Mechatronics， 2020， 70： 102419.
[3]	LIU Yi， WANG Tao， GONG Guofang， et al. Present Status and Prospect of High-frequency Electro-hydraulic Vibration Control Technology［J］. Chinese Journal of Mechanical Engineering， 2019， 32（1）： 93.
[4]	QUAN Zhongyi， QUAN Long， ZHANG Jinman. Review of Energy Efficient Direct Pump Controlled Cylinder Electro-hydraulic Technology［J］. Renewable and Sustainable Energy Reviews， 2014， 35： 336-346.
[5]	YU Huan， CHEN Ying， SHI Wenzhao， et al. State Constrained Variable Structure Control for Active Heave Compensators［J］. IEEE Access， 2019， 7： 54770-54779.
[6]	XU Bing， DING Ruqi， ZHANG Junhui， et al. Modeling and Dynamic Characteristics Analysis on a Three-stage fast-response and Large-flow Directional Valve［J］. Energy Conversion and Management， 2014， 79： 187-199.
[7]	YU Shaojuan， SONG Junjun. Iterative Learning Control of Double Servo Valve Controlled Electro Hydraulic Servo System［C］∥2011 Seventh International Conference on Computational Intelligence and Security. IEEE， 2011： 278-282.
[8]	HE Jianhui， SU Shijie， WANG Hairong， et al. Online PID Tuning Strategy for Hydraulic Servo Control Systems via SAC-based Deep Reinforcement Learning［J］. Machines， 2023， 11（6）： 593.
[9]	姚静，寇成浩，尹钰鑫，等. 超高压大流量比例插装阀测试方法［J］. 中国机械工程， 2020， 31（6）： 638-646.
	YAO Jing， KOU Chenghao， YIN Yuxin， et al. Test Methods of Ultra-high Pressure and Large Flow Proportional Cartridge Valves［J］. China Mechanical Engineering，2020， 31（6）： 638-646.
[10]	BAI Yanhong， QUAN Long. Improving electro-hydraulic System Performance by double-valve Actuation［J］. Transactions of the Canadian Society for Mechanical Engineering， 2016， 40（3）： 289-301.
[11]	SU Shijie， XUE Ting， CHEN Yun， et al. Harmonic Control of a Dual-valve Hydraulic Servo System with Dynamically Allocated Flows［J］. Asian Journal of Control， 2023， 25（3）： 1939-1956.
[12]	WANG Tianzhu， ZHANG Qiang， FANG Jinhui， et al. Active Fault-tolerant Control for the Dual-valve Hydraulic System with Unknown Dead-zone［J］. ISA Transactions， 2024， 145： 399-411.
[13]	SU Shijie， ZHU Yuanyuan， LI Cunjun， et al. Dual-valve Parallel Prediction Control for an Electro-hydraulic Servo System［J］. Science Progress， 2020， 103（1）： 0036850419875662.
[14]	WANG Bowen. Research on Nonlinear Calibration of Mine Catalytic-combustion-based Combustible-gas Sensor Based on RBF Neural Network［J］. Heliyon， 2023， 9（3）： e14055.
[15]	陈志勇，李攀，叶明旭，等.自动驾驶电动车辆基于参数预测的径向基函数神经网络自适应控制［J］.中国机械工程，2024，35（6）：982-992.
	CHEN Zhiyong， LI Pan， YE Mingxu， et al. RBF Neural Network Adaptive Control Based on Parameter Prediction for Self-driving Electric Vehicles［J］. China Mechanical Engineering，2024，35（6）：982-992.