Research on Flexible Job-shop Scheduling Considering Constraints of Peak Power Constrained

doi:10.3969/j.issn.1004-132X.2025.02.011

Abstract

Abstract: Peak power constrained flexible job shop scheduling problem(PPCFJSP) model was established to address the challenges of increased work cycles and increased machine load in flexible job shop scheduling under the constraints of peak power in the workshops. The optimization objectives were to minimize the maximum completion time and the maximum machine loads, taking into account the constraints of peak power in the workshops. For better scheduling decisions, firstly, the problem was transformed into a Markov decision process, then, a scheduling framework combining offline training and online scheduling was designed for solving PPCFJSP. Secondly, a double dueling deep q-network based on priority experience replay(D3QNPER) algorithm was designed based on priority experience replay, and a ε- greedy descent strategy introducing noise was designed to improve the convergence speed of the algorithm, further enhance the solving ability and stability of the solution results. Finally, experimental and algorithmic comparative studies were conducted to verify the effectiveness of the model and algorithm.

Key words: flexible job shop scheduling, Markov decision process, deep reinforcement learning, peak power constrained

摘要： 针对车间峰值功率受限约束下的柔性作业车间调度面临的作业周期增加、机器负荷增大的问题，建立以最小化最大完工时间和最小化机器最大负载为优化目标、考虑车间峰值功率约束的柔性作业车间调度问题（PPCFJSP）模型。为更好地调度决策，首先将该问题转化为马尔可夫决策过程，基于此设计了一个结合离线训练与在线调度的用于求解PPCFJSP的调度框架。然后设计了一种基于优先级经验重放的双重决斗深度Q网络(D3QNPER）算法，并设计了一种引入噪声的ε-贪婪递减策略，提高了算法收敛速度，进一步提高了求解能力和求解结果的稳定性。最后开展实验与算法对比研究，验证了模型和算法的有效性。

关键词: 柔性作业车间调度, 马尔可夫决策过程, 深度强化学习, 峰值功率受限

CLC Number:

TH165
TP18

LI Yibing1, 2, CAO Yan1, GUO Jun1, 2, WANG Lei1, 2, LI Xixing3, SUN Libo4. Research on Flexible Job-shop Scheduling Considering Constraints of Peak Power Constrained[J]. China Mechanical Engineering, 2025, 36(02): 280-293.

李益兵1, 2, 曹岩1, 郭钧1, 2, 王磊1, 2, 李西兴3, 孙利波4. 考虑峰值功率受限约束的柔性作业车间调度研究[J]. 中国机械工程, 2025, 36(02): 280-293.

References

［1］李凯文, 张涛, 王锐, 等. 基于深度强化学习的组合优化研究进展［J］. 自动化学报, 2021, 47(11):2521-2537.
LI Kaiwen, ZHANG Tao, WANG Rui, et al. Research Reviews of Combinatorial Optimization Methods Based on Deep Reinforcement Learning［J］. Acta Automatica Sinica, 2021, 47(11):2521-2537.
［2］李颖俐, 李新宇, 高亮. 混合流水车间调度问题研究综述［J］. 中国机械工程, 2020, 31(23):2798-2813.
LI Yingli, LI Xinyu, GAO Liang. Review on Hybrid Flow Shop Scheduling Problems［J］. China Mechanical Engineering, 2020, 31(23):2798-2813.
［3］黎声益, 马玉敏, 刘鹃. 基于双深度Q学习网络的面向设备负荷稳定的智能车间调度方法［J］. 计算机集成制造系统, 2023, 29(1):91-99.
LI Shengyi, MA Yumin, LIU Juan. Smart Shop Floor Scheduling Method for Equipment Load Stabilization Based on Double Deep Q-learning Network［J］. Computer Integrated Manufacturing Systems, 2023, 29(1):91-99.
［4］贺俊杰, 张洁, 张朋, 等. 基于长短期记忆近端策略优化强化学习的等效并行机在线调度方法［J］. 中国机械工程, 2022, 33(3):329-338.
HE Junjie, ZHANG Jie, ZHANG Peng, et al. Related Parallel Machine Online Scheduling Method Based on LSTM-PPO Reinforcement Learning［J］. China Mechanical Engineering, 2022, 33(3):329-338.
［5］LIU Renke, PIPLANI R, TORO C. Deep Reinforcement Learning for Dynamic Scheduling of a Flexible Job Shop［J］. International Journal of Production Research, 2022, 60(13):4049-4069.
［6］LI Yuxin, GU Wenbin, YUAN Minghai, et al. Real-time Data-driven Dynamic Scheduling for Flexible Job Shop with Insufficient Transportation Resources Using Hybrid Deep Q Network［J］. Robotics and Computer-Integrated Manufacturing, 2022, 74:102283.
［7］WU Wenbo, HUANG Zhengdong, ZENG Jiani, et al. A Fast Decision-making Method for Process Planning with Dynamic Machining Resources via Deep Reinforcement Learning［J］. Journal of Manufacturing Systems, 2021, 58:392-411.
［8］LEE Y H, LEE S. Deep Reinforcement Learning Based Scheduling within Production Plan in Semiconductor Fabrication［J］. Expert Systems with Applications, 2022, 191:116222.
［9］HE Zhenglei, TRAN K P, THOMASSEY S, et al. Multi-objective Optimization of the Textile Manufacturing Process Using Deep-Q-network Based Multi-agent Reinforcement Learning［J］. Journal of Manufacturing Systems, 2022, 62:939-949.
［10］郭具涛, 吕佑龙, 戴铮, 等. 基于复合规则和强化学习的混流装配线调度方法［J］. 中国机械工程, 2023, 34(21):2600-2606.
GUO Jutao, LYU Youlong, DAI Zheng, et al. Compound Rules and Reinforcement Learning Based Scheduling Method for Mixed Model Assembly Lines［J］. China Mechanical Engineering, 2023, 34(21):2600-2606.
［11］刘亚辉, 申兴旺, 顾星海, 等. 面向柔性作业车间动态调度的双系统强化学习方法［J］. 上海交通大学学报, 2022, 56(9):1262-1275.
LIU Yahui, SHEN Xingwang, GU Xinghai, et al. A Dual-system Reinforcement Learning Method for Flexible Job Shop Dynamic Scheduling［J］. Journal of Shanghai Jiao Tong University, 2022, 56(9):1262-1275.
［12］ZHANG Jiadong, HE Zhixiang, CHAN W H, et al. DeepMAG:Deep Reinforcement Learning with Multi-agent Graphs for Flexible Job Shop Scheduling［J］. Knowledge-Based Systems, 2023, 259:110083.
［13］GUI Yong, TANG Dunbing, ZHU Haihua, et al. Dynamic Scheduling for Flexible Job Shop Using a Deep Reinforcement Learning Approach［J］. Computers & Industrial Engineering, 2023, 180:109255.
［14］ZHANG Lu, FENG Yi, XIAO Qinge, et al. Deep Reinforcement Learning for Dynamic Flexible Job Shop Scheduling Problem Considering Variable Processing Times［J］. Journal of Manufacturing Systems, 2023, 71:257-273.
［15］何彦, 王乐祥, 李育锋, 等. 一种面向机械车间柔性工艺路线的加工任务节能调度方法［J］. 机械工程学报, 2016, 52(19):168-179.
HE Yan, WANG Lexiang, LI Yufeng, et al. A Scheduling Method for Reducing Energy Consumption of Machining Job Shops Considering the Flexible Process Plan［J］. Journal of Mechanical Engineering, 2016, 52(19):168-179.
［16］DU Yu, LI Junqing, LI Chengdong, et al. A Reinforcement Learning Approach for Flexible Job Shop Scheduling Problem with Crane Transportation and Setup Times［J］. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(4):5695-5709.
［17］NAIMI R, NOUIRI M, CARDIN O. A Q-learning Rescheduling Approach to the Flexible Job Shop Problem Combining Energy and Productivity Objectives［J］. Sustainability, 2021, 13(23):13016.
［18］LI Rui, GONG Wenyin, LU Chao, et al. A Learning-based Memetic Algorithm for Energy-efficient Flexible Job-shop Scheduling with Type-2 Fuzzy Processing Time［J］. IEEE Transactions on Evolutionary Computation, 2023, 27(3):610-620.
［19］张凯, 毕利, 焦小刚. 集成强化学习算法的柔性作业车间调度问题研究［J］. 中国机械工程, 2023, 34(2):201-207.
ZHANG Kai, BI Li, JIAO Xiaogang. Research on Flexible Job-shop Scheduling Problems with Integrated Reinforcement Learning Algorithm［J］. China Mechanical Engineering, 2023, 34(2):201-207.
［20］陈睿奇, 黎雯馨, 王传洋, 等. 基于深度强化学习的工序交互式智能体Job shop调度方法［J］. 机械工程学报, 2023, 59(12):78-88.
CHEN Ruiqi, LI Wenxin, WANG Chuanyang, et al. Interactive Operation Agent Scheduling Method for Job Shop Based on Deep Reinforcement Learning［J］. Journal of Mechanical Engineering, 2023, 59(12):78-88.

[1]	HU Kaixiong1, SONG Yuanhang1, ZHOU Yong1, LI Weidong2. A Cooperative Strategy for Pushing and Grasping Target Object in Cluttered Scenes Based on Deep Reinforcement Learning [J]. China Mechanical Engineering, 2025, 36(01): 133-140.
[2]	WEI Shupeng, TANG Hongtao, LI Xixing, YANG Guanyu, ZHANG Jian. Dual-resource Constrained Flexible Machining Workshop Inverse Scheduling Problem [J]. China Mechanical Engineering, 2024, 35(03): 457-471.
[3]	GUO Jutao, LYU Youlong, DAI Zheng, ZHANG Jie, GUO Yu. Compound Rules and Reinforcement Learning Based Scheduling Method for Mixed Model Assembly Lines [J]. China Mechanical Engineering, 2023, 34(21): 2600-2606,2614.
[4]	ZHANG Kai, BI Li, JIAO Xiaogang. Research on Flexible Job-shop Scheduling Problems with Integrated Reinforcement Learning Algorithm [J]. China Mechanical Engineering, 2023, 34(02): 201-207.
[5]	LI Xixing, YANG Daoming, LI Xin, WU Rui, . Flexible Job Shop AGV Fusion Scheduling Method Based on HGWOA [J]. China Mechanical Engineering, 2021, 32(08): 938-950，986.
[6]	YANG Dongjing, LEI Deming. A Novel Shuffled Frog-leaping Algorithm for FJSP withTotal Energy Consumption Constraints [J]. China Mechanical Engineering, 2018, 29(22): 2682-2689.
[7]	Zhang Hongqi, Shao Xiaodong, Hu Xiangtao. Dynamic Reliability Assessment Method Based on POMDP for Electromechanical Equipment [J]. China Mechanical Engineering, 2016, 27(18): 2482-2486,2493.
[8]	Zhu Chuanjun, Song Wenjia, Zhang Chaoyong , Cao Jing, Zhu Mengzhou. Research on Optimization of FJSP Based on Maintenance Time Window [J]. China Mechanical Engineering, 2016, 27(10): 1337-1343.
[9]	Zhao Min, Yin Huan, Sun Dihua, Zheng Linjiang, He Wei, Yuan Chuan. Flexible Job Shop Scheduling Based on Modified Artificial Fish Swarm Algorithm [J]. China Mechanical Engineering, 2016, 27(08): 1059-1065.
[10]	Song Wenjia, Zhang Chaoyong, Ying Yong, Shao Xinyu. Integrated Optimization of Equipment Maintenance and Shop Scheduling Problem Based on Multi-objective Hybrid Imperialist Competitive Algorithm [J]. China Mechanical Engineering, 2015, 26(11): 1478-1487.
[11]	Ling Haifeng, Wang Xishan. A Two-stage Parameter Adaptive Ant Colony Algorithm for Flexible Job Shop Scheduling Problem [J]. China Mechanical Engineering, 2013, 24(24): 3380-3385.