面向柔性作业车间生产调度的深度强化学习方法

doi:10.3969/j.issn.1004-132X.2024.11.012

摘要/Abstract

摘要： 针对多品种、小批量生产模式下柔性作业车间生产调度问题，以最小化订单总拖期时间为优化目标，提出一种基于组合规则和强化学习的智能调度方法。将柔性作业车间生产调度问题转换为马尔可夫决策过程，根据问题特点与优化目标，利用7种特征表征车间状态，设计6种组合式规则作为动作库，通过改进后的深度Q网络（DQN）算法对该问题进行求解。以航天结构件加工车间为案例，分别在5种不同规模大小的算例中，与其他常见的规则式方法进行对比，验证了所提方法缩短任务交付期的可行性和有效性。

关键词: 生产调度, 柔性作业车间, 深度强化学习, 深度Q网络

Abstract: Aiming at the flexible job shop scheduling problems under the mode of multi variety and small batch production, an intelligent scheduling method was proposed to minimize the total tardiness of orders based on combination rules and reinforcement learning. Transforming the flexible job shop production scheduling problem into a Markov decision process, according to the characteristics and optimization objectives of the problems, seven features were used to represent the workshop states, and six combination rules were designed as an action library. The problem was solved by using the improved DQN algorithm. Taking the aerospace structural parts machining workshop as a case study, the feasibility and effectiveness of the proposed method in shortening task delivery time are verified by comparing with other common rule-based methods in five different scale calculation examples.

Key words: production scheduling, flexible job-shop, reinforcement learning, deep Q-network（DQN）

中图分类号:

TP18

祝正宇1, 郭具涛2, 吕佑龙3, 左丽玲1, 张洁3. 面向柔性作业车间生产调度的深度强化学习方法[J]. 中国机械工程, 2024, 35(11): 2007-2014，2034.

ZHU Zhengyu1, GUO Jutao2, LYU Youlong3, ZUO Liling1, ZHANG Jie3. Deep Reinforcement Learning Method for Flexible Job Shop Scheduling[J]. China Mechanical Engineering, 2024, 35(11): 2007-2014，2034.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: http://www.cmemo.org.cn/CN/10.3969/j.issn.1004-132X.2024.11.012

http://www.cmemo.org.cn/CN/Y2024/V35/I11/2007

参考文献

［1］姜一啸, 吉卫喜, 何鑫, 等. 基于改进非支配排序遗传算法的多目标柔性作业车间低碳调度［J］. 中国机械工程, 2022, 33(21):2564-2577.
JIANG Yixiao, JI Weixi, HE Xin, et al. Low-carbon Scheduling of Multi-objective Flexible Job-shop Based on Improved NSGA-Ⅱ［J］. China Mechanical Engineering, 2022, 33(21):2564-2577.
［2］PARK J, CHUN J, KIM S H, et al. Learning to Schedule Job-shop Problems:Representation and Policy Learning Using Graph Neural Network and Reinforcement Learning［J］. International Journal of Production Research, 2021, 59(11):3360-3377.
［3］马雪梅, 胡良元, 沈艳波, 等. 我国航天工业能力布局回顾与展望［J］. 航天工业管理, 2019(10):75-80.
MA Xuemei, HU Liangyuan, SHEN Yanbo, et al.Review and Prospect of Chinas Aerospace Industry Capability Layout［J］.Aerospace Industry Management,2019(10):75-80.
［4］李新宇, 黄江平, 李嘉航, 等. 智能车间动态调度的研究与发展趋势分析［J］. 中国科学:技术科学, 2023, 53(7):1016-1030.
LI Xinyu, HUANG Jiangping, LI Jiahang, et al. Research and Development Trend of Intelligent Shop Dynamic Scheduling［J］. Chinese Science:Technology Science, 2023, 53(7):1016-1030.
［5］乔东平, 裴杰, 文笑雨, 等. 一种求解单机总加权延迟调度问题的改进蚁群算法［J］. 中国机械工程, 2018, 29(22):2703-2710.
QIAO Dongping, PEI Jie, WEN Xiaoyu, et al. An Improved Ant Colony Algorithm for Solving Single Machine Total Weighted Delay Scheduling Problem［J］. China Mechanical Engineering, 2018, 29(22):2703-2710.
［6］黄学文, 陈绍芬, 周阗玉, 等. 求解柔性作业车间调度的遗传算法综述［J］. 计算机集成制造系统, 2022, 28(2):536-551.
HUANG Xuewen, CHEN Shaofen, ZHOU Tianyu, et al. Survey on Genetic Algorithms for Solving Flexible Job-shop Scheduling Problem［J］.Computer Integrated Manufacturing Systems,2022,28(2):536-551.
［7］CHEN B, MATIS T. A Flexible Dispatching Rule for Minimizing Tardiness in Job Shop Scheduling［J］. International Journal of Production Economics, 2013, 141(1):360-365.
［8］李益兵, 黄炜星, 吴锐. 基于改进人工蜂群算法的多目标绿色柔性作业车间调度研究［J］. 中国机械工程, 2020, 31(11):1344-1350.
LI Yibing, HUANG Weixing, WU Rui. Research on Multi-objective Green Flexible Job-shop Scheduling Based on Improved ABC Algorithm［J］. China Mechanical Engineering, 2020, 31(11):1344-1350.
［9］ZHAO N, YE S, LI K, et al. Effective Iterated Greedy Algorithm for Flow-shop Scheduling Problems with Time Lags［J］. Chinese Journal of Mechanical Engineering, 2017, 30(3):652-662.
［10］王秋莲, 段星皓. 基于高维多目标候鸟优化算法的柔性作业车间调度［J］. 中国机械工程, 2022, 33(21):2601-2612.
WANG Qiulian, DUAN Xinghao. Scheduling of Flexible Job Shop Based on High-dimension and Multi-objective Migrating Bird Optimization Algorithm［J］. China Mechanical Engineering, 2022, 33(21):2601-2612.
［11］DU Y, LI J, LI C, et al. A Reinforcement Learning Approach for Flexible Job Shop Scheduling Problem with Crane Transportation and Setup Times［J］. IEEE Transactions on Neural Networks and Learning Systems, 2022, 35(4):5695-5709.
［12］贺俊杰, 张洁, 张朋, 等. 基于长短期记忆近端策略优化强化学习的等效并行机在线调度方法［J］. 中国机械工程, 2022, 33(3):329-338.
HE Junjie, ZHANG Jie, ZHANG Peng, et al. Related Parallel Machine Online Scheduling Method Based on LSTM-PPO Reinforcement Learning［J］. China Mechanical Engineering, 2022, 33(3):329-338.
［13］XUE T, ZENG P, YU H. A Reinforcement Learning Method for Multi-AGV Scheduling in Manufacturing［C］∥2018 IEEE International Conference on Industrial Technology(ICIT). Lyon, 2018:1557-1561.
［14］WEI Y, PAN L, LIU S, et al. DRL-scheduling:an Intelligent QoS-aware Job Scheduling Framework for Applications in Clouds［J］. IEEE Access, 2018, 6:55112-55125.
［15］钟敬伟, 石宇强. 基于DQN的智能工厂作业车间调度［J］. 现代制造工程, 2021(9):17-23.
ZHONG Jingwei, SHI Yuqiang. Job Shop Scheduling Based on DQN Algorithm in Intelligent Factory［J］.Modern Manufacturing Engineering,2021(9):17-23.
［16］HE Z, KIM P T, SEBASTIEN T, et al. Multi-objective Optimization of the Textile Manufacturing Process Using Deep-Q-network Based Multi-agent Reinforcement Learning［J］. Journal of Manufacturing Systems, 2022,62:939-949.
［17］LUO S. Dynamic Scheduling for Flexible Job Shop with New Job Insertions by Deep Reinforcement Learning［J］. Applied Soft Computing, 2020, 91:106208.
［18］WANG Y, LIU H, ZHENG W, et al. Multi-objective Workflow Scheduling with Deep-Q-network-based Multi-agent Reinforcement Learning［J］. IEEE Access, 2019, 7:39974-39982.
［19］刘亚辉, 申兴旺, 顾星海, 等. 面向柔性作业车间动态调度的双系统强化学习方法［J］. 上海交通大学学报, 2022, 56(9):1262-1275.
LIU Yahui, SHEN Xingwang, GU Xinghai, et al.A Dual-system Reinforcement Learning Approach for Dynamic Scheduling of Flexible Job Shops［J］.Journal of Shanghai Jiao Tong University,2022,56(9):1262-1275.
［20］张凯, 毕利, 焦小刚. 集成强化学习算法的柔性作业车间调度问题研究［J］. 中国机械工程, 2023, 34(2):201-207.ZHANG Kai, BI Li, JIAO Xiaogang. Research on Flexible Job-shop Scheduling Problems with Integrated Reinforcement Learning Algorithm［J］. China Mechanical Engineering, 2023, 34(2):201-207.
［21］胡一凡, 张利平, 白雪, 等. 深度强化学习求解柔性装配作业车间调度问题［J］. 华中科技大学学报, 2023, 51(2):153-160.
HU Yifan, ZHANG Liping, BAI Xue, et al. Deep Reinforcement Learning for Solving Flexible Assembly Workshop Scheduling Problem［J］. Journal of Huazhong University of Science and Technology, 2023, 51(2):153-160.
［22］VAN HASSELT H, GUEZ A, SILVER D. Deep Reinforcement Learning with Double Q-learning［C］∥Proceedings of the AAAI Conference on Artificial Intelligence. Phoenix, 2016:2094-2100.
［23］LIU Peng, XIA Boyuan, YANG Zhiwei, et al. A Deep Reinforcement Learning Method for Multi-stage Equipment Development Planning in Uncertain Environments［J］. Journal of Systems Engineering and Electronics, 2022, 33(6):1159-1175.

[1]	钟意, 杨勇, 姜学涛, 潘顺洋, 朱其新, 王磊. 基于多智能体深度Q网络交互的板壳加强筋生长式设计[J]. 中国机械工程, 2024, 35(08): 1397-1404.
[2]	魏书鹏, 唐红涛, 李西兴, 杨冠宇, 张健. 考虑双资源约束的柔性机械加工车间逆调度问题研究[J]. 中国机械工程, 2024, 35(03): 457-471.
[3]	王洪, 吴立辉, 陈达, 张洁. 基于孪生人工蜂鸟算法的多作业模式半导体封测环节调度[J]. 中国机械工程, 2024, 35(02): 260-267，279.
[4]	郭具涛, 吕佑龙, 戴铮, 张洁, 郭宇. 基于复合规则和强化学习的混流装配线调度方法[J]. 中国机械工程, 2023, 34(21): 2600-2606,2614.
[5]	梁向檩, 宋豫川, 雷琦, 孙爱红. 考虑工人数量配置优化的柔性作业车间调度问题研究[J]. 中国机械工程, 2023, 34(17): 2065-2076.
[6]	李西兴, 周文龙, 唐红涛, 吴锐, . 考虑设备预维护的特种车辆白车身试制车间绿色调度优化方法[J]. 中国机械工程, 2023, 34(15): 1832-1847.
[7]	张洪亮, 徐公杰, 鲍蔷, 潘瑞林. 考虑运输时间的分布式柔性作业车间绿色调度[J]. 中国机械工程, 2022, 33(21): 2554-2563,2645.
[8]	姜一啸, 吉卫喜, 何鑫, 苏璇. 基于改进非支配排序遗传算法的多目标柔性作业车间低碳调度[J]. 中国机械工程, 2022, 33(21): 2564-2577.
[9]	孙爱红, 宋豫川, 杨云帆, 雷琦. 考虑关键件加工质量的双资源约束车间调度算法[J]. 中国机械工程, 2022, 33(21): 2590-2600.
[10]	王秋莲, 段星皓. 基于高维多目标候鸟优化算法的柔性作业车间调度[J]. 中国机械工程, 2022, 33(21): 2601-2612.
[11]	邹裕吉, 宋豫川, 王馨坤, 王毅. 自动导向小车与加工设备多目标集成调度的聚类遗传算法[J]. 中国机械工程, 2022, 33(01): 97-108.
[12]	易茜, 何爽, 宁轻, 李聪波, 易树平. 汽车试制车间考虑员工作业能力的多目标优化生产调度[J]. 中国机械工程, 2021, 32(13): 1617-1629.
[13]	李西兴, 杨道明, 李鑫, 吴锐, . 基于混合遗传鲸鱼优化算法的柔性作业车间自动导引车融合调度方法[J]. 中国机械工程, 2021, 32(08): 938-950，986.
[14]	李益兵1,2;黄炜星1;吴锐1. 基于改进人工蜂群算法的多目标绿色柔性作业车间调度研究[J]. 中国机械工程, 2020, 31(11): 1344-1350,1385.
[15]	刘彩洁;徐志涛;张钦;张力菠;姚坤. 分时电价下基于NSGA-Ⅱ的柔性作业车间绿色调度[J]. 中国机械工程, 2020, 31(05): 576-585.