A conceptual model for provider production scheduling in a manufacturing-as-a-service systems using deep reinforcement learning
Submitted: 2025-07-18
|Accepted: 2026-01-30
|Published: 2026-01-31
Copyright (c) 2026 Mateo Del Gallo, Raul Poler, Filippo E. Ciarapica
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Downloads
Keywords:
Manufacturing-as-a-Service, Dynamic Scheduling, Deep Reinforcement Learning, Unrelated parallel machine scheduling
Supporting agencies:
Horizon Europe Framework Programme (HORIZON)
Abstract:
Manufacturing as a Service (MaaS) is an emerging paradigm in which a Provider exposes manufacturing capabilities such as CNC machines, additive manufacturing systems, and other production assets as on-demand services to multiple Consumers. In this setting, Provider side scheduling becomes a critical aspect, as orders arrive dynamically and must be allocated to heterogeneous resources while meeting contractual constraints. This paper proposes an implementable conceptual framework for Provider production scheduling in MaaS, formulating the problem as a Markov Decision Process and enabling Deep Reinforcement Learning based decision making. The proposed Provider Planner models resource allocation as an unrelated parallel machine scheduling problem, where decisions are taken at discrete event-driven decision epochs (e.g., order arrivals and resource releases). The framework explicitly specifies the observation space, action space, and discrete-event transition logic, incorporating practical features such as resource availability and efficiency, operating cost rates, setup states across product families, batch-size constraints, order time windows (earliest/latest start), due dates, and delay penalties. A multi-objective reward formulation is defined to jointly minimise tardiness and associated penalties, overall makespan, and total production cost computed from resource uptimes. The resulting model provides a structured basis for developing and evaluating adaptive scheduling policies for MaaS Providers under demand variability and complex operational constraints.
References:
Alinani, K., Liu, D., Zhou, D., & Wang, G. (2020). Service Composition and Optimal Selection in Cloud Manufacturing: State-of-the-Art and Research Challenges. IEEE Access, 8, 223988–224005. https://doi.org/10.1109/ACCESS.2020.3045008
Al-Mahmud, S., Cano, J. A., Campo, E. A., & Weyers, S. (2025). Optimizing cut order planning: A comparative study of heuris-tics, metaheuristics, and MILP algo-rithms. International Journal of Produc-tion Management and Engineering, 13(1), 1–26. https://doi.org/10.4995/ijpme.2025.22196
Bari, P., Karande, P., & Bag, V. (2024). Hybrid genetic algorithm to minimize scheduling cost with unequal and job dependent ear-liness tardiness cost. International Jour-nal of Production Management and Engi-neering, 12(1), 19–30. https://doi.org/10.4995/ijpme.2024.19277
Borangiu, T., Trentesaux, D., Leitão, P., Cardin, O., & Lamouri, S. (n.d.). Studies in Com-putational Intelligence 952 Service Ori-ented, Holonic and Multi-Agent Manufac-turing Systems for Industry of the Future Proceedings of SOHOMA 2020. http://www.springer.com/series/7092
Cai, J., Lei, D., Wang, J., & Wang, L. (2023). A novel shuffled frog-leaping algorithm with reinforcement learning for distribut-ed assembly hybrid flow shop scheduling. International Journal of Production Re-search, 61(4), 1233–1251. https://doi.org/10.1080/00207543.2022.2031331
Chen, W., Feng, P., Luo, X., & Nie, L. (2024). Task-service matching problem for plat-form-driven manufacturing-as-a-service: A one-leader and multi-follower Stackel-berg game with multiple objectives. Omega (United Kingdom), 129. https://doi.org/10.1016/j.omega.2024.103157
Damodaran, P., Diyadawagamage, D. A., Ghra-yeb, O., & Vélez-Gallego, M. C. (2012). A particle swarm optimization algorithm for minimizing makespan of nonidentical parallel batch processing machines. In-ternational Journal of Advanced Manu-facturing Technology, 58(9–12), 1131–1140. https://doi.org/10.1007/S00170-011-3442-Z
Del Gallo, M., Antomarioni, S., Mazzuto, G., Marcucci, G., & Ciarapica, F. E. (2024). A self-learning framework combining as-sociation rules and mathematical models to solve production scheduling programs. Production and Manufacturing Research, 12(1). https://doi.org/10.1080/21693277.2024.2332285
Del Gallo, M., Mazzuto, G., Ciarapica, F. E., & Bevilacqua, M. (2023). Artificial Intelli-gence to Solve Production Scheduling Problems in Real Industrial Settings: Sys-tematic Literature Review. In Electronics (Switzerland) (Vol. 12, Issue 23). Multi-disciplinary Digital Publishing Institute (MDPI). https://doi.org/10.3390/electronics12234732
Duran, E., Ozturk, C., & O’Sullivan, B. (2024). Planning and scheduling shared manufac-turing systems: key characteristics, cur-rent developments and future trends. In International Journal of Production Re-search. Taylor and Francis Ltd. https://doi.org/10.1080/00207543.2024.2442549
Dutra, D., Castelhano De Oliveira, V., & Silva, J. R. (2013). Manufacturing as Service: the challenge of Intelligent Manufactur-ing. http://www.sparxsystems.com.au/
Fisher, O., Watson, N., Porcu, L., Bacon, D., Rigley, M., & Gomes, R. L. (2018). Cloud manufacturing as a sustainable process manufacturing route. Journal of Manufacturing Systems, 47, 53–68. https://doi.org/10.1016/j.jmsy.2018.03.005
Ford, S., Rauschecker, U., & Athanassopoulou, N. (2012). System-of-system approaches and challenges for multi-site manufactur-ing. Proceedings - 2012 7th International Conference on System of Systems Engi-neering, SoSE 2012, 543–548. https://doi.org/10.1109/SYSoSE.2012.6384164
Garey, M. R., Johnson, D. S., & Sethi, R. (1976). The Complexity of Flowshop and Jobshop Scheduling. In Source: Mathe-matics of Operations Research (Vol. 1, Issue 2).
Gil, C. B., & Lee, J. H. (2022). Deep Rein-forcement Learning Approach for Mate-rial Scheduling Considering High-Dimensional Environment of Hybrid Flow-Shop Problem. Applied Sciences (Switzerland), 12(18). https://doi.org/10.3390/app12189332
Goldhar, J. D., & Jelinek, M. (1990). Manufac-turing as a service business: CIM in the 21st century. Computers in Industry, 14(1–3), 225–245. https://doi.org/10.1016/0166-3615(90)90126-A
Hu, Y., Pan, L., & Pan, X. (2024). Dynamic scheduling of workshop resource in cloud manufacturing environment. Engineering Applications of Artificial Intelligence, 138. https://doi.org/10.1016/j.engappai.2024.109405
Hu, Y., Zhu, F., Zhang, L., Lui, Y., & Wang, Z. (2019). Scheduling of manufacturers based on chaos optimization algorithm in cloud manufacturing. Robotics and Com-puter-Integrated Manufacturing, 58, 13–20. https://doi.org/10.1016/j.rcim.2019.01.010
Julaiti, J., Oh, S. C., Das, D., & Kumara, S. (2022). Stochastic parallel machine scheduling using reinforcement learning. Journal of Advanced Manufacturing and Processing, 4(4). https://doi.org/10.1002/amp2.10119
Karamanli, A., Xanthopoulos, A., Gasteratos, A., & Koulouriotis, D. (2025a). A Bibli-ometric and Systematic Review of Manu-facturing-as-a-Service: Literature In-sights, Challenges, and Future Trends. In Applied Sciences (Switzerland) (Vol. 15, Issue 5). Multidisciplinary Digital Pub-lishing Institute (MDPI). https://doi.org/10.3390/app15052440
Karamanli, A., Xanthopoulos, A., Gasteratos, A., & Koulouriotis, D. (2025b). A Bibli-ometric and Systematic Review of Manu-facturing-as-a-Service: Literature In-sights, Challenges, and Future Trends. In Applied Sciences (Switzerland) (Vol. 15, Issue 5). Multidisciplinary Digital Pub-lishing Institute (MDPI). https://doi.org/10.3390/app15052440
Liu, Y., Fan, J., Zhao, L., Shen, W., & Zhang, C. (2023). Integration of deep reinforce-ment learning and multi-agent system for dynamic scheduling of re-entrant hybrid flow shop considering worker fatigue and skill levels. Robotics and Computer-Integrated Manufacturing, 84, 102605. https://doi.org/10.1016/J.RCIM.2023.102605
Mahmoodi, E., & Fathi, M. (2024). Policy Making to Encourage Platform Thinking in Manufacturing: A System Dynamics-based Multi-Objective Optimization Ap-proach. IFAC-PapersOnLine, 58(27), 1164–1169. https://doi.org/10.1016/j.procir.2024.10.222
Namjoshi, J., & Rawat, M. (2022). Role of smart manufacturing in industry 4.0. Ma-terials Today: Proceedings, 63, 475–478. https://doi.org/10.1016/j.matpr.2022.03.620
Nicoletti, L., Solina, V., Amin, K., Lessi, C., McHard, P., Qiu, R., & Tedeschi, S. (2024). Exploiting Extended Reality un-der the Manufacturing as a Service para-digm. Procedia Computer Science, 232, 2213–2219. https://doi.org/10.1016/j.procs.2024.02.040
Pan, Z., Wang, L., Wang, J., & Lu, J. (2023). Deep Reinforcement Learning Based Op-timization Algorithm for Permutation Flow-Shop Scheduling. IEEE Transac-tions on Emerging Topics in Computa-tional Intelligence, 7(4), 983–994. https://doi.org/10.1109/TETCI.2021.3098354
Pietrangeli, I., Mazzuto, G., Ciarapica, F. E., Bevilacqua, M., & Ortenzi, M. (2024). Smart Retrofit Solution: An Architecture for Digital Innovation. Proceedings of the 30th ICE IEEE/ITMC Conference on En-gineering, Technology, and Innovation: Digital Transformation on Engineering, Technology and Innovation, ICE 2024. https://doi.org/10.1109/ICE/ITMC61926.2024.10794309
Prata, B. de A. (2025). An overview of industri-al engineering and operations manage-ment over the first fifty years of Engi-neering Optimization. In Engineering Op-timization. Taylor and Francis Ltd. https://doi.org/10.1080/0305215X.2024.2423181
Said, N. E. D. A., Samaha, Y., Azab, E., Shihata, L. A., & Mashaly, M. (2021). An Online Reinforcement Learning Ap-proach for Solving the Dynamic Flexible Job-Shop Scheduling Problem for Multi-ple Products and Constraints. Proceed-ings - 2021 International Conference on Computational Science and Computa-tional Intelligence, CSCI 2021, 134–139. https://doi.org/10.1109/CSCI54926.2021.00095
Serrano-Ruiz, J. C., Mula, J., & Poler, R. (2021). Smart manufacturing scheduling: A literature review. In Journal of Manu-facturing Systems (Vol. 61, pp. 265–287). Elsevier B.V. https://doi.org/10.1016/j.jmsy.2021.09.011
Song, W., Chen, X., Li, Q., & Cao, Z. (2023). Flexible Job-Shop Scheduling via Graph Neural Network and Deep Reinforcement Learning. IEEE Transactions on Industri-al Informatics, 19(2), 1600–1610. https://doi.org/10.1109/TII.2022.3189725
Vatankhah Barenji, A., Li, Z., & Wang, W. M. (2018). Smart SysTech 2018 : European Conference on Smart Objects, Systems, and Technologies, June 12-13, 2018, Fraunhofer Institute for Photonic Mi-crosystems (IPMS) in Dresden, Germany. VDE Verlag GmbH.
Wu, X., Yan, X., Guan, D., & Wei, M. (2024). A deep reinforcement learning model for dynamic job-shop scheduling problem with uncertain processing time. Engineer-ing Applications of Artificial Intelligence, 131. https://doi.org/10.1016/j.engappai.2023.107790
Xu, C., Yu, H., Jin, X., Xia, C., Li, D., & Zeng, P. (2024). Industrial Internet for intelli-gent manufacturing: past, present, and fu-ture. In Frontiers of Information Tech-nology and Electronic Engineering (Vol. 25, Issue 9, pp. 1173–1192). Zhejiang University. https://doi.org/10.1631/FITEE.2300806
Zhang, M., Wang, L., Qiu, F., & Liu, X. (2023). Dynamic scheduling for flexible job shop with insufficient transportation resources via graph neural network and deep rein-forcement learning. Computers and In-dustrial Engineering, 186. https://doi.org/10.1016/j.cie.2023.109718
Zhang, W., & Diettench, T. G. (1995). A Rein-forcement Learning Approach to Job-shop Scheduling.
Zhang, X., & Zhu, G. Y. (2025). A literature review of reinforcement learning meth-ods applied to job-shop scheduling prob-lems. In Computers and Operations Re-search (Vol. 175). Elsevier Ltd. https://doi.org/10.1016/j.cor.2024.106929
