A user-centred approach to computer vision-assisted changeover operations

Abstract

This research project aimed to investigate the root causes of frequent mistakes in changeover operations and to design a technical solution to address these issues. The focus of the study was on designing an operator assistance system (OAS) to augment the abilities of factory workers. This project was done in collaboration with a manufacturing company, and a user-centred design approach was used to interact with company representatives and gather feedback on the OAS design. The research process included identifying concrete objectives for the OAS based on three identified problems and the design of a computer vision model and an OAS user interface. The computer vision model was trained using a smaller than usual, and therefore more realistic, amount of data collected by a worker in real factory conditions. We are the first to use a (tiny-) Yolo algorithm for the bolt elongation-based method of detecting loose screws. Using the Tiny Yolo v4 algorithm, which is suitable for local deployment on mobile devices, we reached a mean average precision (mAP) of 100% on our test dataset. The model is thus ready for initial factory deployment as it will not replace human judgement and any additional error detection is beneficial in industry. The OAS user interface was tested for usability through user studies with factory workers. The study's findings showed that workers found the navigation intuitive, thought the features were useful and valued the editability. They also provided recommendations for further improvement. Overall, this research contributes to both industry and research by addressing a pressing issue in manufacturing and supplying a proof-of-concept solution that could improve efficiency and reduce mistakes in changeover operations.