Ongoing developments in digitisation open up new opportunities in the logistics sector too. Fast response times, and consequently short system downtimes, are an important success factor in intralogistics. Gilgen Logistics has taken a further successful step in this direction with its “Augmented Reality” pilot project. Customers, service technicians and fitters can be given visual instructions and support in any location with the help of cutting-edge aids.
Digitisation in the working world has become a matter of course for many companies. It is already well advanced in the planning and engineering of logistics systems. The demands on cycle and response times as well as the use of sophisticated technologies have intensified in recent years. Augmented reality is primarily used in warehouses for inventories and picking. It is as yet less commonly used for the efficient set-up and operational maintenance of logistics systems. If, for example, the installation progress is delayed, this can have serious consequences for the project lead time; a system standstill and idle times lead to lost turnover and extra costs. The manufacturer has to intervene as quickly as possible. Specialist know-how is needed in many cases. Unfortunately, the specialists are often far away, so that valuable time is lost. This is a significant cost factor with globally installed systems and may tie up valuable resources for several days. If we also bear in mind the fact that the problem can often be solved by a specialist in next to no time, we quickly realise just how big the influence of travelling time is on the downtime or cycle time.
The question thus arises as to how fitters, service technicians, commissioning teams and customers can be given fast and competent support on site using the know-how of specialists. How can unnecessary service calls and empty trips be avoided? The use of augmented reality (AR) appears to show great promise here. This prompted Gilgen Logistics to launch a pilot project.
A number of applications already use mixed-reality headsets or smart phones for some basic functions. Live image transmissions have become an economical option. But they are not user-friendly and are rather impractical at the plant, where every hand is needed for work. “An augmented reality solution can satisfy the needs of both the manufacturer and customer,” explains Daniel Fricker, Head of Developments at Gilgen Logistics, and sees numerous application cases: “AR can be used in the development, construction, installation and operation of the system, for example.”
This has the following advantages: Gilgen service technicians can consult a specialist directly if any problems arise, either during the installation, if more detailed documents are needed, with changing ambient conditions or for necessary decisions. The Gilgen service department can assess the situation on site shortly after a support ticket has been received and take appropriate action. A specialist can supervise a complicated repair without having to be there in person. The customer can receive remote support for service work or even carry out repairs on their own by following the specialist’s instructions.
Gilgen Logistics found valuable partners for the development in the Institute for Information Systems of the University of Applied Sciences and Arts Northwestern Switzerland and the Computer Perception & Virtual Reality Lab of the Institute for Human Centred Engineering of the Bern University of Applied Sciences. The project was also supported by Innosuisse, the Swiss Innovation Agency.
Since the basic functions of a mixed-reality headset are already in general use, the team focussed on marking and animating components of the elements. The supporter hereby marks those components of the virtual 3D model that are to be highlighted in colour. The operator can then be shown animations about the assembly or installation sequence, position or setting/adjustment.
The operator is on site and experiencing problems with the system that cannot be solved without the help of a specialist. They therefore put on an AR headset, connecting them to a supporter via the microphone and Voice over IP. The supporter can see the live image from the AR headset’s camera on their own monitor. The headset can identify the type of element by means of a marker and downloads the corresponding 3D data from the Gilgen Logistics data server. The element is then augmented, i.e. the virtual 3D model is superimposed over the real element. The supporter sees the same 3D model on their PC and can now mark, show and hide individual faulty components for the operator and guide them through the repair, installation or troubleshooting process. All of this time, the operator has both their hands free for work.
A number of relatively new and complex IT technologies had to be combined for this pilot project:
Several network servers were needed to exchange the data. On the one hand, this guarantees a continuous flow of data for the 3D synchronisation. Because this exchange has to work with a very low latency, a software was used that can also be found in multiplayer online games. On the other hand, the two clients (HoloLens and browser app) had to be linked.
A technology that is familiar from video conference tools was used for real-time communication. This allows the web browser and the HoloLens to exchange video and audio streams directly. The HoloLens is hereby connected to the Internet via WLAN.
Gilgen uses the HoloLens headset from Microsoft for the application. This AR headset shows the wearer 3D objects that are not really there in the room. This takes place stereoscopically (separately for both eyes) and in real-time. The most important aspect here is that the headset can determine the wearer’s position and orientation, and thus their line of vision, very quickly and precisely. The HoloLens detects the room with a 3D depth camera and infra-red sensors. The Unity game engine is used to generate the 3D graphics. Gilgen machines and their exact positions are identified with the HoloLens by means of a QR code.
The 3D objects that are shown in the HoloLens also appear in the supporter’s browser. The supporter can also rotate the 3D model in any direction. The most important innovation in this project was that the supporter can now select and highlight a part that is causing problems in their 3D image of the machine. The selected part is transmitted to the HoloLens and the supporter can give the operator precise instructions, just as if they were standing next to them.
“This project has shown us that augmented reality in intralogistics has a great potential for improving efficiency,” concludes Daniel Fricker, Head of Developments. “The use of augmented reality improves the safety and trust of customers in using the system. This pilot project has laid the foundations for further projects, such as developing a network link between the system and the Gilgen support department, bringing us closer to our goal of the ‘efficient processing of realisation projects and the best possible customer support’.”