Factory of the future? (AERTEC Solutions)

Gesture controls and augmented reality are making theleap from consumer electronic devices and video gaming to the aerospace factory in a pioneering project in the UK. TIM ROBINSON reports.

Perhaps the most memorable scene in the 2002 science-fiction film ‘Minority Report’ was in its depiction  of a future gesture-controlled computer  interface — with Tom Cruise using his hands to perform a data search, waving his hands like a orchestra conductor in front of a giant display to find information.

Twelve years later in 2014, gesture controls are now becoming mainstream, with smart phones and televisions that can be controlled by ‘swiping’ or pointing your hands. Perhaps the biggest application has been in consumer video games, where the Nintendo Wii, Microsoft Xbox Kinect and Playstation Eye have all brought movement or gesture-controlled games to the living room. By tracking the position of a controller precisely in 3D space (or in the Kinect’s case the person itself by tracking the position of the body and limbs) the game can use the input of this to replicate it in a virtual environment — allowing players to play golf, tennis or bowling for example, by using the same movements they would in the real game.

So far, so fun — but outside gesture controls for passenger IFE — is there any real relevance for this kind of technology for aerospace?

Into the factory    

Microsoft Kinect camera view of a 'player'. The sensor tracks skeleton and body movements. (Wikipedia)

Surprisingly there is — and it is not inside the flight deck — but on the factory floor. Under a project that partners Airbus Group and Cranfield University with Spanish engineering and aviation infrastructure consultancy AERTEC Solutions, researchers are investigating whether low-cost gaming technology can be used to pass on aerospace manufacturing knowledge and skills. The Digitising Expressions for Manufacturing Optimisation (DEMO) is a 30 month project that started in October 2012 which ‘seeks to leverage commercial gaming interface technology (Microsoft Kinect) to capture and re-use human knowledge of complex manufacturing processes that use deformable materials.’ As its first focus, DEMO is examining laminate lay-up processes in composites manufacturing and is set to run until the middle of next year. Says the Director of AERTEC’s new Bristol office, Pedro de Melo: “It's about finding novel ways to apply gaming technology to an industrial context”.

Simon Astwood, Research Team Leader — Digital Factory, Airbus Group Innovations (formerly EADS Innovation Works) explains more about DEMO: “The philosophy behind the project was to take this low-cost but quite advanced technology available in the gaming industry and see if we can apply it in quite complex tasks in industrial manufacturing.” He went on: “The gaming world has invested heavily in the infrastructure, the software and the technology to make these kind of games happen. In the Airbus Group we end up spending a lot of money developing very expensive bespoke solutions. Sometimes there is a crossover between the two technologies — and sometimes you can apply something from gaming at a very low cost that gets very good results.”

Motion or gesture controllers, it turns out can be a way of passing on key worker skills. The problem is that many skills, acquired though years of experience are difficult or impossible to write down. The result is then, for global companies like Airbus and others that much time and money is expended in attempting to train and pass on these skills — especially to newer companies of the supply chain in emerging economies. That can necessitate a lot of travel to teach and pass on these ‘unwritten’ skills to less experienced workers. For example, experienced workers hand-laying up composites will work from the centre of the material first, then work towards the edges in one design, yet work from left to right in another to get the best finish.

Astwood elaborates: “The real value comes when you take something that is very highly skilled labour. If you take the ‘black art’ of composite lay-up although Airbus is moving towards mass automation, a lot of components are still made by hand. With a move towards using low-cost production in places like China, there is a need to transfer some skills from typically highly skilled plants to these new plants, that don’t have the 20-years of laminating experience of Spanish workers”. Enter gesture-control technology — or, more specifically, Microsoft’s Kinect. By ‘recording’ the positions and actions of experienced workers doing skilled manufactured tasks (for example deformable composite layup) these can be stored and transmitted to a factory on the other side of the world. The Kinect software which, as well as recording the positions of limbs and body, also allows easy sharing — allowing players to dance or play tennis against each other in real time — also allows with broadband internet these ‘moves’ to be shared with others with the device. “What we are able to do with the Kinect”, says Astwood: “is watch somebody in Spain carrying out that task and then relay that to somebody in China, either as a training tool or actually watching them in their workspace”.

Playing the DEMO

Airbus Futurassy robot. Could Kinect one day be used to teach robots complex skills?  (Airbus)

Says Astwood: “Our first demonstration involved performing some composite repairs and we used a normal low-cost desktop projector. We mounted the projector above the workbench and every time they moved their hands around on the job, depending on where they moved their hands, it projected additional information — such as what orientation the material needs to be lined up in. There was also a notes page at the side that gave helpful tips”.

Thus, in the factory of the future, a new worker with TV screen and a Kinect device may not only ‘see’ the correct movements of arms and limbs but also when working the computer may tell them whether they are matching the original ‘recording’ of the highly skilled worker. Like a video gamer playing a ‘rhythm dance’ game step-by-step, the technician, over time would able to build up muscle memory in learning a new skill. Says Astwood, “What we are trying to do is to build ways of augmenting the workspace of one of the Chinese workers with lots of additional information captured by the Spanish workers”.

Astwood explains: “Our first step was to use a projector to throw in some additional information using the Kinect. Now we are looking to going into a gaming platform, to take the knowledge and learning from that initial experience and looking at building a computer game simulator — so someone could actually go through the motions of carrying out a task in front of a projector, using hand gestures to select parts and bring them into the workspace.”

The benefits of this are - as well as saving money through travel — as a non-verbal mode on communication it allows the nuances of skilled tasks to be passed on — without needing extensive translation. “Using graphics, icons and gestures there’s no real need to be able to write anything in Spanish or Mandarin” observes Astwood. Another benefit of this is increased health and safety for workers – wherever they are. Finally, by capturing this data non-intrusively it may give insights into better workflow or optimisation of manufacturing processes that simply wasn’t available before — potentially leading to increased efficiency and reduced cost.

Further in the future, it may also aid in automation of factories. In February 2014, Airbus launched the Futurassy R&D project where  humanoid robots will work alongside humans in aircraft assembly. Could the ‘digitized skills’ of experienced human aerospace workers be passed on to humanoid robots using this technology?

Google Glass on the factory floor

Google Glass display view. The eyepiece projects information which can include images and video into the top-right corner. (Google)

Meanwhile, another consumer electronic device may also be heading for the aircraft factory floor – that of a wearable personal HUD — Google Glass. Head mounted displays of course, are nothing new for the aerospace industry — having been pioneered by helmet-mounted sights for fighter and attack helicopter pilots. However, now the technology has come full circle with the advent of the first consumer wearable head-mounted displays about to break into the market. Foremost among these is Google Glass — a personal ‘HUD’ that displays information from a linked smartphone such as directions, as well as being equipped with a video and still camera.

The result, when paired with accurate geo-location software and the smart-phones inbuilt GPS, is ‘augmented reality’ where the ‘HUD’ can overlay driving directions, points of interest or other information in 3D space. In the factory, this could be used so engineers could see CAD and CATIA diagrams ‘overlaid’ on the aircrafts structure, or see a real fuselage become a ‘virtual cutaway’ model with wiring, hydraulics, avionics and other systems highlighted beneath the skin when the user looks at the that direction. Says Astwood “Ultimately we are looking at things like Google Glass so that someone on the shop floor, has access to all the CAD data, work instructions all geo-located in their workspace”.

The technology, when paired with ‘smart tools’ or RFID tracking could also allow supervisors to find lost or missing tools – or optimise work flows by ‘seeing’ the way in which tools are passed on – wherever they may be in the factory. “If you looked at a spanner and you didn’t know what it was for — the Glass would tell you” notes Astwood.


Other serious uses for the Kinect gesture control technology include rehabilitation for stroke or brain injury patients. (Microsoft).

In short, although these are early days, there is much potential using this technology to optimise and transfer skills in the aerospace supply chain using such low-cost technology from video gaming. With civil aerospace manufacturers struggling to keep up with demand, yet constrained by the need to maintain quality with new emerging partners this is one way of bridging the gap. Finally the massive adoption of the iPad by the aerospace and aviation sector demonstrates that the industry is not reluctant to adopt consumer technology where necessary. Will the personal HUD of Google Glass be the next success?

3 May 2014