Integrating gesture recognition in airplane seats for in-flight entertainment

Rick Westelaken, Jun Hu, Hao Liu, Matthias Rauterberg

Technische Universiteit Eindhoven,
Den Dolech 2 5600MB Eindhoven, The Netherlands
H.F.M.v.d.Westelaken@student.tue.nl, {J.Hu, hao.liu, G.W.M.Rauterberg}@tue.nl

Abstract.In order to reduce both the psychological and physical stress in air travel, sensors are integrated in airplane seats to detect the gestures as input for in-flight entertainment systems. The content provided by the entertainment systems helps to reduce the psychological stress, and the gesture recognition is used as input for the content and hence stimulates people to move, which as a result would reduce the physical stress as well.

Keywords: In-flight entertainment, gesture recognition, air travel, air travel thrombosis.

1 Introduction

Today a large number of people use air travel as a way of transportation. This number is increasing every year, as well as the number of long-haul flights. At the same time the flight duration is increased because of better fuel efficient airplanes, which makes intermediate landings unnecessary. The air travel market is highly competitive and therefore airlines try to maximize the number of seats [1]. Often this results in a very limited amount of seating space for passengers, especially in economy class [2]. In this context the EU project “SEAT” (Smart tEchnologies for stress free Air Travel) was set-up [3]. This project, sponsored by a European commission focuses on improvements and solutions to increase comfort in air travel. The partners in this project are Imperial College London, Acusttel, Aitex, Antecuir, Czech Technical University, Wearable Computing Lab ETH Zürich, Inescop, Queen Mary University of London, Starlab, Eindhoven University of Technology, Thales and DHS. The work described here is part of the SEAT project and is executed in Eindhoven University of Technology.

The combination of long flight duration, limited space and an unusual cabin environment in terms of air pressure, humidity and continues noise causes physical and psychological discomfort for a large group of passengers as well as the crew [4]. Most of the airlines offer in-flight entertainment to their passengers. This provides mental distraction and might lead to reduction of psychological stress. To reduce physical stress some airlines recommend passengers in-flight exercises[2, 5]. The ways these exercises are presented vary from paper versions to complete instruction videos and crew demonstrations. The main goal of these exercises is to stimulate the blood flow and prevent health related issues like deep vein thrombosis (DVT), stiffness and fatigue. The problem with these recommendations is that it depends on the passenger whether the exercises are executed. Another way to prevent DVT is to wear compression stocking to prevent formation of DVT [6]. Ordinary this is only used by passengers with increased risk of DVT, which means that only a relative small group of people use this means.

Physical stress can cause a number of health related problems, as shown by different institutes like the World Health Organization [7] and the British Medical Association [8]. Therefore this work focuses on how passengers can be motivated to move during long-haul flights in order to reduce physical stress. The solution discussed here is an airplane chair equipped with sensors to detect the body movements and gestures of passengers. The detected gestures could than be used as input for interactive applications in in-flight entertainment systems.

The idea of controlling interactive content by means of body movement is not new. In contrary it is very popular at the moment. Examples of movement controlled game platforms are the Nintendo Wii and Sony Eye-Toy. Although there is a big difference between these platforms, these platforms assume that the user has enough space in which they can move. This assumption does not hold in an economy class flight environment. There the amount of space is very limited, which means that new solutions had to be explored.

2 Problem and context

Prolonged immobility by sitting in a chair during long haul flights can lead to pooling of blood in the legs. It is known that immobility cause formation of blood clots in the body and especially in deep veins (Deep Vein Thrombosis, or DVT in short). “Larger clots may cause symptoms such as swelling of the leg, tenderness, soreness and pain. Occasionally a piece of the clot may break off and travel with the bloodstream to become lodged in the lungs. This is known as pulmonary embolism and may cause chest pain, shortness of breath and, in severe cases, sudden death. This can occur many hours or even days after the formation of the clot.” [7]

A well known way to reduce the risk on DVT is by wearing compression stockings. Compression on the leg surfaces forces blood to flow from the small surface vessels into the larger, deep venous system. Besides it also prevents back-flow of blood and the formation of clots. Generally only passengers with increased chance on DVT use these stockings. Although these stockings reduce risk on DVT it does not deal with physical discomfort in general.

In order to reduce physical discomfort contraction of muscles is very important. Muscle activity helps to keep the blood flowing through the veins, particularly in the deep veins. A way of generating muscle activity is by moving around in the cabin. However people don’t want to cause inconvenience to neighbor passengers while passing or are too tired. Also potential health benefits should be balanced against the risk of allowing a lot of passengers walking around in the plane with possible chance on unexpected turbulence. For this reason it is recommended to stimulate muscle activity while sitting in a chair. Many Airlines already provide a number of exercises to their passengers which can be performed during the flight. Research proved that the most effective exercises to stimulate circulation and thereby reduce discomfort, fatigue, stiffness and DVT. Research has been done to evaluate recommended lower leg exercises by airlines to investigate which exercises induce optimum calf muscle pump activity[5], and these exercise provide important information as starting point for a solution (Fig. 1).

Fig. 1. recommended exercises and produced EMG in calf muscle [5]

In an economy class cabin environment of an airplane, the dimensions of the chairs and space between the chairs are an important factor towards a solution. There are no global regulations concerning seat spacing in airplanes. The “Civil Aviation Authority” (CAA) based in the United Kingdom formulated regulations concerning seat spacing for planes registered in the UK. These Regulations entitled “Airworthiness Notice No. 64 (AN64)” [6] dates from 1989 and is not revised since. Currently these regulations are still used as leading regulations for seat spacing within JAA countries (Countries connected to the Joint Aviation Authorities, which include a large number of European countries). The seat requirements are split into three leading dimensions, shown in Fig. 2.

These dimensions introduce a real challenge because when the AN64 regulations were made it satisfied 95 percentile for dimension “A”. Nowadays the same regulation only satisfies 77 percentile of European passengers as result of increased average length of Europeans.

The AN64 guidelines do not describe a minimum width of the chairs. An investigation to chair widths among different airlines in economy class shows a minimum of 420 mm and a maximum width of 480 mm. Out of the seat measures in economy class can be concluded that this is a big constraint for in-chair exercises.

Today in-flight entertainment is offered to reduce psychological stress. This project on the other hand focuses on reducing physical stress. By reduction of physical stress the chance on health related issues like DVT, stiffness and fatigue are reduced [9]. In order to achieve this, passengers are required to exercise. These exercises can be described as a set of simple movements. However the real challenge is to stimulate passengers to make these movements. To reach this goal the in-flight entertainment system could by used or it could simply be an expansion of the in-flight entertainment.

Fig. 2 Seat dimensions according to AN64 [10]

3 Concepts

With the problem and the context as starting points, an idea generation session was held with four industrial designers. The most promising ideas generated during this session are shortly described below. A remarkable detail is that all ideas except one are based on games because games were considered to be typical interactive content that motivates people to take active roles [11].

1.   Quiz game; Answering questions happens by body movement.

2.   Drive game; with physical steering wheel and pedals.

3.   Agility game; balancing digital objects by physical movement.

4.   Copy game; try to copy movements of others as well as possible.

5.   Movement of passengers will be triggered by changes somewhere in the cabin.

Fig. 3. Control in-flight entertainment by body movement

A benchmark based on a Pugh matrix (also called a decision matrix) has done to find the most promising idea. A Pugh matrix is a way to assess ideas based on pre selected criteria and matching weight factors. The result of this matrix should show the most promising idea(s) according to the set criteria. The used assessment criteria from most important to less important are:

1.   Usability of basic principle; can the basic principle behind the idea be used for a wide variety of games / exercises or is it only applicable for a small range of games and exercises.

2.   Feasibility within space; to what extend might the limited amount of space in the plane’ chair lead to problems in executing the exercises.

3.   Movement intensity; to what extend do passengers have to move concerning the intensity and repetitions.

4.   Disturbance of other passengers; Does the idea lead to disturbance or annoyance of other passenger.

The benchmark showed an almost equal score for a few ideas. The similarity among these ideas was that it contained digital content that is controlled by physical movement. The difference on the other hand is the digital content used for every idea. This means that a generic interaction platform could be a solid basis for a width range of applications. The differences would be made by different content. With this assumption the project continued with finding solutions on how to translate physical movements or gestures to digital information that could be used as input for digital content.

Solutions to measure movements or gestures can be found in two directions: with attached sensors or detached sensors.

Attached sensors

A set of first solutions shown in figure 3 are based on attached sensors (sensors that need to be attached to the passenger). The attached sensors are therefore in most cases loose objects. This introduces some feasibility problems, which can be summarized by three keywords: Safety, effort and dependency.

Fig. 4. Suggested detached sensors

Safety: Whether this are written rules or not, it is logical that loose objects will cause serious safety issues in certain circumstance. These circumstances could be turbulence or passengers who are diminished responsible that use it for other purposes.

Effort: In case that a technological solution is found in a loose (embodied) object the chance on success smaller. The user has to put in extra effort in order to use the system. For some users this threshold is too high for tem to use the system.

Dependency: Regular objects can be used to obtain data from the user. For example the standard headphones could be equipped with sensors to measure a passenger’s head and neck movement. The problem with this is that a percentage of the passengers won’t use this headphone or will bring their own. This results in uncertainty about usage of the objects, which could lead to failure of the system.

The usage of separate attached objects is not desired and therefore ideally the solution should be fully integrated in the airplane.

Detached sensors

A number of technical solutions with detached sensors were considered. These solutions are described below.

Video-based gesture recognition: This way of gesture recognition is not ideal in the context of an economy class airplane. The biggest problem lies in the fact that the amount of free space is very limited. When the camera(s) is placed too close to the passenger there is the problem of not being able to cover enough of the passenger’s body.

Sensors integrated in the floor: With sensors integrated in the plane’s floor it is possible to measure the angle and distance of a passenger’s feet in relation to the floor. With this data the user’s lower body position can be calculated. This could be quite accurate because human anatomy and the usable movement space are known. However it would become integrated in the plain which makes adaptations or repairs very difficult and costly.

Sensors integrated in the chair: By a grid of pressure sensors integrated in an airplane chair it is possible to measure a person’s weight distribution. When there are changes in this weight distribution it is possible to derive a person’s position or gesture out of this data.

The last suggestion seems to be the most promising solution. However the question is how many sensors are needed to obtain a certain accuracy level and how many gestures can be recognized. Since the main goal is to stimulate people to move, motivating people to move is the more important than knowing exactly what movements are made. The system might work while it can only recognize a few different gestures. Therefore a system with a relative low accuracy might be sufficient. However this assumption needs to be further validated.

3 Solution and prototyping

The suggested solution is an airplane chair equipped with a grid of pressure sensors to detect a passenger’s body weight distribution. The seating surface is the most important part because there is always contact between this surface and the passengers. At the same time the weight changes in this surface are relative large compared to weight changes in the back area. This does not mean that sensors in the back surface are useless. This could be a good enrichment of the other data obtained from the seating surface. The obtained sensor values are processed and by means of pattern recognition the corresponding gestures are derived. When the gesture is determined this information can be used as input for the in-flight entertainment systems.

To validate the concept and to prove its effectiveness a prototype was build. Starting point for this prototype is a chair made out of MDF with similar dimensions of an average airplane chair. The seating surface of this chair is eventually equipped with a grid of 28 FSR’s (Force Sensing Resistor). In between the FSR and the user a four millimeter thick rubber plate is placed. On top of this rubber a round plate with a diameter of 40 millimeter is placed. This top plate is used to increase the surface that press on the FSR. The rubber plate is used to equally divide the weight on the FSR. The grid of sensors is connected with a 4 to 16 line decoder to a microcontroller. The data is send to a computer by means of a serial data connection. It takes about 50 ms. to read all the sensor values and to send them to a computer. The sensor values are fed into a neural network and the output of this neural network is the corresponding gesture.

For implementing the neural network a software package “JOONE” was used. JOONE (Java Object Oriented Neural Engine) is a Java based application to simulate neural networks and to train them [12, 13]. The neural network used for the prototype is a three-layer sigmoid network that consists out of 28 input neurons. In-between the input and output layer there is one hidden layer consisting out of 30 neurons. The output layer, which has nine neurons represent the different gestures.

The prototype is able to recognize nine gestures. A first test application had been created to test the principle of a gesture controlled game, where users have to balance a ball on a tray by lifting their legs was created. This game showed that a capability of only detecting three gestures (lifted right leg, lifted left leg, both legs down) already worked quite well.

Fig. 5 First prototype

4 Conclusion and future research

To reduce both the psychological and physical stress in air travel, the work presented here suggests a solution of integrating sensors in airplane seats to detect the gestures as input for in-flight entertainment systems. The content provided by the entertainment systems would help to reduce the psychological stress, and the gesture recognition is used as input for the interaction and hence stimulates people to move which, in turn reduces the physical stress as well.

The future research is needed to validate the assumption that this concept can really stimulate people to move. If so the next research question would consider the effectiveness of the concept. Do passengers move enough to reduce physical discomfort compared to current situations? And is the chance on health related issues like thrombosis really smaller by using this chair? A series of user studies has been planned and will be carried out to stress these questions.

Acknowledgement

This project is sponsored by the European Commission DG H.3 Research, Aeronautics Unit under the 6^th Framework Programme under contract Number: AST5-CT-2006-030958.

References

1.  C. Quigley, et al., Anthropometric Study to Update Minimum Aircraft Seating Standards,  EC1270, ICE Ergonomics Ltd, prepared for Joint Aviation Authorities, 2001.

2.  H. Hinninghofen and P. Enck, “Passenger well-being in airplanes,” Auton Neurosci, vol. 129, no. 1-2, 2006, pp. 80-85.

3.  SEAT Project Consortium, SEAT Project, http://www.seat-project.org, 2006.

4.  B.J. Hickman and R. Mehrer, “Stress and the effects of air transport on flight crews,” Air Medical Journal, vol. 20, no. 6, 2001, pp. 2-56.

5.  K.J. O'Donovan, et al., “An investigation of recommended lower leg exercises for induced calf muscle activity,” Proc. Proceedings of the 24th IASTED international conference on Biomedical engineering, ACTA Press, 2006 of Conference, pp. 214-219.

6.  K. Ball, “Deep vein thrombosis and airline travel--the deadly duo,” AORN Journal, vol. 77, no. 2, 2003, pp. 346-354.

7.  World Health Organisation, “Chapter 2: Travel by Air: Health Considerations,” International travel and health: Situation as on 1 January 2005, http://www.who.int/ith, 2005.

8.  N. Dowdall and T. Evans, The impact of flying on passenger health: a guide for healthcare professionals, BMA policy report, Board of Science and Education, British Medical Association, 2004.

9.  M.B. Alonso, “Affective Tangible Interaction; Towards Reducing Stress,” Proc. HCI Close 2U - 9th Sigchi.nl conference, 2005 of Conference, pp.

10. Civil Aviation Authority, CAP 747: Mandatory Requirements for Airworthiness, http://www.caa.co.uk/docs/33/CAP747.PDF, 2006.

11.  H. Breuer, Interaction Design for Flight Entertainment, bovacon, 2006.

12.  T.H. Jeff, Introduction to Neural Networks with Java, Heaton Research, Inc., 2005.

13.  P. Marrone, The Joone Complete Guide, www.joone.org, 2007.