Session 1

From ICE PhD Wiki
Jump to: navigation, search

Activity monitoring / Interactions

Main topics:

  • Wearable/unobtrusive monitoring systems
  • Person tracking in cluttered and wide scenarios with multiple and heterogeneous sensors
  • Group and crowd activity monitoring
  • Behavior understanding
  • Emotion detection and classification
  • Learning and modeling cognitive entities interactions in smart environments
  • Behavior forecasting
  • Incorporating devices and subsystems in the conceptual interaction model
  • Simulation of interactions
  • Activity Recognition and Human Behaviour Understanding over smartphone platform

Students: Tewodros A. Biresaw, Anh-Tuan Nguyen, Waqar Baiq, Panagiotis Karanikas, Isah Abdullahi Lawal, Chitra Hapsari Ayuningtyas, Bo Zhang, Andrea Sciarrone

Supervisor: Prof. Andrea Cavallaro

Behavior Forecasting

Proposals

Original papers

No Paper Main Contribution Student
1 Alex Pentland and Andrew Liu, "Modeling and prediction of human behavior", Neural Computation, Vol.11, No.1, 1999, pp.229-242 Proposed a dynamic Markov chain model for predicting human behaviour by considering human as a device with many internal mental states, each with its own control behaviour and interstate transistion probabilities IAL
2 B.D. Ziebart, A.L. Maas, J.A. Bagnell, and A.K. Dey, "Human Behavior Modeling with Maximum Entropy Inverse Optimal Control" Proc. AAAI Spring Symposium: Human Behavior Modeling, 2009, pp.92-. Proposed a conditional probability model for predicting human decision given a contextual situation by considering human behaviour as a structured sequence of context-sensitive decisions. IAL
3 Koller, D. and Fei-Fei, L. and Tang, K. Learning Latent Temporal Structure for Complex Event Detection. 2012 IEEE Conference on Computer Vision and Pattern Recognition, 1250-1257, 2012. To provide a novel method which can model the temporal structure of complex events by using the Latent Structural SVM BZ
4 Patron-Perez, A., Marszalek, M., Reid, I., Zisserman, A.:Structured learning of human interactions in tv shows. IEEE Transactions on Pattern Analysis and Machine Intelligence (2012) To provide a framework which can incorporate the spatial relationship among different people into the human interaction recognition system by using the Structured SVM BZ
5 Laptev, I.: On space-time interest points. International Journal of Computer Vision 64 (2005) 107-123 To provide a 3D Harris detector which can capture the obvious variation in the video sequences and use HOG + HOF to describe the motion feature around each spatial-temporal interest point BZ
6 Nuria M. Oliver, Barbara Rosario, and Alex P. Pentland: A Bayesian Computer Vision System for Modelling Human Interactions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), pp 831-843 , 2000 Proposed a Bayesian model for integration of both prior knowledge and evidence from data, through combining top-down and bottom-up information in a closed feedback loop ATN
7 Nathan Eagle and Alex P. Pentland: Eigenbehaviors: identifying structure in routine. Behavioral Ecology and Sociobiology, 63(7), pp.1057-1066. Introduced 'eigenbehaviors', which are a set of characteristics vectors. Behaviors of a person is modelled by a weighted sum of eigenbehaviors, and then the model will be able to predict the upcoming behaviors. ATN
8 Nathan Eagle and Alex P. Pentland: Event Detection and Analysis from Video Streams. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(8), pp.873-889. Proposed a framework for the recognition and modelling scenario, thus taking information from context into the detection process ATN
9 Ling Bao and Stephen S. Intille: Activity Recognition from User-Annotated Acceleration Data. In Proceceedings of the 2nd International Conference on Pervasive Computing, 1–17, 2004. Proposed an algorithm to recognize physical activities from data acquired using five small biaxial accelerometers worn simultaneously on different parts of the body. AS
10 Juha Parkka, Luc Cluitmans, and Miikka Ermes: "Personalization Algorithm for Real-Time Activity Recognition Using PDA, Wireless Motion Bands, and Binary Decision Tree". IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 14, NO. 5, SEPTEMBER 2010. Proposed an automatic recognition of type of physical activity, a decision tree classifier, that can be used to show the user the distribution of his daily activities and to motivate him into more active lifestyle. AS
11 Sian Lun Lau and Klaus David: Movement Recognition using the Accelerometer in Smartphones. Future Network & MobileSummit, 2010. Recognized the human activity using the built-in accelerometer by comparing the influences of classification algorithms, features and the combination of sampling rates and window sizes for features extraction have on the classification accuracy. AS
12 P. Dollar, V. Rabaud, G. Cottrell, S. Belongie, "Behavior recognition via sparse spatio-temporal features", 2nd Joint International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, 2005 pp. 65-72 Proposed a technique for behavior recognition from video by characterizing behavior with spatio-temporal feature derived from interest points in the video sequence IAL
13 M. Luber, J.A. Stork, G.D.Tipaldi, K.O. Arras, "People tracking with Human Motion Predictions from Social Forces", International Conference on Robotics and Automation, 2010 pp. 467-469 Proposed a technique to track and predict human motions using social force model by taking into the account all the forces acting on human at any particular instant WB
14 A.Dore, A.F. Cattoni, C.S. Regazzoni, "Interaction modeling and prediction in smart spaces: a bio-inspired approach based on autobiographical memory", IEEE Transactions on Systems, Man and Cybernetics, 2010 pp. 1191-1205 Proposed a technique to model interactions based on dynamic Bayesian model and predicting events based on bio-inspired autobiographical memory WB
15 N. Kern, B. Schiele, A. Schmidt, "Recognizing context for annotating a live life recording", Personal and Ubiquitous Computing, vol. 11, no 4, pp. 251-263, 2007 Assessed different wearable technologies and algorithms for creating contextual annotations and investigating how meta-information for any kind of data gathered in real world environments can be automatically obtained using a variety of sensors. PK
16 C. Voigtmann, S. L. Lau, K. David, "A Collaborative Context Prediction Technique", Vehicular Technology, IEEEE Conference-VTC-Spring, pp. 1-5, 2011 Proposed the Collaborative Context Prediction(CCP)approach which deals more effectively with sudden changes in the behaviour of a user and overcomes the gap of missing context information in the user's context history. PK
17 D. Cook, G. Youngblood et al., "MavHome: An Agent-Based Smart Home", IEEE International Conference on Pervasive Computing and Communications, pp. 521-524, 2003 Presented the MavHome smart home architecture in which several prediction algorithms are introduced that play critical roles in an adaptive and automated environment such as a smarthome including the prediction of the inhabitant's next action. PK
18 W.Weihua , L. Zhijing , "Real-time human behavior recognition based on articulated model," IEEE, Computer Science and Information Technology (ICCSIT), 2010 Use of angle and length of articulated structure as features for classifying human behaviors. TAB
19 A. Ess, K. Schindler, L. van Gool: "You'll never walk alone: Modeling social behaviour for multi-target tracking". International conference on computer vision (ICCV), 2009 Modeling and incorporating social behavior of objects allow improving performance of multiple target tracking. TAB
20 C. Chi-Hung, H. Jun-Wei, C. Yi-Da, T. I-Ru, J. Ming-Hui , "Human behavior recognition from arbitrary views," IEEE, Circuits and Systems (ISCAS), 2010 Perform view alignment procedure in order to obtain view invariant behavior recognitions. TAB
21 YA Ivanov and AF Bobick. "Recognition of visual activities and interactions by stochastic parsing". IEEE Transactions on Pattern Analysis and Machine Intelligence. 22(8), pp. 852–872. 2000. Proposed a hierarchical approach to recognize activities and interactions between multiple agents from video streams, with the lower level recognition performed using probabilistic models (e.g., HMMs) and the higher level performed using stochastic context-free parser that allows inclusion of a priori knowledge about the domain CHA
22 MS Ryoo and JK Aggarwal. "Stochastic representation and recognition of high-level group activities". International Journal of Computer Vision. 93(2), pp. 183-200. 2011. Proposed a stochastic methodology for recognizing high-level group activities with complex temporal, spatial and logical structures from video streams CHA
23 T Gu, Z Wu, X Tao, HK Pung, J Lu. "epSICAR: an emerging patterns based approach to sequential, interleaved and concurrent activity recognition". IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 1-9. 2009. Proposed a pattern based algorithm to recognize sequential, interleaved, and concurrent activities from sensor readings CHA
24 Kalev H. Leetaru. "Culturomics 2.0: Forecasting large-scale human behavior using global news media tone in time and space". First Monday Peer Reviewed Journal on Internet, Vol. 16. 2011. Proposed an interesting technique using text mining techniques to predict future events. It predicted correctly the Arab uprising e.g., in Tunisia, Egypt and Libya. It belongs to new area of research Culturomics WB


Open research issues

  • How to model accurately human behaviour considering the inherent uncertainty and variability of behaviour exhibit by individual in different scenarios (IAL)
  • Behavior interpretation is highly context dependent. How to correctly model behavior when contextual information is not directly observable? (IAL)
  • As human behaviour is influenced by physical, emotional, cognitive and social factors, how to incorporate these factors in a model to predict human behavior? (IAL)
  • How to extract the motion features which could better describe the special character of different human interactions. The motion features should be robust enough, not restrict to the mutual occlusions of different body parts, background clutter, and illumination change, etc. (BZ)
  • How to find the most discriminate parts of video sequences which can better represent different human interactions? (BZ)
  • How to model the temporal structure of human interactions? (BZ)
  • How to fuse data captured from various sources, such as video, audio, wearable sensors... as an unified model? (ATN)
  • How to take into account the role of context as a cue for prediction? (ATN)
  • How to represent human body movement? (ATN)
  • How the elaboration of the accelerometer raw signals, acquired from several different sources, can be employed in order to extract high-level context information? (AS)
  • Since smartphones have become very popular and powerful devices how we can use them:
  • - 1) AS A HUB, to collect, transmit and elaborate the accelerometer signals acquired from other sensors? (AS)
  • - 2) AS A HUB and AS A SENSOR, to collect, transmit and elaborate the accelerometer signal acquired directly from the built-in accelerometer? (AS)
  • Considering that a mobile phone has limited battery duration and computational capacity and that its main aim is communicate (to call or send SMS, for example) with others, how can we find the best trade-off between Activity Recognition accuracy and energy consumption/computational load needed? (AS)
  • How to manipulate the emotional content data to correctly predict the behavior ? (WB)
  • How to use the previous experiences effectively in predicting behavior ? (WB)
  • How to track human activity with new, more efficient or smaller modalities ? (PK)
  • How to trace important sections in bigger parts of data without knowing beforehand its structure or pattern? (PK)
  • How to better understand the neurophysiological structure of the human brain and link it to behaviour forecasting ? (PK)
  • Developing feature extraction and recognition method that minimize intra-class differences and maximize inter-class differences for behaviors. (TAB)
  • Multi modal approaches, such as use of environment or background and other agents’ information, for accurate behavior recognition.(TAB)
  • Robust methods of view independent human action recognition.(TAB)
  • How to model complex activities such as interleaving-, concurrent-, parallel- or joint-activities? (CHA)
  • How to recognize "abnormal activities", which are important, for example, to security and health monitoring, but exist only scarcely in the data? (CHA)
  • How to perform transfer learning, i.e., exploit the previously learned behaviour models for new environments with (possibly) different sensors? (CHA)
  • How to analyze big data to predict future trends in social interaction analysis ? (WB)
  • How to correctly identify the important information cues to predict the future events ? (WB)

Results

Selected papers in behavior forecasting

No Paper Main Contribution
1 Alex Pentland and Andrew Liu, "Modeling and prediction of human behavior", Neural Computation, Vol.11, No.1, 1999, pp.229-242 Proposed a dynamic Markov chain model for predicting human behaviour by considering human as a device with many internal mental states, each with its own control behaviour and interstate transistion probabilities
2 B.D. Ziebart, A.L. Maas, J.A. Bagnell, and A.K. Dey, "Human Behavior Modeling with Maximum Entropy Inverse Optimal Control" Proc. AAAI Spring Symposium: Human Behavior Modeling, 2009, pp.92-. Proposed a conditional probability model for predicting human decision given a contextual situation by considering human behaviour as a structured sequence of context-sensitive decisions.
3 Nathan Eagle and Alex P. Pentland: Eigenbehaviors: identifying structure in routine. Behavioral Ecology and Sociobiology, 63(7), pp.1057-1066. Introduced 'eigenbehaviors', which are a set of characteristics vectors. Behaviors of a person is modelled by a weighted sum of eigenbehaviors, and then the model will be able to predict the upcoming behaviors.
4 C. Voigtmann, S. L. Lau, K. David, "A Collaborative Context Prediction Technique", Vehicular Technology, IEEEE Conference-VTC-Spring, pp. 1-5, 2011 Proposed the Collaborative Context Prediction(CCP)approach which deals more effectively with sudden changes in the behaviour of a user and overcomes the gap of missing context information in the user's context history.
5 Kalev H. Leetaru. "Culturomics 2.0: Forecasting large-scale human behavior using global news media tone in time and space". First Monday Peer Reviewed Journal on Internet, Vol. 16. 2011. Proposed an interesting technique using text mining techniques to predict future events. It predicted correctly the Arab uprising e.g., in Tunisia, Egypt and Libya. It belongs to new area of research Culturomics

Behavior forecasting from ICE perspectives - The selected papers above, highlight methodologies for forecasting behavior in different contextual scenarios. All with a single goal of proffering a means of taking proactive measures to improve life. For instance,in a human-machine systems [1], If a machine could recognize, and even better, anticipate the human’s behavior, it could adjust itself to serve the human’s needs better. Similarly, in social domain [5], by observing the social interaction and behavior of people from different sources (e.g news media, and social network) its possible to predict future events (e.g uprising) in order to control in advance any adverse effect those events could have in the socio-economic life of people.

Open research issues

  1. How to correctly model behavior when contextual information is not directly observable?
  2. How to fuse data captured from various sources, such as video, audio, wearable sensors as a unified model for forecasting?
  3. Uncertainty: How to model accurately human behaviour considering the inherent uncertainty and variability of behaviour exhibit by individuals?
  4. Behavior is mostly affected by many factors (e.g. physical, emotional, cognitive, social etc.). How to make a long term prediction given that these factors are likely to change with time.
  5. How to choose the correct cues among the many correlated cues available to predict a future event?