Advertisement

Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

Simulating and predicting others’ actions

When we observe another person perform an action, like cracking an egg or kicking a ball, we are able to anticipate quite precisely the future course of the observed action. This ability is important in understanding others and in coordinating our actions with them, whether baking a cake together or playing football. This special issue includes 14 papers examining the cognitive and brain mechanisms underlying the ability to predict and simulate other people’s actions.

Prediction and simulation are two closely related processes that contribute to the ability to comprehend and respond to other people’s actions. Strictly speaking, prediction of the future course of an action makes use of simulation mechanisms, but simulation does not require prediction (for example, simulation could be retrodictive). An increasing body of research suggests that people run internal simulations based on their own motor repertoire when predicting the future course of other people’s actions (Blakemore & Frith, 2005; Kilner, Marchant, & Frith, 2009; Wilson & Knoblich, 2005). For instance, eye movements during action observation are predictive (Flanagan & Johansson, 2003; Rotman, Troje, Johansson, & Flanagan, 2006), and motor regions of the brain are engaged when participants must predict upcoming action events (Kilner, Vargas, Duval, Blakemore, & Sirigu, 2004; Stadler et al., 2011). Earlier findings from monkey neurophysiological work revealed activation of mirror neurons within premotor cortex when the monkeys observed the relevant phase of an observed grasping movement being occluded (Umiltà et al., 2001).

The high accuracy of action prediction has been demonstrated using paradigms that require participants to predict an action that is temporarily occluded from vision (Graf et al., 2007; Parkinson, Springer, & Prinz, 2011; Springer & Prinz, 2010; Sparenberg, Springer, & Prinz, 2012). For instance, observers were shown to be accurate in differentiating between time-coherent and time-incoherent continuations of temporarily occluded human full-body actions, suggesting the use of an internal prediction process that runs in real-time to the observed action (Sparenberg et al., 2012). The importance of motor experience for prediction is documented in a body of evidence showing that observers are most accurate in predicting those actions that they are able perform themselves. Recent imaging-based studies comparing the prediction abilities of motor experts versus novices (e.g., athletes) revealed increased motor activation during the observation of actions that are in the observer’s own motor repertoire (Buccino et al., 2004; Calvo-Merino, Glaser, Grezes, Passingham, & Haggard, 2005; Cross, Hamilton, & Grafton, 2006). Together, these findings suggest that action prediction is accurate, runs in real-time and depends on sensorimotor brain regions. This supports the broader notion that an observer can use his or her motor system to simulate observed actions (Jeannerod, 2001; Grèzes & Decety, 2001) and is compatible with the more specific predictive coding model of the mirror neuron system as advanced by Kilner et al. (e.g., Kilner, Friston, & Frith, 2007).

While the predictive functions of the human premotor cortex have already stimulated a plethora of neuroscientific research (see Schubotz, 2007, for a review), the precise cognitive underpinnings of action simulation and action prediction remain underspecified. To better understand the nature of human action prediction abilities, we need to delineate the details of putative mental simulation mechanisms. The papers in this special issue address such questions from a variety of approaches.

To advance our understanding of action prediction, it is valuable to explore the underlying cognitive processes, considering the role of action form and timing in prediction. Behavioural, eye-tracking, psychophysical and brain imaging methods can all provide converging evidence for the role of action simulation. Finally, it is important to consider how action prediction relates to other aspects of cognition such as language and cooperation or domain-specific expertise. Addressing these research questions touches on fields as diverse as action planning, motor control, memory, learning, attention, and understanding action-related language. It is this breadth of approach toward a single theme that we hope to capture in this special issue. Our aim is to advance the study of the fundamentals of action simulation by bringing together state-of-the-art research including theoretical work and empirically grounded papers illuminating the processes involved in the prediction of natural and artificial actions.

Several papers in this special issue examine the mechanisms of action simulation. Many of these papers use the same action occlusion paradigm that has proven to be very valuable in the study of action simulation (Graf et al., 2007; Springer et al., 2011). In this paradigm, participants see a video of an action that is then briefly occluded. After the occlusion period, participants see a second video clip and are asked if the second video is an accurate continuation of the first, or if it has jumped in time (Stadler et al., 2011). Good performance on this task requires detailed simulation of the action continuation during occlusion, and factors that change performance can then be measured. For example, Saygin and Stadler (2012) examine the influence of human, robot or humanoid form and motion on action prediction, and suggest that slower prediction is used for robot actions compared to human actions.

A similar question is examined by Stadler, Springer, Parkinson, and Prinz (2012), who asked whether simulation processes involved in predicting occluded actions are specific for those actions that follow a biological motion profile, such as those we are accustomed to seeing our fellow humans perform. They report that participants are better able to predict the occluded actions of normal point light walkers compared to those whose kinematics had been altered to move according to a non-human, constant velocity profile. A particularly interesting implication of this work that the authors highlight is the importance of designing virtual or robotic agents that move according to human-like biological motion profiles in order to facilitate human interactions with non-human agents. Of relevance to both these studies is a review paper from Gowen and Poliakoff (2012) that examines the influence of human or non-human form on action simulation. The paper finds evidence for a complex interplay of top-down and bottom-up features in processing observed actions.

The role of motion and kinematic features in action simulation is explored in the paper from Parkinson et al. (2012). They found accurate action simulation after even quite short action primes (before the occlusion) but that performance was degraded when short test clips (after the occlusion) were used. This demonstrates the exceptional sensitivity of the human action simulation system. Stapel et al. (2012) examined the impact of the broader action context on action prediction with a task where participants predicted if an actor would continue to walk or would bend to crawl in the next frames of a video. The results showed that prediction is better when actions are constrained by the context of the scene, suggesting that goal-directedness is useful in action prediction.

Two further studies examine the relationship between action simulation and action-related language. Liepelt, Dolk, and Prinz (2012) find clear evidence for interference between language and action, with word perception tasks affecting hand actions and hand actions affecting language production. The results suggest that language tasks and hand action tasks draw on overlapping cognitive resources, and thus support an embodied cognition hypothesis. A related study from Springer et al. (2012) examined the influence of masked prime words on action processing. Masked priming with dynamic action verbs altered subsequent performance on the action occlusion task, demonstrating close links between linguistic processing and action simulation.

An alternative approach is to consider how online simulation processes impact our appraisal of actions we perform in collaboration with other people (Doerrfeld, Sebanz, & Shiffrar, 2012). In an elegant series of experiments, these authors demonstrate that our perception of actions is shaped not only by what we are capable of doing ourselves, but also by what we can achieve in cooperation with other people. Specifically, Doerrfeld et al. found that a participant’s judgement of the weight of an object to be lifted fluctuates depending on whether he or she plans to lift it by himself or herself or with a co-actor who is either healthy or injured. Thus, it appears that simulation processes that underpin how we perceive, predict, and perform actions within a social world are sensitive to the others around us, which Doerrfeld et al. suggest might serve as a driving force for social collaboration.

The study of the neural correlates of action simulation and prediction can be very valuable in understanding the relationship between these and other cognitive processes. Recent brain imaging studies have indicated the involvement of motor programs in action prediction by showing increased activation in the motor system when we are engaged in predicting actions performed by others (Kilner et al., 2004; Stadler et al., 2011). The premotor cortex forms a core part of the network recruited during action observation (Grèzes & Decety, 2001), which has recently been described as mainly serving predictive functions (Wilson & Knoblich, 2005; Schütz-Bosbach & Prinz, 2007). Further, it is now well established that the action-observation network responds to both naturalistic and abstract stimuli (e.g., Petroni, Baguear, & Della-Maggiore, 2010; Schubotz & von Cramon, 2004). But how is this possible and what does a predictive account of the motor system imply?

In this issue, Fleischer, Christensen, Caggiano, Thier, and Giese (2012) propose a neural model that highlights both the commonalities and differences of abstract and naturalistic action observation. Based on human psychophysical data and model simulations, the authors argue that the commonalities emerge in parietal and premotor regions representing the causality of an action both for abstract and natural stimuli. Moreover, the model implies that a specific visual tuning is required to process abstract as compared to naturalistic stimuli. Both these inspiring new predictions can be applied to previous findings from imaging-based studies on this topic.

A second brain imaging paper addresses how observing an actor’s eye gaze and grasp affects the ability to predict a subsequent action. While previous research has typically studied these simulation processes separately, Ramsey, Cross, & de C. Hamilton (2012) made use of functional magnetic resonance imaging to investigate to what extent gaze- and grasp-perception rely on common or distinct brain networks using a ‘peeping window’ protocol. In object-present versus object-absent conditions, gaze observation revealed increased activations in left anterior inferior parietal lobule (aIPL), while grasp observation yielded increased activations in the dorsal premotor, posterior parietal, fusiform and middle occipital brain regions (regions composing the action observation network; AON). Interestingly, the authors suggest that the left aIPL activity supports the notion of a predictive process that signals upcoming hand-object interactions based on another person’s eye gaze. Activity within the AON, on the other hand, may reflect a motor simulation process for observed object-directed hand actions.

Another pertinent experiment that highlights the role of the motor system for action prediction is reported by Alaerts, de Beukelaar, Swinnen, and Wenderoth (2011). Using transcranial magnetic stimulations, the authors explored how the excitability of the observer’s primary motor cortex (M1) changes according to the force-requirements of an observed action. Specifically, observation-induced force-encoding was studied during various phases of executed and observed reach to grasp to lift movements with objects of different weights. The results showed that M1 excitability was indeed modulated by object weight during the grasp/lift phases for both execution and observation. However, surprisingly, during observation, M1 excitability was also modulated in an early reach phase when no visual cues on the object’s weight could be derived from the action movies. As the observers were aware of the fact that the same weight condition was presented repeatedly, these interesting new findings suggest that the motor system represents motor predictions as well as muscular requirements by taking into account information from previous trials for this predictive activation.

New insights related to aspects of motor control can be gained from a study by Kourtis, Sebanz, and Knoblich (2012) investigating whether actors represent the difficulty of a planned action before its actual execution. The authors recorded high-density EEG while participants planned and executed actions. Results showed that, despite a long cued preparation, the movement times varied according to Fitt’s Law and the amplitude of the P3b potential varied with movement difficulty. This interesting new finding is discussed in terms of an internal reflection of perceived action difficulty, which may correspond to the perceived risk of error in a planned action.

Another approach that has proven to be particularly fruitful in furthering our understanding of action simulation is one that considers interindividual differences and the role played by expertise in predicting others’ actions. Two papers here have turned to expert sporting populations to further elucidate the impact of hours of experience on predicting others’ actions. Prior work in this domain has demonstrated that how we perceive others’ actions is substantially impacted by physical expertise, in domains ranging from basketball to contemporary dance (e.g., Aglioti, Cesari, Romani, & Urgesi, 2008; Calvo-Merino et al., 2005; Cross et al., 2006). In this issue, a paper by Diersch, Cross, Stadler, Schutz-Bosbach, and Rieger (2012) tested figure skating experts and novices to examine not only how physical expertise impacts prediction of ongoing occluded actions, but also how the aging process might interact with expertise when simulating others’ actions. Diersch et al. show that figure skating expertise positively impacts young and older adults’ ability to predict ongoing actions, but only when those actions are within the observer’s domain of expertise. This study provides the first evidence that sensorimotor expertise, even when gained many decades ago, still influences how we perceive others and are able to predict their ongoing movements.

A second paper on expertise examines the impact of physical compared to observational experience on volleyball players’ ability to predict the outcome of volleyball serves. While prior literature has shown that perceptual experience alone can impact how we perceive or predict others’ actions (e.g., Buchanan & Wright, 2011; Cross et al., 2009, 2011; Higuchi, Holle, Roberts, Eickhoff, & Vogt et al., 2012), a number of significant unexplored questions remain regarding the role of physical compared to visual expertise. Here, Urgesi, Savonitto, Fabbro, & Aglioti (2011) demonstrate that adult volleyball players and spectators are more accurate than novices in predicting volleyball serves, but only the players were able to base their predictions on kinematic information. To more fully understand the mechanism behind this finding, Urgesi et al. ran a follow up experiment where adolescent volleyball players were taught a new serve either by physical or observational practice. The authors report that only physical practice positively impacts an observer’s ability to read body kinematics, while observational practice helps to improve an observer’s prediction of the ball’s trajectory. Urgesi et al. thus conclude that physical practice is required to form the most accurate perceptuo-motor representations of novel actions.

In sum, action simulation can be conceived of as a real-time process supported by sensorimotor brain regions that is impacted by myriad contextual features ranging from the social relevance of the agent or action to the observer’s prior experience with the observed action. The research featured in this special issue collectively advances the state-of-the-art of action cognition through careful investigation of the factors that shape how we perceive and predict other people in action. As editors, our aim is for this special issue to inspire further research that continues to define the functions and features of action simulation by bridging behavioural and brain-based approaches. We would like to urge stronger consideration of cognitive paradigms in neurosciences and to encourage experimental psychologists to put the proposed cognitive processes of action simulation and prediction to test at the neural level. As the papers in the following pages of this special issue suggest, interdisciplinary inquiry into action simulation has the potential to yield the most relevant and novel insights into how we make sense of and interact with other people moving around us in a social world.

References

  1. Aglioti, S. M., Cesari, P., Romani, M., & Urgesi, C. (2008). Action anticipation and motor resonance in elite basketball players. Nature Neuroscience, 11(9), 1109–1116. doi:10.1038/nn.2182.

  2. Alaerts, K., de Beukelaar, T. T., Swinnen, S. P., & Wenderoth, N. (2011). Observing how others lift light or heavy objects: Time-dependent encoding of grip force in the primary motor cortex. Psychological Research. doi:10.1007/s00426-011-0380-1.

  3. Blakemore, S. J., & Frith, C. (2005). The role of motor contagion in the prediction of action. Neuropsychologia, 43(2), 260–267. doi:10.1016/j.neuropsychologia.2004.11.012.

  4. Buccino, G., Lui, F., Canessa, N., Patteri, I., Lagravinese, G., Benuzzi, F., et al. (2004). Neural circuits involved in the recognition of actions performed by nonconspecifics: An FMRI study. Journal of Cognitive Neuroscience, 16(1), 114–126.

  5. Buchanan, J. J., & Wright, D. L. (2011). Generalization of action knowledge following observational learning. Acta Psychologica, 136(1), 167–178. doi:10.1016/j.actpsy.2010.11.006.

  6. Calvo-Merino, B., Glaser, D. E., Grezes, J., Passingham, R. E., & Haggard, P. (2005). Action observation and acquired motor skills: An fMRI study with expert dancers. Cerebral Cortex, 15(8), 1243–1249. doi:10.1093/cercor/bhi007.

  7. Cross, E. S., Hamilton, A. F., & Grafton, S. T. (2006). Building a motor simulation de novo: Observation of dance by dancers. Neuroimage, 31(3), 1257–1267. doi:10.1016/j.neuroimage.2006.01.033.

  8. Cross, E. S., Kraemer, D. J., Hamilton, A. F., Kelley, W. M., & Grafton, S. T. (2009). Sensitivity of the action observation network to physical and observational learning. Cerebral Cortex, 19(2), 315–326. doi:bhn08310.1093/cercor/bhn083.

  9. Cross, E. S., Stadler, W., Parkinson, J., Schutz-Bosbach, S., & Prinz, W. (2011). The influence of visual training on predicting complex action sequences. Human Brain Mapping. doi:10.1002/hbm.21450.

  10. Diersch, N., Cross, E. S., Stadler, W., Schutz-Bosbach, S., & Rieger, M. (2011). Representing others’ actions: The role of expertise in the aging mind. Psychological Research. doi:10.1007/s00426-011-0404-x.

  11. Doerrfeld, A., Sebanz, N., & Shiffrar, M. (2011). Expecting to lift a box together makes the load look lighter. Psychological Research. doi:10.1007/s00426-011-0398-4.

  12. Flanagan, J. R., & Johansson, R. S. (2003). Action plans used in action observation. Nature, 424(6950), 769–771. doi:10.1038/nature01861.

  13. Fleischer, F., Christensen, A., Caggiano, V., Thier, P., & Giese, M. A. (2012). Neural theory for the perception of causal actions. Psychological Research. doi:10.1007/s00426-012-0437-9.

  14. Gowen, E., & Poliakoff, E. (2012). How does visuomotor priming differ for biological and non-biological stimuli? A review of the evidence. Psychological Research. doi:10.1007/s00426-011-0389-5.

  15. Graf, M., Reitzner, B., Corves, C., Casile, A., Giese, M., & Prinz, W. (2007). Predicting point-light actions in real-time. Neuroimage, 36(Suppl 2), T22–T32. doi:10.1016/j.neuroimage.2007.03.017.

  16. Grèzes, J., & Decety, J. (2001). Functional anatomy of execution, mental simulation, observation, and verb generation of actions: a meta-analysis. Human Brain Mapping, 12(1), 1–19.

  17. Higuchi, S., Holle, H., Roberts, N., Eickhoff, S. B., & Vogt, S. (2012). Imitation and observational learning of hand actions: Prefrontal involvement and connectivity. NeuroImage, 59(2), 1668–1683. doi:10.1016/j.neuroimage.2011.09.021.

  18. Jeannerod, M. (2001). Neural simulation of action: A unifying mechanism for motor cognition. NeuroImage, 14(1), S103–S109. doi:10.1006/nimg.2001.0832.

  19. Kilner, J., Friston, K., & Frith, C. (2007). Predictive coding: An account of the mirror neuron system. Cognitive Processing, 8(3), 159–166. doi:10.1007/s10339-007-0170-2.

  20. Kilner, J. M., Marchant, J. L., & Frith, C. D. (2009). Relationship between activity in human primary motor cortex during action observation and the mirror neuron system. PLoS ONE, 4(3), e4925. doi:10.1371/journal.pone.0004925.

  21. Kilner, J. M., Vargas, C., Duval, S., Blakemore, S. J., & Sirigu, A. (2004). Motor activation prior to observation of a predicted movement. Nature Neuroscience, 7(12), 1299–1301. doi:10.1007/s10339-007-0170-2.

  22. Kourtis, D., Sebanz, N., & Knoblich, G. (2012). EEG correlates of Fitts’s law during preparation for action. Psychological Research. doi:10.1007/s00426-012-0418-z.

  23. Liepelt, R., Dolk, T., & Prinz, W. (2011). Bidirectional semantic interference between action and speech. Psychological Research. doi:10.1007/s00426-011-0390.

  24. Parkinson, J., Springer, A., & Prinz, W. (2011). Can You See Me In The Snow? Action simulation aids the detection of visually degraded human motion. Quarterly Journal of Experimental Psychology, 64(8), 1463–1472. doi:10.1080/17470218.2011.594895.

  25. Parkinson, J., Springer, A., & Prinz, W. (2012). Before, during and after you disappear: Aspects of timing and dynamic updating of the real-time action simulation of human motions. Psychological Research. doi:10.1007/s00426-012-0422-3.

  26. Petroni, A., Baguear, F., & Della-Maggiore, V. (2010). Motor resonance may originate from sensorimotor experience. Journal of Neurophysiology, 104(4), 1867–1871. doi:10.1152/jn.00386.2010.

  27. Ramsey, R., Cross, E. S., & de, C. Hamilton, A. F. (2011). Predicting others’ actions via grasp and gaze: Evidence for distinct brain networks. Psychological Research. doi:10.1007/s00426-011-0393-9.

  28. Rotman, G., Troje, N. F., Johansson, R. S., & Flanagan, J. R. (2006). Eye movements when observing predictable and unpredictable actions. Journal of Neurophysiology, 96(3), 1358–1369. doi:10.1152/jn.00227.2006.

  29. Saygin, A. P., & Stadler, W. (2012). The role of appearance and motion in action prediction. Psychological Research. doi:10.1007/s00426-012-0426-z.

  30. Schubotz, R. I. (2007). Prediction of external events with our motor system: Towards a new framework. Trends in Cognitive Science, 11(5), 211–218.

  31. Schubotz, R. I., & von Cramon, D. Y. (2004). Sequences of abstract nonbiological stimuli share ventral premotor cortex with action observation and imagery. Journal of Neuroscience, 24(24), 5467–5474. doi:10.1523/jneurosci.1169-04.2004.

  32. Schütz-Bosbach, S., & Prinz, W. (2007). Perceptual resonance: Action-induced modulation of perception. Trends in Cognitive Science, 11(8), 349–355. doi:10.1016/j.tics.2007.06.005.

  33. Sparenberg, P., Springer, A., & Prinz, W. (2012). Predicting others’ actions: Evidence for a constant time delay in action simulation. Psychological Research, 76(1), 41–49. doi:10.1007/s00426-011-0321-z.

  34. Springer, A., Brandstädter, S., Liepelt, R., Birngruber, T., Giese, M., Mechsner, F., & Prinz, W. (2011). Motor execution affects action prediction. Brain and Cognition, 76, 26–36. doi:10.1016/j.bandc.2011.03.007.

  35. Springer, A., Huttenlocher, A., & Prinz, W. (2012). Language-induced modulation during the prediction of others’ actions. Psychological Research. doi:10.1007/s00426-012-0411-6.

  36. Springer, A., & Prinz, W. (2010). Action semantics modulate action prediction. Quarterly Journal of Experimental Psychology, 63(11), 2141–2158. doi:10.1080/17470211003721659.

  37. Stadler, W., Schubotz, R. I., von Cramon, D. Y., Springer, A., Graf, M., & Prinz, W. (2011). Predicting and memorizing observed action: Differential premotor cortex involvement. Human Brain Mapping, 32(5), 677–687. doi:10.1002/hbm.20949.

  38. Stadler, W., Springer, A., Parkinson, J., & Prinz, W. (2012). Movement kinematics affect action prediction: Comparing human to non-human point-light actions. Psychological Research. doi:10.1007/s00426-012-0431-2.

  39. Stapel, J. C., Hunnius, S., & Bekkering, H. (2012). Online prediction of others’ actions: The contribution of the target object, action context and movement kinematics. Psychological Research. doi:10.1007/s00426-012-0423-2.

  40. Umiltà, M. A., Kohler, E., Gallese, V., Fogassi, L., Fadiga, L., Keysers, C., & Rizzolatti, G. (2001). I know what you are doing: A neurophysiological study. Neuron, 31(1), 155–165.

  41. Urgesi, C., Savonitto, M. M., Fabbro, F., & Aglioti, S. M. (2011). Long- and short-term plastic modeling of action prediction abilities in volleyball. Psychological Research. doi:10.1007/s00426-011-0383-y.

  42. Wilson, M., & Knoblich, G. (2005). The case for motor involvement in perceiving conspecifics. Psychological Bulletin, 131(3), 460–473. doi:10.1037/0033-2909.131.3.460.

Download references

Author information

Correspondence to Anne Springer.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Springer, A., de C. Hamilton, A.F. & Cross, E.S. Simulating and predicting others’ actions. Psychological Research 76, 383–387 (2012). https://doi.org/10.1007/s00426-012-0443-y

Download citation