Loading LikeButton...0
Loading LikeButton...
Comments
Loading Comment Form...
Loading Comment Form...
Post-Symbolic Communication
Overlooking Tokyo, nestled within the National Museum of Emerging Science and Innovation (Miraikan) lies The Park of Aging, a realm where time bends into the distant future, offering a rare portal to your mind and body after years worn by aging.[^Odaiba] Visors blur vision, mimicking cataracts. Sounds are stripped of high pitches. In a photo booth that mirrors the trials of aged perception, facial expressions are faded and blurred. The simple act of recalling a shopping list committed to memory becomes an odyssey as one is ceaselessly interrupted in a bustling market. Walking in place on pedals with ankle weights on and while leaning on a cart simulates the wear of time on the body or the weight of age on posture. The Park of Aging is not just an exhibit, but an immersive conversation across time; a dialogue with your older self through the senses of sight, sound, and the aches and pains of age. This journey of empathy extends beyond the future self, to also foster a deeper connection with a present overlooked cohort: the elderly.
The Park of Aging is a poignant example of proprioceptive, post-symbolic communication, where participants receive information through an intimate, sensorial experience beyond merely interpreting words and symbols, utilizing all sensemaking of the body; the Park immerses participants in the sensations of being old, unlocking the first-hand experience of the deteriorating senses, including seeing and hearing words and symbols.
Temporal conversation with aging, experiences of proprioceptive, non-symbolic communication today are ubiquitous and include meditation, psychedelics, religious experience, romantic intimacy, dance, yoga, combat, and sports. Not coincidentally, these experiences which harness information gathered from many senses including proprioception (“higher-bandwidth communication”) are also correlated with long-lasting human bonding and connection when shared. Perhaps it is the diversity of information simultaneously presented to the senses (depth) that contribute to the significance of the experience and therefore strength of connection and “union.”
Technological innovations such as neural interfaces, mediated reality, and generative foundation models (GFMs) expands the possibility space of post-symbolic communication, where unions within oneself and among people can occur not only in person, but across temporal, physical, and social distance facilitated by technology. In this chapter, we describe these technologies and explore their frontiers. We examine how these technologies could revolutionize interpersonal connections, education, and collaboration by enabling the transmission of thoughts, feelings, and sensory experiences beyond the compression of words and symbols. We consider the opportunities to rectify cultural misunderstandings and conflicts by fostering profound empathy and shared experience. Yet, in a space where ideas flow as seamlessly as emotions, we must also explore the risks of this level of connectivity, including surveillance, homogenization, disconnection and oblivion.
[^Odaiba]: The entire artificial island of Odaiba on which Miraikan sits is something of a monument to ⿻. It contains a monument to Doraemon, a Manga robotic cat from the future who in Japanese cartoons of the 1970s returned to guide the imagination of children in the present, who has been a major inspiration in Taiwan. The largest mall on the island is named DiverCity and is devoted to the role of diversity in innovation.
Post-symbolic communication, a term coined by Jaron Lanier , ventures beyond the realm of language and symbols to explore the potential for direct and immersive shared experience by harnessing all senses, including proprioception.[^Jaron] Our first experience of non-verbal communication is in the natal womb; the synchronization of heartbeats between a mother and her unborn child, especially when the mother breathes rhythmically, suggests an intrinsic communication pathway.[^vanLeeuwen] As we develop, humans convey information nonverbally through body language, facial expression, tone, touch, laughter, crying, facial blood flow and smell. Biochemical messengers can convey emotional states and trigger responses in others, often unconsciously. For instance, research has shown that human sweat contains compounds that, when detected by others, can convey stress or fear, influencing the receiver's perception and behavior.[^JudithFernandez]. We also see glimpses of the post-symbolic potential in long-established practices of intimacy among humans:
Dance: stepping deeper into music, participants synchronizing their movements, both feel their body and anticipate their partner’s movements to create shared experience. Traditional dances like the Adumu ritual of the Maasai people also serve as a way for communal shared experiences through synchronicity.
Combat: on the battlefield, soldiers experience heightened senses, adrenaline, and a battery of sounds, leading to intense awareness of their surroundings and their comrades. Non-verbal cues become critical for survival and strategy. This shared high-stress environment creates a bond and deep understanding and trust develop among those who rely on each other for their lives.
Sports: in team sports, players develop a keen sense of each other's presence and movements. Teammates often anticipate each other's actions and work as a cohesive unit, relying on signals communicated through sound, movement, and hand gestures. This synchrony is somewhere between dance and combat, also fueled by common purpose.
Romantic intimacy: through touch, eye contact, and emotional vulnerability, partners create a unique, shared experience. Attention to the internal experience of a partner is critical to a successful bond, building proprioceptive empathy that demands perhaps the deepest mutual trust and understanding humans are capable of.[^Clancy]
Religious experience: in mystical practices like Sufism, participants collectively engage in rituals like whirling and chanting. These shared spiritual practices engage the senses in the same way, and create a sense of unity and connection with something greater than oneself, fostering community not just among people, but with a greater spiritual presence.
Yoga: in a yoga class, practitioners move together through sequences of poses, guided by the rhythm of their breath. Despite the individual nature of the practice, there is a collective harmony in the movements and the shared goal of health and peace, as well as a heightening nervous system through stretches and poses. This shared physical and meditative experience fosters a sense of communal energy and focus.
Each of these contexts illustrates how shared experiences, beyond the scope of verbal communication, can create deep bonds and understanding between participants through intensive, shared sensorial inputs. Yet, communication can also happen within oneself (or between oneself and the spiritual world), as a form of introspection, yielding some of the most profound and transformative experiences.
Meditation: a practice of inward focus and mindfulness, meditation has a diverse set of practices across a range of traditions. One common practice is to observe one's own thoughts and feelings without judgment, fostering greater recognition and understanding of one’s own mental and emotional states, offering an opportunity to bridge and reconcile internal differences and contradictions.
Psychedelics: The use of psychedelics can lead to profound internal experiences and altered states of consciousness that disrupt usual patterns of thought and perception, opening alternative narratives to make sense. Users often report experiencing deep introspective insights, a sense of oneness with the universe, confronting deep-seated emotions and memories, or communication with seemingly non-embodied forms of consciousness.
Prayer: Similarly, through prayer, contemplation, or participation in religious rituals, individuals open communication to something beyond their sensory experience. Whether through a feeling of divine presence or an internal sense of clarity and peace, these experiences can be deeply impactful and transformative, and foster a greater connection to themselves and their place in the universe.
Intimate experience today is rich with examples that touch upon the edges of post-symbolic communication. From physiological signals to chemosignals we are just beginning to learn the non-verbal, information-dense modes that can synchronize human experience and form the deepest human bonds between oneself and other humans, groups, and the universe broadly.
[^Clancy]: In her 2024 Playing With Reality, physicist and neuroscientist Kelly Clancy speculates that Homo sapiens' large pre-frontal cortex evolved not to manage large numbers of social relationships, as is commonly believed, but rather to manage a single highly complex relationship with a romantic partner. That the larger brain was ultimately useful for managing large numbers of secondary relationships was, to Clancy, a happy accident.
Today, we stand on the cusp of a Cambrian explosion in technologies that can advance post-symbolic communication and enable participants to communicate their physiological, psychological, and even phenomenological states of being. The Park of Aging is an example of an early experiment, but we can expect deeper, immersive, wholly sensorial experiences with novel integrations of technology that combine in supermodular ways. A number of these technologies include:
Neural and Brain-Computer Interfaces (BCI): devices that connect directly to the brain capture neural activity and offer a direct pathway for communicating complex thoughts and emotions.[^B2B] BCIs enable direct communication between the brain and external devices. Future developments could allow for the sharing of thoughts, emotions, and experiences directly from one mind to another, allowing for an unprecedented level of interaction.
Haptic feedback and homuncular flexibility: Haptic devices provide tactile sensations, simulating touch and physical interactions in virtual environments and allowing users to feel and respond to virtual stimuli as if they were real. Similarly, with homuncular flexibility, individuals can learn to control virtual bodies that differ significantly from their own, thereby transcending the limitations of their physical bodies.[^Homuncular] A leading example is the near-universal capacity of humans to "regain" from their evolutionary past a sense of agency over a tail, given sufficient feedback and control in a virtual world.[^tail]
3D audio and immersive soundscapes: Advanced sound technologies that create three-dimensional auditory experiences can deeply enhance the sense of immersion in a virtual space, conveying emotions and atmosphere in a way that traditional stereo sound cannot.
Wearable tracking of affect and physiology: Devices that monitor heart rate, skin conductance, and other physiological markers can provide insight into a user's emotional and physical state, enabling shared experiences that are responsive to these states.
Projection mapping and spatial computing: Technologies allow for the transformation of physical spaces into interactive, digitally augmented environments. They can create shared, multi-sensory experiences that blend the physical and digital worlds.
Neurofeedback and mindfulness: These applications use real-time displays of brain activity to teach self-regulation of brain function. They can be used for meditation, mental health therapy, and enhancing self-awareness and empathy.[^feedback]
Neuromodulation: Neuromodulation techniques, such as deep brain stimulation have implications for enhancing cognitive function, treating neurological disorders, and even augmenting human capabilities. However, they also introduce neuroethical challenges such as moral bio-enhancement (MBE) - the use of biomedical technology to morally improve humans.
[^B2B]: Rao et al., op. cit. [^tail]:William Steptoe, Anthony Steed and Mel Slater, "Human Tails: Ownership and Control of Extended Humanoid Avatars" IEEE Transactions on Visualization and Computer Graphics 19, no. 4 (2013): 583-590. [^feedback]: Hengameh Marzbani, Hamid Reza Marateb and Marjan Mansourian, "Neurofeedback: A Comprehensive Review on System Design, Methodology and Clinical Applicaitons", Basic Clinical Neuroscience 7, no, 2: 143-158.
When combined with GFMs, these technologies will further enable us to generate visual representations of our thoughts at a pace that closely mirrors the speed of our imagination. These technologies together are unlocking responsive, adaptive environments or characters in virtual spaces that can react in real-time to users' emotions, actions, or choices beyond simple interpretation of natural language inputs.[^RealTime] Researchers have already shown how brain implants can connect the intentions of a paralyzed patient into physical movements, demonstrating the remarkable potential of neural interfaces to bridge the gap between thought and action.[^WalkingNat]
Combined, these capabilities enable the transmission of ideas and emotions that can occur directly and seamlessly and have profound implications for how we share and understand one another's internal experiences, creative visions, aspirations and even past traumas to facilitate reconciliation, healing and forgiveness. For example, imagine a child immersing themselves in the sensory experiences of their parents at the same age. Or two warring groups experiencing the pain and loss of their adversary’s family members.
In the more distant future, the evolution of non-symbolic communication promises to profoundly reshape our understanding of intimacy and the very essence of life's touchstones, such as childhood and relationships. Imagine a world where the boundaries of personal experiences blur, redefining intimacy not as a physical or emotional proximity but as a deep, seamless sharing of consciousness. Telepathy, once a realm of science fiction or religious practice, becomes a scientific reality, allowing for the direct transmission of thoughts, feelings, and sensory experiences from mind to mind. Human relationships evolve into deeper, more meaningful connections where misunderstandings are a choice and empathy abound. Children, in this new paradigm, grow not just by learning from others' words or observing actions but by immersing themselves in the lived experiences of others from any culture or epoch, including their ancestors.[^Lowry] This experiential osmosis accelerates empathy and wisdom, fostering a society where learning is as much about absorbing direct experiences as it is about traditional education.
Long-distance relationships, too, can expect to undergo a radical transformation. Physical distance becomes a matter only of connection speeds, allowing for the sharing of thoughts, emotions, and sensory experiences in real-time, irrespective of geographic separation. Lovers, friends, and family members can experience each other's joys, sorrows, and mundane moments as if they were in the same room, creating a form of intimacy that transcends physical presence. This paradigm shift brings profound changes in societal structures – the traditional nuclear family could give way to more fluid, globally interconnected familial networks. As we steer towards this horizon, the very fabric of human connection and communication is poised to undergo a metamorphosis, redefining what it means to be intimately connected and be “human.”
[^Lowry] As vividly illustrated in Lois Lowry's classic 1993 young adult novel The Giver.
The journey into post-symbolic communication is not without its perils. The very tools that promise deeper connection could also lead to a loss of individuality. Worse, participants could be surveilled, leaving our thoughts, emotions, and motivations open to manipulation; any window into our mind is also a window of influence. At the extreme, there is a risk that participants no longer have a private inner world while also disconnecting from the outer world, a dystopia of which we have be warned at least since the time of E. M. Forster and most vividly in recent years by the 1999 film The Matrix.[^dystopias] Unchecked visibility into our minds allows for unprecedented levels of manipulation and control, potentially by technology companies or governments that distract humanity with an alternate reality, or simulation. As humans lose touch with the reality of the physical world, over-reliance on telepathic communication could lead to the atrophy of traditional communication skills and cultural practices, with people becoming dependent on direct mental connections. Furthermore, in a world where the boundaries between self and other blur, the sanctity of individual thought and experience could be threatened. High-bandwidth communication could lead to a homogenization of thoughts and experiences as individual perspectives merge into collective consciousness, erasing our differences.
[^dystopias]: E. M. Forster, "The Machine Stops", Oxford and Cambridge Review November 1909.
Balancing telepathic communication with lower-bandwidth, structured forms of communication is essential to preserve privacy, autonomy, diversity, and human governance in the future. Communication, like speech and text, is less direct and immediate than telepathy but is structured and deliberate. It requires the sender to formulate thoughts into words or sentences, providing a level of control and reflection that instantaneous telepathic communication lacks. Markets and voting systems also serve as quintessential examples of lower-bandwidth, structured forms of communication, offering a counterbalance. In markets, the myriad decisions made by consumers and producers are communicated through the price mechanism. This system, while less immediate and detailed than telepathic communication, provides a structured and aggregated way of expressing preferences and values. It allows for privacy in decision-making, as individuals do not have to reveal the full spectrum of their thoughts and motivations. Similarly, voting is a deliberate, structured form of communication where individuals express their political and social preferences at a fixed point in time. Unlike continuous and invasive telepathic streams, voting encapsulates the will of the populace in a manner that is both manageable and interpretable, preserving the autonomy of the individual voter. This structured approach is crucial in maintaining a balance between efficient communication and the safeguarding of personal autonomy, privacy, and democratic processes, thereby acting as a vital check against the overreach of an all-encompassing telepathic matrix.
[^Jaron]: Jaron Lanier, You Are Not a Gadget a Manifesto, (London [Etc.]: Penguin Books, 2011).
[^vanLeeuwen]: P. van Leeuwen, D. Geue, Michael Thiel, Dirk Cysarz, S Lange, Marino Romano, Niels Wessel, Jürgen Kurths, and Dietrich Grönemeyer, “Influence of Paced Maternal Breathing on Fetal–Maternal Heart Rate Coordination,” Proceedings of the National Academy of Sciences of the United States of America 106, no. 33 (August 18, 2009): 13661–66. https://doi.org/10.1073/pnas.0901049106.
[^JudithFernandez]: Judith Fernandez, "Olfactory interfaces: toward implicit human-computer interaction across the consciousness continuum," Diss. Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2020.
[^Homuncular]: Andrea Won, Jeremy Bailenson, Jimmy Lee, and Jaron Lanier, "Homuncular Flexibility in Virtual Reality," Journal of Computer-Mediated Communication, Volume 20, Issue 3, 1 May 2015, Pages 241–259, https://doi.org/10.1111/jcc4.12107
[^RealTime]: Han Huang, Fernanda De La Torre, Cathy Fang, Andrzej Banburski-Fahey, Judith Amores, and Jaron Lanier. “Real-Time Animation Generation and Control on Rigged Models via Large Language Models,” arXiv (Cornell University, February 15, 2024), https://arxiv.org/pdf/2310.17838.pdf (Originally at the 37th Conference on Neural Information Processing Systems (NeurIPS) Workshop on ML for Creativity and Design in 2023).
[^WalkingNat]: Henri Lorach, Andrea Galvez, Valeria Spagnolo, et al., Walking naturally after spinal cord injury using a brain–spine interface, Nature 618, 126–133 (2023), https://doi.org/10.1038/s41586-023-06094-5