For humans, the ability to communicate and use language is instantiated not only in the vocal modality but also in the visual modality. The main examples of this are sign languages and gestures. Sign languages, the natural languages of Deaf communities, use systematic and conventionalized movements of the hands, face, and body for linguistic expression. Co-speech gestures, though non-linguistic, are produced in tight semantic and temporal integration with speech and constitute an integral part of language together with (...) speech. The articles in this issue explore and document how gestures and sign languages are similar or different and how communicative expression in the visual modality can change from being gestural to grammatical in nature through processes of conventionalization. As such, this issue contributes to our understanding of how the visual modality shapes language and the emergence of linguistic structure in newly developing systems. Studying the relationship between signs and gestures provides a new window onto the human ability to recruit multiple levels of representation in the service of using or creating conventionalized communicative systems. (shrink)

The aim of the present study is to evaluate Sonja Foss’s Rhetorical Schema for the Evaluation of Visual Imagery as well as reflect upon several points for further consideration; and finally suggest a renewed triadic approach as a method for analyzing art-relevant visual imagery. The triadic approach to be discussed assumes three correlative layers: the intellectual knowledge, function of the artistic content as the visual representational component, and aesthetics as modality. This study will include the analysis (...) of a print advertisement that used an artwork as its content of visual rhetoric, and this will inform further discussions on the proposed approach. The contribution of this renewed triadic approached to the field of visual rhetoric has the advantage of expanding and possibly improving rhetorical analysis methods of visual imagery. (shrink)

Three priming experiments were conducted to determine how information about the self from different sensory modalities/cognitive domains affects self-face recognition. Being exposed to your body odor, seeing your name, and hearing your name all facilitated self-face recognition in a reaction time task. No similar cross-modal facilitation was found among stimuli from familiar or novel individuals. The finding of a left-hand advantage for self-face recognition was replicated when no primes were presented. These data, along with other recent results suggest the brain (...) processes/represents information about the self in highly integrated ways. (shrink)

Training can influence behavioral performance and lead to brain reorganization. In particular, training in one modality, for example, auditory, can improve performance in another modality, for example, visual. Previous research suggests that one of the mechanisms behind this phenomenon could be the cross-modal recruitment of the sensory areas, for example, the auditory cortex. Studying expert musicians offers a chance to explore this process. Rhythm is an aspect of music that can be presented in various modalities. We designed (...) an fMRI experiment in which professional pianists and non-musicians discriminated between two sequences of rhythms presented auditorily or visually. Behavioral results showed that musicians performed in both visual and auditory rhythmic tasks better than non-musicians. We found no significant between-group differences in fMRI activations within the auditory cortex. However, we observed that musicians had increased activation in the right Inferior Parietal Lobe when compared to non-musicians. We conclude that the musicians’ superior visual rhythm discrimination is not related to cross-modal recruitment of the auditory cortex; instead, it could be related to activation in higher-level, multimodal areas in the cortex. (shrink)

This article presents a fresh perspective on the interplay among visual, verbal, and auditory modalities, positing that these modalities, as semogenic resources, compete to express dynamic meanings. The theoretical paradigm emphasizes that whether a modality or an element within a modality gets or loses semantic status, it will elicit an additional layer of social meaning to depict a comprehensive picture of a story together with an explicit semiotic meaning. The article adopts a qualitative method to analyze the (...) data, which are drawn from The Good Wife and My Roommate is a Gumiho and annotated in ELAN 6.3. It was found that modal competition can shed light on the dynamic meaning-making processes in semiotic and societal orientations. Modal competition may distort space and time of different stories, and reconstruct a different discursive spatio-temporal dimension in the TV world. It can diversify the dynamic orientations from New to Given in visual, verbal, and auditory texts of multimodal discourses to tell stories. Modal competition provides a lens to understand the multidimensional reality and to appreciate the aesthetics of a modern TV series. (shrink)

The distinction between true belief and knowledge is one of the most fundamental in philosophy, and a remarkable effort has been dedicated to formulating the conditions on which true belief constitutes knowledge. For decades, much of this epistemological undertaking has been dominated by a single strategy, referred to here as the modal approach. Shared by many of the most widely influential constraints on knowledge, including the sensitivity, safety, and anti-luck/risk conditions, this approach rests on a key underlying assumption — the (...) modal profiles available to known and unknown beliefs are in some way asymmetrical. The first aim of this paper is to deconstruct this assumption, identifying its plausibility with the way in which epistemologists frequently conceptualize human perceptual systems as excluding certain varieties of close error under conditions conducive to knowledge acquisition. The second aim of this paper is to then argue that a neural phase phenomenon indicates that this conceptualization is quite likely mistaken. This argument builds on the previous introduction of this neural phase to the context of epistemology, expanding the use of neural phase cases beyond relatively narrow questions about epistemic luck to a much more expansive critique of the modal approach as a whole. (shrink)

Although Pessoa et al. imply that many aspects of the filling-in debate may be displaced by a regard for active vision, they remain loyal to naive neural reductionist explanations of certain pieces of psychophysical evidence. Alternative interpretations are provided for two specific examples and a new category of filling-in (of visual detail) is proposed.

ABSTRACTBoth auditory and visual emotional memories can be made less emotional by loading working memory during memory recall. Taxing WM during recall can be modality specific (giving an audit...

This research delves into the potential of implementing artificial intelligence in architecture. It specifically provides a critical assessment of AI-enabled workflows, encompassing creative ideation, representation, materiality, and critical thinking, facilitated by prompt-based generative processes. In this context, the paper provides an examination of the concept of hybrid human–machine intelligence. In an era characterized by pervasive data bias and engineered injustices, the concept of hybrid intelligence emerges as a critical tool, enabling the transcendence of preconceived stereotypes, clichés, and linguistic prejudices. This (...) paper not only explores the applied and generative capacities of AI-enabled workflows but also suggests fundamental approaches that can enhance the creative process and confront the embedded biases and injustices within data-driven systems. (shrink)

In a recent paper, Brogaard presents counter-arguments to the conclusions of an experiment with blindsight subject GR. She argues that contrary to the apparent findings that GR’s preserved visual abilities relate to degraded visual experiences, she is in fact fully unconscious of the stimuli she correctly identifies. In this paper, we present arguments and evidence why Brogaard’s argument does not succeed in its purpose. We suggest that not only is relevant empirical evidence in opposition to Brogaard’s argument, her (...) argument misconstrues necessary criteria to decide whether a conscious experience is visual or not visual. (shrink)

Perceptual modalities have been traditionally considered the product of dedicated biological systems producing information for higher cognitive processing. Psychological and neuropsychological evidence is offered which undermines this point of view and an alternative account of modality from the enactive approach to understanding cognition is suggested. Under this view, a perceptual modality is a stable form of perception which is structured not just by the biological sensitivities of the agent, but by their goals and the set of skills or (...) expertise which they are deploying at a given time. Such a view suggests that there is no such thing as an experience that is purely visual, auditory, or otherwise modal and that our attempts to understand consciousness and the mind must be conducted within a framework that provides an account of embodied, goal-directed adaptive coping with the world. (shrink)

Maintaining and coordinating multiple task-sets is difficult and leads to costs, however task-switching training can reduce these deficits. A recent study in young adults demonstrated that this training effect occurs at an amodal processing level. Old age is associated with reduced cognitive plasticity and further increases the performance costs when mixing multiple tasks. Thus, cognitive aging might be a limiting factor for inducing cross-modal training effects in a task-switching environment. We trained participants, aged 62–83 years, with an auditory task-switching paradigm (...) over four sessions, to investigate whether training-related reductions in task-switching costs would also manifest in an untrained visual modality version of the task. Two control groups trained with single tasks or not trained allowed us to identify improvements specific to task-switching training. To make statistical evaluations of any age differences in training and cross-modal transfer, the data from the Kattner cohort were incorporated into the present analysis. Despite the tendency for older adults to respond more cautiously, task-switching training specifically led to a mixing cost reduction in both trained and untrained modalities, the magnitude of which was statistically similar regardless of age. In line with a growing body of research, we failed to observe any far transfer effects in measures of inhibition, working memory or fluid intelligence. Overall, we conclude that any apparent cognitive limitations associated with aging do not prevent cognitive control processes which support set-shifting from improving at an amodal level. (shrink)

The disjuncture between narrative representation and affective embodiment poses a central problematic for film analysis: how cinema articulates subjectivities resistant to stable definition amid historical crises. Zoltan Korda’s The Macomber Affair (1947), set against a charged backdrop of post-war anxiety and colonial ideology, provides a potent case study. Through close analysis of visual composition, modal configurations, and passional trajectories, this article argues the film stages masculine subjectivity not as a coherent, attainable identity, but as a precarious process of corporeal (...) signification. Synthesizing post-Greimasian semiotics (Zilberberg, Greimas, Fontanille) with psychoanalysis (Freud, Lacan, Mulvey), it analyses masculinity as a field of modal fluctuations, recalibrated through desire, knowledge, capacity, and obligation (vouloir, savoir, pouvoir, devoir), that gives form to unconscious drives within a triangulated field of gazes. The film’s formal architecture is central: the safari landscape operates as a colonial dispositif where masculine anxiety becomes legible, while its recursive temporality, by presenting death at the outset to inscribe meaning retroactively, destabilizes linear causality, foregrounding subjectivity as re-articulation over teleological attainment. Crucially, the article contends this individual psychodrama is predicated upon the ideological erasure of the “native gaze,” revealing the colonial frame as the narrative’s precondition. Ultimately, this study proposes a methodological shift from reading masculinity as thematic representation to apprehending it as a corporeal, affective process produced by cinema’s formal and colonial operations. (shrink)

Theories of embodied cognition hold that the conceptual system uses perceptual simulations for the purposes of representation. A strong prediction is that perceptual phenomena should emerge in conceptual processing, and, in support, previous research has shown that switching modalities from one trial to the next incurs a processing cost during conceptual tasks. However, to date, such research has been limited by its reliance on the retrieval of familiar concepts. We therefore examined concept creation by asking participants to interpret modality-specific (...) compound phrases (i.e., conceptual combinations). Results show that modality switching costs emerge during the creation of new conceptual entities: People are slower to simulate a novel concept (e.g., auditory jingling onion) when their attention has already been engaged by a different modality in simulating a familiar concept (e.g., visual shiny penny). Furthermore, these costs cannot be accounted for by linguistic factors alone. Rather, our findings support the embodied view that concept creation, as well as retrieval, requires situated perceptual simulation. (shrink)

Restrictivists hold that visual experience only represents low-level properties such as shape, spatial location, motion, color, etc. Expansionists contend that visual experience also represents high-level properties such as being a pine tree. I outline a new approach to support expansionism called the conflict cross-modal argument. What I call the conflict cross-modal effects occur when at least two perceptual systems disagree about some property belonging to a common stimulus, and this disagreement causes a change in the representational and phenomenal (...) content of the perceptual experience associated with one, or both, modalities. The conflict cross-modal argument works by accepting that if a property figures in a conflict cross-modal effect, then prima facie that property is a strong common sensible between the two modalities, and vision and another modality disagree about a property that is high-level for vision. After outlining this argument, and showing how it overcomes two obstacles that face traditional methods employed to defend expansionism, I turn to the well known Mcgurk effect as a case where vision and audition disagree about phoneme properties in order to employ my argument for the conclusion that phoneme properties are represented in visual experience. The upshot is that since phoneme properties are high-level for vision, we have an argument supporting expansionism. (shrink)

This article approaches sensory modalities as semiotically active factors and organizing principles in meaning-making. The focus will be on the special case where modalities mismatch in film – i.e., the soundtrack and visuals present contradictory meanings. The conflict can be characterized by the concept of synthesis that emerges in theories of Eisenstein, Barthes, Jakobson, Lotman, and cognitivists. The artistic functions of such synthesis will be discussed with the help of examples from selected feature films. In the first place, conflicting modalities (...) are inspected in the light of Juri Lotmanʼs theory of two incompatible, but still complementary languages that make up a mechanism for generating new information. In addition, the prospects of evaluating modality conflicts will be touched upon, dismissing synchrony and redundancy as the scale parameters, but acknowledging Lotmanʼs model of space as a primary modeling system that is capable of representing semantic conflicts. (shrink)

Most everyday tasks involve multiple modalities, which raises the question of how the processing of these modalities is coordinated by the cognitive system. In this paper, we focus on the coordination of visual attention and linguistic processing during speaking. Previous research has shown that objects in a visual scene are fixated before they are mentioned, leading us to hypothesize that the scan pattern of a participant can be used to predict what he or she will say. We test (...) this hypothesis using a data set of cued scene descriptions of photo-realistic scenes. We demonstrate that similar scan patterns are correlated with similar sentences, within and between visual scenes; and that this correlation holds for three phases of the language production process (target identification, sentence planning, and speaking). We also present a simple algorithm that uses scan patterns to accurately predict associated sentences by utilizing similarity-based retrieval. (shrink)

Early modern humans developed mental capabilities that were immeasurably greater than those of non-human primates. We see this in the rapid innovation in tool making, the development of complex language, and the creation of sophisticated art forms, none of which we find in our closest relatives. While we can readily observe the results of this high-order cognitive capacity, it is difficult to see how it could have developed. We take up the topic of cave art and archeoacoustics, particularly the discovery (...) that cave art is often closely connected to the acoustic properties of the cave chambers in which it is found. Apparently, early modern humans were able to detect the way sound reverberated in these chambers, and they painted artwork on surfaces that were acoustic “hot spots,” i.e., suitable for generating echoes. We argue that cave art is a form of cross-modality information transfer, in which acoustic signals are transformed into symbolic visual representations. This form of information transfer across modalities is an instance of how the symbolic mind of early modern humans was taking shape into concrete, externalized language. We also suggest that the earliest rock art found in Africa may constitute one of the first fossilized proxies for the expression of full-fledged human linguistic behavior. (shrink)

We investigate the dynamics of sensory integration for perceiving musical performance, a complex natural behavior. Thirty musically trained participants saw, heard, or both saw and heard, performances by two clarinetists. All participants used a sliding potentiometer to make continuous judgments of tension (a measure correlated with emotional response) and continuous judgments of phrasing (a measure correlated with perceived musical structure) as performances were presented. The data analysis sought to reveal relations between the sensory modalities (vision and audition) and to quantify (...) the effect of seeing the performances on participants' overall subjective experience of the music. In addition to traditional statistics, functional data analysis techniques were employed to analyze time-varying aspects of the data. The auditory and visual channels were found to convey similar experiences of phrasing but different experiences of tension through much of the performances. We found that visual information served both to augment and to reduce the experience of tension at different points in the musical piece (as revealed by functional linear modeling and functional significance testing). In addition, the musicians' movements served to extend the sense of phrasing, to cue the beginning of new phrases, to indicate musical interpretation, and to anticipate changes in emotional content. Evidence for an interaction effect suggests that there may exist an emergent quality when musical performances are both seen and heard. The investigation augments knowledge of human communicative processes spanning language and music, and involving multiple modalities of emotion and information transfer. (shrink)

It is being increasingly recognized that the Saussurean dictum of “the arbitrariness of the linguistic sign” is in conflict with the pervasiveness of the phenomenon commonly known as “sound symbolism”. After first presenting a historical overview of the debate, however, we conclude that both positions have been exaggerated, and that an adequate explanation of sound symbolism is still lacking. How can there, for example, be (perceived) similarity between expressionsand contents across different sensory modalities? We offer an answer, based on the (...) Peircian notion of iconic ground, and G. Sonesson’s distinction betweenprimary and secondary iconicity. Furthermore, we describe an experimental study, in a paradigm first pioneered by W. Kohler, and recently popularized by V.Ramachandran, in which we varied vowels and consonants in fictive word-forms, and conclude that both types of sounds play a role in perceiving an iconic ground between the word-forms and visual figures. The combination of historical conceptual analysis, semiotic explication and psychological experimentationpresented in this article is characteristic of the emerging paradigm of cognitive semiotics. (shrink)

Clinical and laboratory studies have demonstrated that executing a demanding dual-task while recollecting emotional memories weakens the emotional intensity and vividness of these memories. While t...

We address the thesis recently proposed by Andy Clark, that skill-mediated access to modality implies phenomenal feel. We agree that a skill theory of perception does indeed offer the possibility of a satisfactory account of the feel of perception, but we claim that this is not only through explanation of access to modality but also because skill actually provides access to perceptual property in general. We illustrate and substantiate our claims by reference to the recently proposed 'sensorimotor contingency' (...) theory of visual awareness. We discuss why this theory offers a distinctively attractive access-based approach to perceptual consciousness because it 'dereifies' experience and permits otherwise problematic aspects of phenomenal perceptual consciousness to be explained. We suggest our approach thus offers the prospect of 'naturalizing phenomenology'. (shrink)

Thepuzzle of cross‐modal shape experienceis the puzzle of reconciling the apparent differences between our visual and haptic experiences of shape with their apparent similarities. This paper proposes that we can resolve the cross‐modal puzzle by reflecting on another puzzle. Thepuzzle of perspectival characterchallenges us to reconcile the variability of shape experience through shifts in perspective with its constancy. An attractive approach to the latter puzzle holds that shape experience is complex, involving bothperspectivalaspects andconstantaspects. I argue here that parallel distinctions (...) between perspectival and constant aspects of shape experience arise in sight and touch, and that perspectival aspects are modality‐specific while at least some constant aspects are constitutively multisensory. I then address a powerful challenge to the idea that aspects of spatial phenomenology are shared cross‐modally. (shrink)

Visuo-motor adaptation to optical prisms displacing the visual scene (prism adaptation, PA) is a method used for investigating visuomotor plasticity in healthy individuals and, in clinical settings, for the rehabilitation of unilateral spatial neglect. In the standard paradigm, the adaptation phase involves repeated pointings to visual targets, while wearing optical prisms displacing the visual scene laterally. Here we explored differences in PA, and its aftereffects (AEs), as related to the sensory modality of the target. Visual, (...) auditory, and multisensory - audio-visual - targets in the adaptation phase were used, while participants wore prisms displacing the visual field rightwards by 10◦. Proprioceptive, visual, visual-proprioceptive, auditory-proprioceptive straight-ahead shifts were measured. Pointing to auditory and to audio-visual targets in the adaptation phase produces proprioceptive, visual-proprioceptive, and auditory-proprioceptive AEs, as the typical visual targets did. This finding reveals that cross-modal plasticity effects involve both the auditory and the visual modality, and their interactions (Experiment 1). Even a shortened PA phase, requiring only 24 pointings to visual and audio-visual targets (Experiment 2), is sufficient to bring about AEs, as compared to the standard 92-pointings procedure. Finally, pointings to auditory targets cause AEs, although PA with a reduced number of pointings (24) to auditory targets brings about smaller AEs, as compared to the 92-pointings procedure (Experiment 3). Together, results from the three experiments extend to the auditory modality the sensorimotor plasticity underlying the typical AEs produced by PA to visual targets. Importantly, PA to auditory targets appears characterized by less accurate pointings and error correction, suggesting that the auditory component of the PA process may be less central to the building up of the AEs, than the sensorimotor pointing activity per se. These findings highlight both the effectiveness of a reduced number of pointings for bringing about AEs, and the possibility of inducing PA with auditory targets, which may be used as a compensatory route in patients with visual deficits. (shrink)