When I am looking at my coffee machine that makes funny noises, this is an instance of multisensory perception – I perceive this event by means of both vision and audition. But very often we only receive sensory stimulation from a multisensory event by means of one sense modality, for example, when I hear the noisy coffee machine in the next room, that is, without seeing it. The aim of this paper is to bring together empirical findings about multimodal perception (...) and empirical findings about (visual, auditory, tactile) mental imagery and argue that on occasions like this, we have multimodal mental imagery: perceptual processing in one sense modality (here: vision) that is triggered by sensory stimulation in another sense modality (here: audition). Multimodal mental imagery is not a rare and obscure phenomenon. The vast majority of what we perceive are multisensory events: events that can be perceived in more than one sense modality – like the noisy coffee machine. And most of the time we are only acquainted with these multisensory events via a subset of the sense modalities involved – all the other aspects of these multisensory events are represented by means of multisensory mental imagery. This means that multisensory mental imagery is a crucial element of almost all instances of everyday perception. (shrink)

During the last decade, one source of debate in argumentation theory has been the notion that there are different modes of arguing that need to be distinguished when analyzing and evaluating arguments. Visual argument is often cited as a paradigm example. This paper discusses the ways in which it and modes of arguing that invoke non-verbal sounds, smells, tactile sensations, music and other non-verbal entities may be defined and conceptualized. Though some attempts to construct a ‘multimodal’ theory of argument are (...) criticized, it advocates for an argumentation theory that makes room for visual arguing and for other non-verbal modes that have not been explored in depth. In the process, the paper provides a method for identifying the structure of multimodal arguments and argues that adding modes to our theoretical tool box is an important step toward a comprehensive account of argument. (shrink)

The aim of this paper is to argue that our experience of artworks is normally multimodal. It is the result of perceptual processing in more than one sense modality. In other words, multimodal experience of art is not the exception; it is the rule. I use the example of music in order to demonstrate the various ways in which the visual sense modality influences the auditory processing of music and conclude that this should make us look more closely at our (...) practices of engaging with artworks. (shrink)

There has been a lot of discussion about how the cognitive penetrability of perception may or may not have important implications for understanding perceptual justification. The aim of this chapter is to argue that a different set of findings in perceptual psychology poses an even more serious challenge to the very idea of perceptual justification. These findings are about the importance of perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality (such as amodal completion (...) and multimodal completion). These findings show that everyday perception is in fact a mixture of sensory-stimulation-driven perceptual processing and perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality and that we have strong reasons to doubt the epistemic pedigree of the latter process. The implication of this is not that we should become skeptics or deny the possibility of perceptual justification. It is, rather, that the only way in which we can understand when and whether a perceptual state justifies beliefs is by paying close attention to empirical facts about the reliability of perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality. In this sense (a very narrow sense) epistemology needs to be naturalized. (shrink)

We present an embedding of quantified multimodal logics into simple type theory and prove its soundness and completeness. A correspondence between QKπ models for quantified multimodal logics and Henkin models is established and exploited. Our embedding supports the application of off-the-shelf higher-order theorem provers for reasoning within and about quantified multimodal logics. Moreover, it provides a starting point for further logic embeddings and their combinations in simple type theory.

ABSTRACTCritical Discourse Studies positions itself as a critical, transformative practice that seeks to expose the ways in which discourse is able to constitute social, political, economic, gendered, racial, and sexual inequalities as normal and unremarkable. In this respect, CDS is an expressly political and therefore ethical project. Nevertheless, this article posits that, quite remarkably, the ethical frameworks that guide CDS have remained largely under-theorized and taken for granted. Accordingly, this paper seeks to contribute to this much-needed area of theoretical inquiry (...) by offering a brief discussion of how CDS has tended to define its own criticality and identifies three problems that emerge from how CDS practitioners have tended to formulate the praxis of critique: namely, the tendency to conceive of CDS as a demystificatory practice, the implicit logocentrism of modeling ethics upon the Habermasian Ideal Speech Situation, and finally, the deontological nature of discourse ethics. Each of the three problems is discussed in relation to a brief multimodal critical discourse analysis of three online employer profile videos. As such, the study is offered to demonstrate not only the analytical strength of multimodal analysis but also the political-ethical relations beyond Discourse Ethics that are immanent in such analyses. (shrink)

ABSTRACTThis study investigates the multimodal strategies used in social marketing to emotionally manipulate and persuade children and their parents to adopt healthier lifestyles. The UK government's anti-obesity ‘Change4Life’ campaign is an example of the increasing reliance on ‘nudge’ tactics in public policy. This approach seeks to develop a range of intervention strategies, inspired by behavioural psychology, in order to change people's behaviours without them necessarily recognising this has happened. I argue it is a form of governmentality which uses subtle semiotic (...) techniques to secure voluntary compliance with policy goals. I bring the concepts of biopolitics and governmentality into dialogue with multimodal critical discourse analysis to examine how this is realised in a series of cartoon government adverts. Three intersecting strategies are identified: the representation of lifestyles as delinquent a discourse of risk and threat mobilised through emotional manipulation and a discourse of ‘smarter’ consumerism. The increasing popularity of nudge over the last decade coincides with a sustained programme of fiscal austerity which disproportionately penalises the poor, while food poverty and social inequality have increased. I argue that it helps legitimate and instantiate neoliberal political rationalities by privatising responsibility for public health care. Despite the importance it places on communication strategies, critical scholarship on nudge has not yet investigated the linguistic techniques it uses in practice. This paper addresses that omission with a detailed multimodal analysis of the UK's longest running policy nudge. (shrink)

Repeated comparatives joined by the conjunction and (for instance more and more, bigger and bigger, etc.) are used to express a gradually increasing/decreasing degree of quality or quantity. The repetition in these phrases reveals an even higher/lower degree of the quality or quantity than with a single comparative form. The aim of this study is to determine whether this gradation is associated with specific gesture sequences, whether the progressive aspect inherent in the construction’s semantics is also expressed in the verb (...) of the clause or in gesture, and finally whether the semantic prominence expressed in the phrase finds a correlation with prosodic emphasis or emphasis marked with the use of bodily behaviour like head and eyebrow movements. In the perspective of CxG, the study aims at exploring whether these forms are entrenched in usage as multimodal constructions and how speakers express gradation in space. Results show that although gestures and prosodic emphasis are not frequent enough to consider [COMP 1 and COMP 1] as a multimodal construction, some gesture patterns are certainly more frequent than others, especially the use of iconic gestures in sequences that match the verbal repetition of the comparative. (shrink)

We study straightforward embeddings of propositional normal multimodal logic and propositional intuitionistic logic in simple type theory. The correctness of these embeddings is easily shown. We give examples to demonstrate that these embeddings provide an effective framework for computational investigations of various non-classical logics. We report some experiments using the higher-order automated theorem prover LEO-II.

The present paper addresses the notion of multimodal constructions. It argues that Tell me about it is a multimodal construction that consists of a fixed spoken and a variable, but largely obligatory multimodality slot on the formal side of the construction. To substantiate this claim, the paper reports on an experiment that shows that, first, hearers experience difficulties in interpreting Tell me about it when it is neither sequentially nor multimodally marked as either requesting or stance-related and, second, hearers (...) considerably rely on multimodal features when a sequential context is missing. In addition, the experiment also shows that the more features are used, the better hearers get at guessing the meaning of Tell me about it. These results suggest that, independent of the question of whether the multimodal features associated with requesting or stance-related Tell me about it are non-spoken, unimodal constructions themselves (like a raised eyebrows construction), a schematic multimodality slot might be part of the constructions. (shrink)

By the time they reach early adulthood, English speakers are familiar with the meaning of thousands of words. In the last decades, computational simulations known as distributional semantic models have demonstrated that it is possible to induce word meaning representations solely from word co-occurrence statistics extracted from a large amount of text. However, while these models learn in batch mode from large corpora, human word learning proceeds incrementally after minimal exposure to new words. In this study, we run a set (...) of experiments investigating whether minimal distributional evidence from very short passages suffices to trigger successful word learning in subjects, testing their linguistic and visual intuitions about the concepts associated with new words. After confirming that subjects are indeed very efficient distributional learners even from small amounts of evidence, we test a DSM on the same multimodal task, finding that it behaves in a remarkable human-like way. We conclude that DSMs provide a convincing computational account of word learning even at the early stages in which a word is first encountered, and the way they build meaning representations can offer new insights into human language acquisition. (shrink)

This chapter explores the multimodal representation of personal life and cultural identity in the Instagram posts of four gaming influencers from diverse backgrounds: PewDiePie, Valkyrae, Ninja, and Pokimane. Utilising Gunther Kress and Theo van Leeuwen's Social Semiotics framework, the analysis focuses on how these influencers blend visual and textual elements in non-gaming posts to depict themes like family life, professional achievements, and personal moments. This study highlights how influencers use Instagram's platform-specific features to foster a sense of authenticity and relatability (...) with their audience. The findings support the argument that these gaming influencers effectively deploy multimodal communication to connect with their audiences at a human (sometimes parasocial) level. Furthermore, the study reveals how these multimodal strategies contribute to broader discussions on digital communication. (shrink)

One way to characterize the special relation that one has to one's own body is to say that only one's body appears to one from the inside. Although widely accepted, the nature of this specific experiential mode of presentation of the body is rarely spelled out. Most definitions amount to little more than lists of the various body senses (including senses of posture, movement, heat, pressure, and balance). It is true that body senses provide a kind of informational access to (...) one's own body, which one has to no other bodies, by contrast to external senses like vision, which can take many bodies as their object. But a theory of bodily awareness needs to take into account recent empirical evidence that indicates that bodily awareness is infected by a plague of multisensory effects, regardless of any dichotomy between body senses and external senses. Here I will argue in favour of a multimodal conception of bodily awareness. I will show that the body senses fail to fully account for the content of bodily experiences. I will then propose that vision helps compensate for the insufficiencies of the body senses in people who can see. I will finally argue that the multimodality of bodily experiences does not prevent privileged access to one's body. (shrink)

The article investigates a multimodal practice for self-selecting observed in a video-taped corpus of work meetings: the use of pointing gestures predicting possible turn completions and projecting the emergence of possible next speakers. This practice is analyzed in various sequential positions, namely at turn beginnings and at pre-beginnings. It displays recipients' practical online turn parsing, and their orientation to transition spaces, and to TCU, completions in a visible, recognizable, public way. It shows the emergent and progressive establishment of speakership, exploiting (...) both systematic features of turn-taking and specific features of the interactional space. The article explores the emic, locally situated and contingent definition of speakership by considering not only where pointing gestures begin but also where they end: pointing does not only end in pre-completion positions, projecting turn end, but can also persist through sequences, showing speaker's orientations to her turn's sequential implicativeness. Thus, by observing a particular multimodal practice within a specific interactional setting, the article explores participants' orientations to the rights and obligations associated with talk-related categories such as `non current speaker', `possible next speaker', `incipient speaker' and `current speaker'. (shrink)

Traditional medical Artificial Intelligence models, approved for clinical use, restrict themselves to single-modal data e.g. images only, limiting their applicability in the complex, multimodal environment of medical diagnosis and treatment. Multimodal Transformer Models in healthcare can effectively process and interpret diverse data forms such as text, images, and structured data. They have demonstrated impressive performance on standard benchmarks like USLME question banks and continue to improve with scale. However, the adoption of these advanced AI models is not without challenges. While (...) multimodal deep learning models like Transformers offer promising advancements in healthcare, their integration requires careful consideration of the accompanying ethical and environmental challenges. (shrink)

Joaquim Braga deals with the role of sound in multimodal environments and multimodal surfaces. He argues that imagination not only bridges sensory dimensions that are absent from perception but also is central to “the relationship of presence,” that is, the connection between presently perceived sensory input. This involves assessing both the individuating dimension and the relational disposition of sound, which allows sound—through the activation of imagination—to develop sensory relations. Central to his discussion of the function and contribution of sound in (...) different media is the process of “sonic individuation”—a process of individuation of different modalities—that is presented as a precondition for the creation of relations between these modalities. (shrink)

From spider dances to human language, multimodality is ubiquitous in natural communication systems. Much scholarship has been devoted to investigating why multimodality evolved and the role it plays in communication. Here, we highlight the role of multimodality in safeguarding the most fundamental prerequisite of all functioning, extant communication systems: honesty. We begin by introducing the arms race between honesty and deception in natural communication systems, and the critical role socially-mediated controls can play in maintaining signal honesty when (...) classic, intrinsic costs are not sufficient. We next introduce three ways by which multimodality buffers signal honesty by 1) providing insurance against signal unreliability in dynamic environments, 2) forming an honest, multimodal gestalt with which to cross-validate signal honesty, and 3) increasing signal complexity, making the entire signal harder to fake. We then discuss the case of highly cooperative societies, with human language emphasized, and argue that signal honesty is important especially in complex and cooperative societies wherein the need to cooperate and be accepted as part of the group may supersede honesty. Finally, we propose future directions wherein human and non-human communication research could expand beyond the well trodden realms of competition and mate attraction to investigate the role of multimodality and honesty in cooperative, “cheap” signals, and emphasize the importance of drawing from both the human and non-human literatures in investigating the forces that have shaped the evolution of communication. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the multimodal choices (...) that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

According to a dialectical approach to argumentation, a single argument can be seen as a dialogical "Why? Because!" sequence. Does this also apply to multimodal arguments? This paper focuses on multimodal arguments with a predominantly visual character and shows that dialogues are helpful for identifying and reconstructing arguments in multimodal communication. To include nonverbal arguments in dialectical argumentation theory, it is proposed to regard dialogue as mode-fluid. The account of multimodal argument as dialogue will be compared with Champagne and Pietarinen’s (...) account of visual argument as movement. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to multimodal experiences of pain. (...) Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

Multimodal argumentation-combining textual, visual, and symbolic reasoning-has become an essential framework for analyzing and interpreting legal codes. However, its use in legal scholarship faces significant challenges due to the rigid nature of legal language and its strong textual focus. This study critically explores the difficulties in implementing multimodal argumentation for legal code interpretation, emphasizing its effects on clarity, accessibility, and adaptability across various legal systems. Methodologically, the research utilizes qualitative content analysis of secondary data, including legal documents, judicial interpretations, and (...) scholarly literature on semiotics and legal theory. By examining secondary data from comparative legal analysis, case law, and studies on legal semiotics, this research identifies gaps in current literature, particularly the absence of a cohesive framework for integrating multimodal argumentation into legal practice. The findings indicate that although multimodal approaches present new opportunities for improving the understanding and use of legal codes, they also bring up significant issues related to standardization, bias, and the risk of misinterpreting non-textual elements. This paper adds to legal scholarship by suggesting an evaluative model that reconciles the interpretive advantages of multimodality with the importance of preserving legal certainty. The conclusion highlights the need for interdisciplinary collaboration to create strong frameworks that incorporate multimodal argumentation while upholding the fundamental principles of legal reasoning. (shrink)

Philosophers and cognitive scientists of perception by custom have investigated individual sense modalities in relative isolation from each other. However, perceiving is, in a number of respects, multimodal. The traditional sense modalities should not be treated as explanatorily independent. Attention to the multimodal aspects of perception challenges common assumptions about the content and phenomenology of perception, and about the individuation and psychological nature of sense modalities. Multimodal perception thus presents a valuable opportunity for a case study in mature interdisciplinary cognitive (...) science. This chapter aims to raise these issues against the background of unimodal approaches in the study of perception. It presents some of the central empirical findings concerning multimodality, and it explains the philosophical implications of these findings. Foremost, it aims to encourage and open avenues for future research. (shrink)

This book explores the interdisciplinarity of semiotics and communication studies, comprising both theoretical explorations and semiotic applications to communication with theoretical bearings. These disciplines have generally been understood as mutually implicit, but there still are many unexplored research avenues in this area, particularly on a conceptual level. The book offers broad insights into the epistemological relations between semiotics and other approaches to communication from perspectives such as sociology, philosophy of language and communication theory. As such, it sheds light on the (...) communication of knowledge. Semiotics is currently enjoying increasing popularity within the humanities and social sciences. Understood as relational logic or hermeneutics, semiotics fundamentally implies certain positions with regard to communication. Because of the generality and conceptual vagueness of semiosis and communication, how one elucidates the other is still an underexplored theme. With some pioneering studies of this relation, the books examines various fields, such as language, code, learning, embodiment, political communication, media, cinema, cuisine, multimodality and intertextuality. (shrink)

Informal Logic and Argumentation Theory claim to accept a sense of ‘argument’ that is broader than the sense used in classical logic and mainstream philosophy of language. In this allegedly more general sense, pictures, sounds, gestures, maps, etc. can be premises of arguments, even if conclusions are usually seen as propositional. In this chapter, I argue that multimodality is a well-known phenomenon among linguists and among mathematicians. Relevance Theorists and Truth-Conditional Pragmatists have discussed the impact of gestures and body (...) language in communication and the essential ineffability of what they call ‘expressives’. Mathematicians and philosophers of mathematics, in turn, have discussed the relevance of figures and diagrams in proofs. I will argue that perceptible items do not have logical properties. Logical properties are properties of a certain kind of abstract entities whose identification follows the organic intuition, (OI). Argumentation Theory has not provided arguments that advised a revision of Frege’s view that judgeable contents are the sole concern of logic. Still, the multimodality debate brings to the fore significant issues related to how information flows; among these issues, the insight that different representation modes open up different affordances and are differentially suitable for particular communicative aims. (shrink)

We introduce a multimodal logic for order of magnitude reasoning which considers a new logic-based alternative to the notion of closeness, we provide an axiom system and prove its soundness and completeness.

The paper analyses aspects of multilingual and multimodal narratives employed during the COVID-19 lockdown, attempting to identify how they were used in professional and public environments. More specifically, the paper looks at editing choices, photography, design, drawing, colour, and writing to get a better understanding of multimodal communication, semiotics, image analysis, and their correlation with the written text and the way it has been perceived and interpreted, be it in online or print-based contexts. Methodologically, this is a contrastively inspired research (...) project, looking specifically at Portuguese-speaking contexts of the Island of Madeira and English-speaking contexts of the United Kingdom, taking into account instructions about the COVID-19 pandemic. These are mainly bilingual English–Portuguese and multilingual texts, multimodal manifestations, and communicative and semiotic effectiveness. (shrink)

Analysing film. Distinguishing the filmic contribution to meaning -- Examples of filmic "textual organisation" -- Redrawing boundaries -- Organisation of the book -- Semiotics and documents. Semiotics and its relations to film -- The nature of discourse semantics -- The film as cinematographic document -- A combined view: filmic documents for filmic discourse -- Constructing the semiotic mode of film. Semiotic multimodality -- The internal organisation of semiotic strata -- Composing and combining semiotic modes -- Materiality and "epistemological commitment" (...) -- Christian Metz and the grande syntagmatique of the image track. The original model -- Two examples of analysis with the grande syntagmatique -- Revisions and rebuttals -- Foundations for analysis: filmic units. The basic units of film: preliminaries -- Audiovisual iconic representations -- Perception, perceptual realism and reliable measurement -- Multiplicity: from perception to discourse -- Filmic units revisited: discourse-motivated definitions -- The paradigmatic organisation of film. Beyond Metz: towards a grande paradigmatique -- Capturing discourse dependency structures in film -- The paradigmatic dimensions of projection, taxis and plane -- Two examples of paradigmatic analysis -- Summary and conclusions. (shrink)

The use of multimodal systems brings great prospects in solving complex problems unsolvable by the usual unimodal approaches. Therefore, a multimodal video and radar system are proposed to design a multimodal machine-learning problem on edge devices. The system’s architecture uses Docker containers to capture knowledge under models and processes, allowing the system to be easily managed. Furthermore, the importance of object detection is enhanced in the proposed system, as the identification and localization of objects in different data modalities are critical (...) components of several multimodal machine-learning tasks. Hence, it is presented as an overall description of the architecture and a discussion of the data pipeline of this system. It is approached by the challenge of data alignment using homographic transformations with video camera and radar data, as well as using the system calibration to reach the data fusion and consequent predictions. It highlights the advantages of multimodal systems in dealing with complex and dynamic environments and provides a general approach to multimodal machine-learning problems on edge devices. (shrink)

This paper shows that some classes of multimodal paraconsistent logics endowed with weak forms of negation are incompletable with respect to Kripke semantics. The reach of such incompleteness is discussed, and we argue that this shortcoming, more than just a logical predicament, may be relevant for attempts to characterize quantum logics and to handle quantum information and quantum computation.

In the last two decades modal logic has undergone an explosive growth, to thepointthatacompletebibliographyofthisbranchoflogic,supposingthat someone were capable to compile it, would?ll itself a ponderous volume. What is impressive in the growth of modal logic has not been so much the quick accumulation of results but the richness of its thematic dev- opments. In the 1960s, when Kripke semantics gave new credibility to the logic of modalities? which was already known and appreciated in the Ancient and Medieval times? no one could (...) have foreseen that in a short time modal logic would become a lively source of ideas and methods for analytical philosophers,historians of philosophy,linguists, epistemologists and computer scientists. The aim which oriented the composition of this book was not to write a new manual of modal logic buttoo?ertoeveryreader,evenwithnospeci?cbackground in logic, a conceptually linear path in the labyrinth of the current panorama of modal logic. The notion which in our opinion looked suitable to work as a compass in this enterprise was the notion of multimodality, or, more speci?cally, the basic idea of grounding systems on languages admitting more than one primitive modal operator. (shrink)

Recently, it has been observed that the usual type-theoretic restrictions are not enough to block certain paradoxes involving two or more predicates. In particular, when we have a self-referential language containing modal predicates, new paradoxes might appear even if there are type restrictions for the principles governing those predicates. In this article we consider two type-theoretic solutions to multimodal paradoxes. The first one adds types for each of the modal predicates. We argue that there are a number of problems with (...) most versions of this approach. The second one, which we favour, represents modal notions by using the truth predicate together with the corresponding modal operator. This way of doing things is not only useful because it avoids multimodal paradoxes, but also because it preserves the expressive capacity of the language. As an example of the sort of theory we have in mind, we provide a type-theoretic axiomatization that combines truth with necessity and knowledge. (shrink)

Notre article vise à décrire la constitution et l’annotation d’un corpus multimodal et multilingue (français, anglais et chinois mandarin) pour l’étude de l’acquisition d’une langue seconde et du développement professionnel en enseignement des langues. Le corpus a été construit dans le cadre d’un projet de recherche dont l’objectif principal est de déterminer quelles compétences techno-sémio-pédagogiques peuvent être développées de manière informelle et lesquelles nécessitent une formation formelle. Nous expliquons le cadre théorique adopté, caractérisé par une approche écologique de l’environnement interactif. (...) Nous illustrons ensuite les procédures de collecte de données audio, vidéo et oculométriques. Nous détaillons également l’annotation des données brutes pour produire un corpus d’analyse s’appuyant sur différents outils d’annotation automatiques et semi-automatiques. Enfin, nous expliquons comment un tel corpus nous permet d’étudier le développement de la compétence multimodale des apprenants et de la compétence techno-pédagogique des tuteurs et pointons vers d’autres questions de recherche possibles. (shrink)

The 9/11 attacks in the USA had profound political consequences at both domestic and international levels. Specific and controversial policy developments were pursued requiring substantial legitimation to find acceptance. A prime example was the USA PATRIOT Act, which was passed in the immediate aftermath of 9/11 and subsequently received considerable critique due to the sweeping nature of its redefinition of what was acceptable in the cause of ‘fighting terror’. The media, and their construal of events and policies, played a significant (...) role in the necessary process of legitimation. In this paper, we explore how one particular area of the media, the superhero comic book, was also active in this regard. By focusing specifically on the multimodal meaning construction within Marvel's Civil War comics series, we suggest how a close multimodal discourse analysis can shed further light on the mechanisms by which particular discourse contexts of public opinion are propagated. (shrink)

A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A “dialectic” solution to this age-old debate has now been gaining ground: language was fully multimodal from the start and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several lines of (...) interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual and multimodal. Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional profiles of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario. (shrink)

From the perspective of distributed cognition I will stress how abduction is essentially multimodal, in that both data and hypotheses can have a full range of verbal and sensory representations, involving words, sights, images, smells, etc., but also kinesthetic – related to the ability to sense the position and location and orientation and movement of the body and its parts – and motor experiences and other feelings such as pain, and thus all sensory modalities. The presence of kinesthetic and motor (...) aspects demonstrates that abductive reasoning can be manipulative. We can also see, in this regard, how implicit factors take part in the abductive procedure, which consequently acquires the character of a kind of “thinking through doing”. This paper further describes 1) the fact that hypotheses in science can be built through different cognitive mediators and so they can also model the same cognitive aspect in different ways; how they can be carriers/ producers of knowledge in a multimodal way; 2) the problem of the possible non-explanatory and instrumental nature of abductive reasoning and the analysis of the consequences for induction; 3) the role of manipulative abduction in building new evidence/ experiments and how they trigger smart inductive inferences. (shrink)

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, pragmatics, (...) and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

In this paper we compare grammatical inference in the context of simple and of mixed Lambek systems. Simple Lambek systems are obtained by taking the logic of residuation for a family of multiplicative connectives /, *, \, together with a package of structural postulates characterizing the resource management properties of the * connective. Different choices for Associativity and Commutativity yield the familiar logics NL, L, NLP, LP. Semantically, a simple Lambek system is a unimodal logic: the connectives get a Kripke (...) style interpretation in terms of a single ternary accessibility relation modeling the notion of linguistic composition for each individual system.The simple systems each have their virtues in linguistic analysis. But none of them in isolation provides a basis for a full theory of grammar. In the second part of the paper, we consider two types of mixed Lambek systems.The first type is obtained by combining a number of unimodal systems into one multimodal logic. The combined multimodal logic is set up in such a way that the individual resource management properties of the constituting logics are preserved. But the inferential capacity of the mixed logic is greater than the sum of its component parts through the addition of interaction postulates, together with the corresponding interpretive constraints on frames, regulating the communication between the component logics.The second type of mixed system is obtained by generalizing the residuation scheme for binary connectives to families of n-ary connectives, and by putting together families, of different arities in one logic. We focus on residuation for unary connectives, hence on mixed frames, as these already represent the complexities in full. We prove a number of elementary logical results for unary families of residuated connectives and their combination with binary families. The existing proposals for unary ‘structural modalities’ are situated within this general framework, and a number of new linguistic applications is given. (shrink)

ABSTRACTThis article offers the first large-scale study of a multimodal corpus of 210 advertisements. First, the reader is presented with a description of the corpus in terms of the distribution of conceptual operations and use of modal cues. Subsequently, the weight of mode and marketing strategy to trigger more or less amounts of conceptual complexity is analyzed. This corpus-based survey is complemented with the qualitative analysis of three novel metaphor–metonymy interactions that stem from the data and that have not yet (...) been surveyed in multimodal use. The results show that metaphtonymy is the most frequet conceptual operation in the corpus; that there is a significant effect of the use of modes in the activation of different amounts of conceptual complexity; and that the type of advertised product and the marketing strategy has no significant effect on the number and complexity of conceptual mappings in the advert... (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. /gi/) and (...) auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

That human social interaction involves the intertwined cooperation of different modalities is uncontroversial. Researchers in several allied fields have, however, only recently begun to document the precise ways in which talk, gesture, gaze, and aspects of the material surround are brought together to form coherent courses of action. The papers in this volume are attempts to develop this line of inquiry. Although the authors draw on a range of analytic, theoretical, and methodological traditions (conversation analysis, ethnography, distributed cognition, and workplace (...) studies), all are concerned to explore and illuminate the inherently multimodal character of social interaction. Recent studies, including those collected in this volume, suggest that different modalities work together not only to elaborate the semantic content of talk but also to constitute coherent courses of action. In this introduction we present evidence for this position. We begin by reviewing some select literature focusing primarily on communicative functions and interactive organizations of specific modalities before turning to consider the integration of distinct modalities in interaction. (shrink)

The question, “What norms are appropriate for the evaluation of the probative merits of visual arguments?” underlies the investigation of this paper. The notions of argument and of multimodal visual argument employed in the study are explained. Then four multimodal visual arguments are analyzed and their probative merits assessed. It turns out to be possible to judge these qualities using the same criteria that apply to verbally expressed arguments. Since the sample is small and not claimed to be representative, this (...) finding can at best be regarded as suggestive for the probative assessment of multimodal visual arguments in general. (shrink)

Taste is a central sense for humans and animals, and it has been largely studied either from physiological and neurological approaches or from socio-cultural ones. This paper adopts another view, focused on the activity of tasting rather than on the sense of taste, approached within the perspective of ethnomethodology and multimodal conversation analysis. This view addresses the activity of tasting as it is interactionally organized in specific social settings, observed in a naturalistic way, on the basis of video recordings. Focusing (...) on video recorded improvised tastings of cheese in gourmet shop encounters, the paper offers a systematic analysis of the way in which tasting is orderly achieved in an intersubjective way. It follows the various steps characterizing tasting, from the invitation to taste, to the grasping of a bit to taste, which is put in the mouth, chewed, and swallowed; it details how an interactional moment offering the taster a priviledged, individual, focused space in which to devote exclusive attention to the object tasted is actively tailored by all parties. By contrast, the completion of tasting is marked by a return to mutual gaze, the animation of facial expressions and nods, and the final production of a judgment of taste. By offering a systematic reconstruction of how these tasting moments are organized, the paper invites to a multimodal approach of sensoriality in social interaction. (shrink)

Jacopo Castaldi’s Multimodality, Ideology, and Manipulation makes a timely and critical intervention into debates on media, ideology, and public perception. By examining the British Broadcasting Co...

There has been a lot of discussion about how the cognitive penetrability of perception may or may not have important implications for understanding perceptual justification. The aim of this paper is to argue that a different set of findings in perceptual psychology poses an even more serious challenge to the very idea of perceptual justification. These findings are about the importance of perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality (such as amodal completion (...) and multimodal completion). I argue that these findings show that everyday perception is in fact a mixture of sensory stimulation-driven perceptual processing and perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality and we have strong reasons to doubt the epistemic pedigree of the latter process. The implication of this is not that we should become skeptics or deny the possibility of perceptual justification. It is, rather, that the only way in which we can understand when and whether a perceptual state justifies beliefs is by paying close attention to empirical facts about the reliability of perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality. In this sense (a very narrow sense) epistemology needs to be naturalized. (shrink)

Many philosophers use findings about sensory substitution devices in the grand debate about how we should individuate the senses. The big question is this: Is “vision” assisted by (tactile) sensory substitution really vision? Or is it tactile perception? Or some sui generis novel form of perception? My claim is that sensory substitution assisted “vision” is neither vision nor tactile perception, because it is not perception at all. It is mental imagery: visual mental imagery triggered by tactile sensory stimulation. But it (...) is a special form of mental imagery that is triggered by corresponding sensory stimulation in a different sense modality, which I call “multimodal mental imagery.”. (shrink)

This study proposes a multimodal fusion model to account for the cognitive mechanisms involving 56 political cartoons with regard to U.S. beef import issues as reported in two dominant Taiwanese newspapers, the Liberty Times and United Daily News. Specifically, this study claims that multimodal fusion model evolves from two metonymic-metaphoric networks, i.e., related metonymic network and diversified metaphoric network, and combines the conceptual, visual, and verbal modes. Our analysis demonstrates that multimodal fusion is a significant and recurrent representation technique in (...) the genre of political cartoon and has the cognitive function of encapsulating the abstract complex political debates efficiently with irony and humorous effect. Furthermore, our analysis shows the important role of metonymy and demonstrates how metonymies and metaphors are interwoven in the process of multimodal fusion, which underlies the metaphorical mappings of conceptual scenarios related to “POLITICS IS.. (shrink)

Name der Zeitschrift: Semiotica Jahrgang: 2018 Heft: 220 Seiten: 41-67.

Journal Name: Cognitive Linguistics Issue: Ahead of print.

This paper considers a range of so-called image macro Internet memes and describes them as emerging multimodal constructions relying as much on image as on text, and apportioning roles to images much like constructional slots, for instance to fill in a subject role in a subjectless clause, or even to provide the main clause content to a textually given when-clause. In addition to existing or partially altered linguistic constructions, many examples also rely on specific top text/bottom text division of labor, (...) and crucially depend on frame metonymy, with limited formal means quickly cueing richly detailed frames (for instance by using iconic images). The popularity of memes, forming series and cycles of iterations and remixes, and their role in establishing and maintaining discourse communities seems to be driven by a need to express and reconstrue viewpoints, often starting from ideas, affects or stereotypes assumed to be intersubjectively shared with viewers, whose responses they solicit. This paper argues that a proper description of Internet memes of the type considered requires a construction grammar approach, complemented by an understanding of viewpoint dynamics in terms of a Discourse Viewpoint Space regulating the network of spaces and viewpoints. (shrink)

Metaphor pervades discourse and may govern how we think and act. But most studies only discuss its verbal varieties. This book examines metaphors drawing on combinations of visuals, language, gestures, sound, and music. Investigated texts include advertising, political cartoons, comics, film, songs, and oral communication. Where appropriate, the influence of genre and cultural factors is thematized.