The present article is devoted to a logical treatment of some fundamental concepts involved in responsibility attribution. We specify a theoretical framework based on a language of temporal deontic logic with agent-relative operators for deliberate causal contribution. The framework is endowed with a procedure to solve normative conflicts which arise from the assessment of different normative sources. We provide a characterization result for a basic system within this framework and illustrate how the concepts formalized can be put (...) at work in the analysis of examples of legal reasoning. (shrink)

This article explores the socio-discursive competence of young children in Galician pre-schools. In particular, it deals with the way in which children – aged from 2;10 to 4;05 years – combine embodied actions and verbal resources to co-narrate stories with peers and adults. Using an audiovisual corpus of naturally occurring interactions, we have conducted a qualitative and multimodal analysis, observing how children react to diverse footings and negotiate participation frameworks in multiparty interactions. The findings suggest three progressive stages in (...) the development of socio-discursive competence in which children use different strategies: the stage of interactive proto-narrations, the stage of collaborative narrations and the stage of polyphonic monologue narrations. Our detailed examination reveals a progressive increase in the diversity of the participation frameworks represented by children, as well as in the complexity of the multimodal resources they use, and a rise in children’s grammatical repertoire linked to an improvement in children’s pragmatic skills. (shrink)

This book presents the methodological framework of combining Multimodal Conversation Analysis (MCA) with Interpretative Phenomenological Analysis (IPA) to interpretively analyse translanguaging practices in educational contexts. Beginning with an overview of the three uses of translanguaging - translanguaging as a theory of language, as a pedagogical practice and as an analytical perspective - the book goes on to critically examine the different methodological approaches for analysing translanguaging practices in multilingual classroom interactions. It explains how MCA and IPA are useful (...) methodologies for understanding how and why translanguaging practices are constructed by participants in the classroom and discusses types of data collected and data collection procedures. The author, Kevin W. H. Tai, shows how combining these approaches enables researchers to study how translanguaging practices are constructed in multilingual classrooms and how teachers make sense of their own translanguaging practices at particular moments of classroom interaction. This detailed and concise guide is indispensable for students, practitioners, policymakers, and researchers from across the globe, particularly those working in the fields of applied linguistics and language education. (shrink)

This paper presents a multimodal framework to the analysis of meaning-making in theatrical performances, conceptualizing them as multimodal discourses. It posits that the adoption of a discourse-oriented approach to theatrical performances addresses the limitations inherent in traditional syntax-oriented semiotics, thereby facilitating a more nuanced examination of performance interpretation. The central argument of the paper is that the underlying mechanisms of discourse semantics can better explain the meaning-making in theater by elucidating the intermodal and intramodal interactions among diverse (...) theatrical semiotic resources. To demonstrate the practical applicability of this framework, the study conducts a detailed analysis of several instances from two English and Persian performances of the play A Man for All Seasons by Robert Bolt, showcasing its effectiveness in revealing the intricate interplay of various semiotic resources involved in constructing a coherent theatrical discourse. The findings underscore that embracing a discursive perspective in the examination of theater and conceptualizing it as a multimodal discourse substantially enhances the comprehension of its meaning-making mechanisms. (shrink)

ABSTRACTCritical Discourse Studies positions itself as a critical, transformative practice that seeks to expose the ways in which discourse is able to constitute social, political, economic, gendered, racial, and sexual inequalities as normal and unremarkable. In this respect, CDS is an expressly political and therefore ethical project. Nevertheless, this article posits that, quite remarkably, the ethical frameworks that guide CDS have remained largely under-theorized and taken for granted. Accordingly, this paper seeks to contribute to this much-needed area of theoretical inquiry (...) by offering a brief discussion of how CDS has tended to define its own criticality and identifies three problems that emerge from how CDS practitioners have tended to formulate the praxis of critique: namely, the tendency to conceive of CDS as a demystificatory practice, the implicit logocentrism of modeling ethics upon the Habermasian Ideal Speech Situation, and finally, the deontological nature of discourse ethics. Each of the three problems is discussed in relation to a brief multimodal critical discourse analysis of three online employer profile videos. As such, the study is offered to demonstrate not only the analytical strength of multimodal analysis but also the political-ethical relations beyond Discourse Ethics that are immanent in such analyses. (shrink)

What is the correct procedure for determining the contents of perception? Philosophers tackling this question increasingly rely on empirically-oriented procedures in order to reach an answer. I argue that this constitutes an improvement over the armchair methodology constitutive of phenomenal contrast cases, but that there is a crucial respect in which current empirical procedures remain limited: they are unimodal in nature, wrongly treating the senses as isolatable faculties. I thus have two aims: first, to motivate a reorientation of the admissible (...) contents debate into a multimodal framework, charting its various significances. The second is to explore whether any experimental studies of multimodal perception support a so-called Liberal (or ‘high-level’ or ‘rich’) account of perception’s admissible contents. I conclude that the McGurk effect and the ventriloquist effect are both explicable without the postulation of high-level content, but that at least one multimodal experimental paradigm may necessitate such content: the rubber hand illusion. One upshot of this argument is that Conservatives who claim that the Liberal view intolerably broadens the scope of perceptual illusions, particularly from the perspective of perceptual psychology, should pursue other arguments against that view. (shrink)

We study straightforward embeddings of propositional normal multimodal logic and propositional intuitionistic logic in simple type theory. The correctness of these embeddings is easily shown. We give examples to demonstrate that these embeddings provide an effective framework for computational investigations of various non-classical logics. We report some experiments using the higher-order automated theorem prover LEO-II.

This chapter explores the multimodal representation of personal life and cultural identity in the Instagram posts of four gaming influencers from diverse backgrounds: PewDiePie, Valkyrae, Ninja, and Pokimane. Utilising Gunther Kress and Theo van Leeuwen's Social Semiotics framework, the analysis focuses on how these influencers blend visual and textual elements in non-gaming posts to depict themes like family life, professional achievements, and personal moments. This study highlights how influencers use Instagram's platform-specific features to foster a sense of authenticity (...) and relatability with their audience. The findings support the argument that these gaming influencers effectively deploy multimodal communication to connect with their audiences at a human (sometimes parasocial) level. Furthermore, the study reveals how these multimodal strategies contribute to broader discussions on digital communication. (shrink)

Multimodal argumentation-combining textual, visual, and symbolic reasoning-has become an essential framework for analyzing and interpreting legal codes. However, its use in legal scholarship faces significant challenges due to the rigid nature of legal language and its strong textual focus. This study critically explores the difficulties in implementing multimodal argumentation for legal code interpretation, emphasizing its effects on clarity, accessibility, and adaptability across various legal systems. Methodologically, the research utilizes qualitative content analysis of secondary data, including legal documents, (...) judicial interpretations, and scholarly literature on semiotics and legal theory. By examining secondary data from comparative legal analysis, case law, and studies on legal semiotics, this research identifies gaps in current literature, particularly the absence of a cohesive framework for integrating multimodal argumentation into legal practice. The findings indicate that although multimodal approaches present new opportunities for improving the understanding and use of legal codes, they also bring up significant issues related to standardization, bias, and the risk of misinterpreting non-textual elements. This paper adds to legal scholarship by suggesting an evaluative model that reconciles the interpretive advantages of multimodality with the importance of preserving legal certainty. The conclusion highlights the need for interdisciplinary collaboration to create strong frameworks that incorporate multimodal argumentation while upholding the fundamental principles of legal reasoning. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how (...) the multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to (...) class='Hi'>multimodal experiences of pain. Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

When people are engaged in social interaction, they can repeat aspects of each other’s communicative behavior, such as words or gestures. This kind of behavioral alignment has been studied across a wide range of disciplines and has been accounted for by diverging theories. In this paper, we review various operationalizations of lexical and gestural alignment. We reveal that scholars have fundamentally different takes on when and how behavior is considered to be aligned, which makes it difficult to compare findings and (...) draw uniform conclusions. Furthermore, we show that scholars tend to focus on one particular dimension of alignment (traditionally, whether two instances of behavior overlap in form), while other dimensions remain understudied. This hampers theory testing and building, which requires a well‐defined account of the factors that are central to or might enhance alignment. To capture the complex nature of alignment, we identify five key dimensions to formalize the relationship between any pair of behavior: time, sequence, meaning, form, and modality. We show how assumptions regarding the underlying mechanism of alignment (placed along the continuum of priming vs. grounding) pattern together with operationalizations in terms of the five dimensions. This integrative framework can help researchers in the field of alignment and related phenomena (including behavior matching, mimicry, entrainment, and accommodation) to formulate their hypotheses and operationalizations in a more transparent and systematic manner. The framework also enables us to discover unexplored research avenues and derive new hypotheses regarding alignment. (shrink)

In this paper we compare grammatical inference in the context of simple and of mixed Lambek systems. Simple Lambek systems are obtained by taking the logic of residuation for a family of multiplicative connectives /, *, \, together with a package of structural postulates characterizing the resource management properties of the * connective. Different choices for Associativity and Commutativity yield the familiar logics NL, L, NLP, LP. Semantically, a simple Lambek system is a unimodal logic: the connectives get a Kripke (...) style interpretation in terms of a single ternary accessibility relation modeling the notion of linguistic composition for each individual system.The simple systems each have their virtues in linguistic analysis. But none of them in isolation provides a basis for a full theory of grammar. In the second part of the paper, we consider two types of mixed Lambek systems.The first type is obtained by combining a number of unimodal systems into one multimodal logic. The combined multimodal logic is set up in such a way that the individual resource management properties of the constituting logics are preserved. But the inferential capacity of the mixed logic is greater than the sum of its component parts through the addition of interaction postulates, together with the corresponding interpretive constraints on frames, regulating the communication between the component logics.The second type of mixed system is obtained by generalizing the residuation scheme for binary connectives to families of n-ary connectives, and by putting together families, of different arities in one logic. We focus on residuation for unary connectives, hence on mixed frames, as these already represent the complexities in full. We prove a number of elementary logical results for unary families of residuated connectives and their combination with binary families. The existing proposals for unary ‘structural modalities’ are situated within this general framework, and a number of new linguistic applications is given. (shrink)

I analyze multimodal viewpoint construction in comics to engage with how modalities function within the medium as a specific discourse context with distinct conventions and material qualities. I show how comics employ established storytelling practices with character, narrator, and narrative viewpoint levels, while building up and interweaving these through strategic uses of the modalities of the medium. I mobilize the cognitive theories of embodiment, domains, mental simulation, and mental space blending as an analytical framework. I examine the asynchronicity (...) of viewpoint elements between modalities and their synthesis into composite character viewpoints in several examples. I show how modalities can be prioritized and their different qualities and functions strategically manipulated for viewpoint construal. These brief examples show the complexity inherent in multimodal communication and interpretation and the usefulness of encouraging the medium-specific and interdisciplinary analyses of cultural works from a cognitive linguistic perspective. (shrink)

The paper has two main aims: first, I emphasize the need for a multimodal lens in discourse analysis; second, I present a critique of the discursive construction of migrant identity on Internet-based new media platforms that are produced by state institutions (i.e. official websites) and created by individual migrants on their own personal social media profiles. Using a multimodal framework that accounts for not only words but the various semiotic resources available to participants in online media content (...) production, I focus my analysis on two sets of cases of official texts from Philippine and New Zealand government migration websites and three sample cases of social media content by Auckland-based Filipino migrants in New Zealand. In both the official and individual texts, there is an apparent idealization and romanticization of migrant life in New Zealand depicted mainly through the hegemonic images of “home”, “family”, and “childhood” that act as nodal points in setting up discursive boundaries. Although it may seem that individual agency is not realized in the purported democratized avenues of expression of the Internet, I argue that migrant self-presentation online or offline is contingent on the very complexity and contradictions of migrant daily life as configured by both the home and host nations. (shrink)

This paper, framed by UK law, focuses on analyzing the provisions of multimodal transport documents under the Carriage of Goods by Sea Act 1971 and 1992,with particular attention to two key legal issues: First, whether multimodal transport bills of lading can be recognized as traditional bills of lading or seaway bill under the current legal framework; Second, whether the Hague-Visby Rules introduced by the 1971 Act can be directly applied to the maritime segment of multimodal transport (...) practices. This paper analyzes the British legal precedents and legal provisions, and puts forward the following problems that need to be solved: first, the 1992 Act does not clearly define the multimodal transport documents and whether they can be applied to this law, and the 1971 Act does not make clear provisions on the scope of application. In order to deal with the practical challenges brought by the above defects, this paper will analyze the relevant content and put forward suggestions for legal modification: the law should clearly cover multimodal transport. (shrink)

Improvising selected tools from Kress and Van Leeuwen’s inter-semiosis framework, this study explores how, between global and local, TV commercials in India often reframe a cultural third space producing new discursive forms and identities. Three commercials from the food and beverage category are selected on the basis of the country of origin of the endorsing company and the patterns of glocalization. Multimodal discourse analysis reveals that the commercials construct the glocal identity in several ways. In the Knorr Soups (...) commercial, the melody of the title song is contemporary pop, whereas the delivery style and crooning are distinctively Western. The language is predominantly Hindi with deft use of selected English words. The Funda Mint commercial casts a rebel voice-support executive who rejects the pseudo-glocal identity and asserts his true identity. In the Cadbury Bournvita Folk commercial, the two boys represent the global and the local in the realm of creativity. The study advances the knowledge of product positioning and brand identity involving the intersemiosis of discursive elements from different cultural spaces within the contemporary Indian consumerist culture. (shrink)

We present a formalization of some ideas from Chomsky [9] in the framework of multimodal type logical grammar.

In this article I analyse object labelling image macro internet memes as multimodal metaphors, taking the Distracted Boyfriend meme as a case study. Object labelling memes are multimodal texts in which users add labels to a stock photograph to convey messages that are often humorous or satirical in nature. Using the relevance-theoretic account of metaphor, I argue that object labelling memes are multimodal metaphors which are interpreted using the same processes as verbal metaphors. The labelling of the (...) image guides the viewer in the construction of ad hoc concepts, and it is these ad hoc concepts that contribute to the overall meaning that is communicated. The analysis in this article is rooted in the relevance-theoretic claim that pragmatic interpretive processes are triggered by all and any ostensive acts of communication. I also draw heavily on Deirdre Wilson’s work on lexical pragmatics to show how this plays out in the case of a multimodal digital text. Memes, like verbal metaphors, do not require a special theory or framework. They can be understood as ostensive stimuli which trigger the search for an optimally relevant interpretation. (shrink)

Autonomous sensory meridian response (ASMR) and affective touch (AT) are two phenomena that have been independently investigated from separate lines of research. In this article, I provide a unified theoretical framework for understanding and studying them as complementary processes. I highlight their shared biological basis and positive effects on emotional and psychophysiological regulation. Drawing from evolutionary and developmental theories, I propose that ASMR results from the development of biological mechanisms associated with early affiliative behaviour and self‐regulation, similar to AT. (...) I also propose a multimodal interoceptive mechanism underlying both phenomena, suggesting that different sensory systems could specifically respond to affective stimulation (caresses, whispers and affective faces), where the integration of those inputs occurs in the brain's interoceptive hubs, allowing physiological regulation. The implications of this proposal are discussed with a view to future research that jointly examines ASMR and AT, and their potential impact on improving emotional well‐being and mental health. (shrink)

Intersemiotic projection is one of the most common configurations in the knowledge construction process of academic comics. Although previous studies address some general features of intersemiotic projection, further research on interdependency relations of intersemiotic projection is needed in order to map out the whole system. This study, based on the social-semiotic approach to multimodal studies, proposes a systemic framework of image-text paratactic and hypotactic projection in academic comics. This framework identifies three sub-categories of paratactic projection and hypotactic (...) projection, respectively: (1) Emergent, (2) Adjoined, (3) Sequential, (4) Compound, (5) Subsumed, and (6) Fused. The use of these projection configurations contributes to the realization of the communicative purposes of academic comics to evidence, exemplify, emphasize, explain, elaborate, and visualize the discussed knowledge. This research attempts not only to contribute to the theoretical development of multimodality from the social-semiotic perspective by refining and enlarging the multimodal hypotactic projection framework but also to provide an effective and applicable tool for comics analysis. (shrink)

Musical performance is a multimodal experience, for performers and listeners alike. This paper reports on a pilot study which constitutes the first step toward a comprehensive approach to the experience of music as performed. We aim at bridging the gap between qualitative and quantitative approaches, by combining methods for data collection. The purpose is to build a data corpus containing multimodal measures linked to high-level subjective observations. This will allow for a systematic inclusion of the knowledge of music (...) professionals in an analytic framework, which synthesizes methods across established research disciplines. We outline the methods we are currently developing for the creation of a multimodal data corpus dedicated to the analysis and exploration of instrumental music performance from the perspective of embodied music cognition. This will enable the study of the multiple facets of instrumental music performance in great detail, as well as lead to the development of music creation techniques that take advantage of the cross-modal relationships and higher-level qualities emerging from the analysis of this multi-layered, multimodal corpus. The results of the pilot project suggest that qualitative analysis through stimulated recall is an efficient method for generating higher-level understandings of musical performance. Furthermore, the results indicate several directions for further development, regarding observational movement analysis, and computational analysis of coarticulation, chunking, and movement qualities in musical performance. We argue that the development of methods for combining qualitative and quantitative data are required to fully understand expressive musical performance, especially in a broader scenario in which arts, humanities, and science are increasingly entangled. The future work in the project will therefore entail an increasingly multimodal analysis, aiming to become as holistic as is music in performance. (shrink)

The Knowability Paradox starts from the assumption that every truth is knowable and leads to the paradoxical conclusion that every truth is also actually known. Knowability has been traditionally associated with both contemporary verificationism and intuitionistic logic. We assume that classical modal logic in which the standard paradoxical argument is presented is not sufficient to provide a proper treatment of the verificationist aspects of knowability. The aim of this paper is both to sketch a language $$\mathcal {L}_{\Box,K}^{P}$$, where alethic and (...) epistemic classical modalities are combined with the pragmatic language for assertions $$\mathcal {L}^{P}$$, and to analyse the result of the application of our framework to the paradox. (shrink)

A variety of theoretical frameworks predict the resemblance of behaviors between two people engaged in communication, in the form of coordination, mimicry, or alignment. However, little is known about the time course of the behavior matching, even though there is evidence that dyads synchronize oscillatory motions (e.g., postural sway). This study examined the temporal structure of nonoscillatory actions—language, facial, and gestural behaviors—produced during a route communication task. The focus was the temporal relationship between matching behaviors in the interlocutors (e.g., facial (...) behavior in one interlocutor vs. the same facial behavior in the other interlocutor). Cross-recurrence analysis revealed that within each category tested (language, facial, gestural), interlocutors synchronized matching behaviors, at temporal lags short enough to provide imitation of one interlocutor by the other, from one conversational turn to the next. Both social and cognitive variables predicted the degree of temporal organization. These findings suggest that the temporal structure of matching behaviors provides low-level and low-cost resources for human interaction. (shrink)

Multimodal discourse analysis offers a novel lens for the study of legal discourse. Within the closing arguments of a trial, prosecution and defense attorneys utilize various multimodal resources to present evidence, articulate opinions, and adopt stances to achieve their communicative goals. This research focuses on the closing arguments in the criminal trial concerning the death of George Floyd to analyze the multimodal representation of the closing arguments delivered by both prosecution and defense attorneys. It employs the analytical (...) framework of the attitude system and paralinguistic typology proposed by Martin and colleagues, and uses the UAM Corpus Tool and Praat to qualitatively annotate and quantify the linguistic and paralinguistic information. The findings reveal that these attorneys employ different multimodal resources, including language and paralanguage, to convey their attitudes towards the case’s participants. This leads to the construction of competing narratives, shedding light on how justice is seen to be done. (shrink)

Multimodal neuroimaging provides a rich source of data for identifying brain regions associated with disease progression and aging. However, present studies still typically analyze modalities separately or aggregate voxel-wise measurements and analyses to the structural level, thus reducing statistical power. As a central example, previous works have used two quantitative MRI parameters—R2* and quantitative susceptibility —to study changes in iron associated with aging in healthy and multiple sclerosis subjects, but failed to simultaneously account for both. In this article, we (...) propose a unified framework that combines information from multiple imaging modalities and regularizes estimates for increased interpretability, generalizability, and stability. Our work focuses on joint region detection problems where overlap between effect supports across modalities is encouraged but not strictly enforced. To achieve this, we combine L1, total variation, and L2 group lasso penalties. While the TV penalty encourages geometric regularization by controlling estimate variability and support boundary geometry, the group lasso penalty accounts for similarities in the support between imaging modalities. We address the computational difficulty in this regularization scheme with an alternating direction method of multipliers optimizer. In a neuroimaging application, we compare our method against independent sparse and joint sparse models using a dataset of R2* and QS maps derived from MRI scans of 113 healthy controls: our method produces clinically-interpretable regions where specific iron changes are associated with healthy aging. Together with results across multiple simulation studies, we conclude that our approach identifies regions that are more strongly associated with the variable of interest, more accurate, and more stable with respect to training data variability. This work makes progress toward a stable and interpretable multimodal imaging analysis framework for studying disease-related changes in brain structure and can be extended for classification and disease prediction tasks. (shrink)

Theoretical linguistics has made great strides towards a science of language, building on solid foundations in logic and analytical philosophy, and, more recently, joining forces with empirical sciences. Superlinguistics wants to expand the reach of our sophisticated theoretical and empirical linguistic toolkit by applying it to language-like objects beyond the scope of traditional linguistics, like gestures, comics, or music. [...] I offer four case studies showcasing different aspects of supersemantic approaches to multimodal sign systems. In 3.1 I discuss a (...) compositional semantic approach to the interpretation of facial expressions; in 3.2, I review theoretical approaches to emoji meanings from the recent supersemantics literature; in 3.3, I sketch a possible-worlds semantics for pictures and a discourse-level approach to comics; and in 3.4, I explore how to combine the different iconic semantic contributions of sound, music, and moving images in film in a unified formal semantics framework. (shrink)

Breathing is a vital, continuous behavior that maintains physiological homeostasis, yet it is also remarkably flexible—modulated by volitional, emotional, and behavioral states. This review highlights recent advances in understanding how distributed neural circuits, particularly in the ventrolateral medulla and dorsolateral pons, integrate both homeostatic and non‐homeostatic influences on respiratory control. We examine how higher‐order brain regions interact with brainstem rhythm generators such as the preBötzinger complex, emphasizing a dynamic, state‐dependent framework for respiratory regulation. Once considered a reflexive brainstem function, (...) breathing is now recognized as the emergent output of interconnected networks that flexibly adapt rhythm and pattern based on internal state, behavior, and environmental context. Grasping this complexity is critical for understanding both the normal versatility and pathological vulnerability of respiratory control. (shrink)

This paper uses van Leeuwen’s (2008. Discourse and practice: New tools from critical discourse analysis. Oxford: Oxford University Press) Authority Legitimation framework to examine government posters published in the 1950s–1980s in Hong Kong, which serve as a means of shaping public opinion and legitimate social action. Martin and White’s (2005. The language of evaluation: Appraisal in english. London: Palgrave Macmillan) Appraisal framework is also applied to provide the study with relevant analytical tools by which to construct evaluatively coherent (...) authorial reading positions propagated by the government in the posters as well as aligning viewers with these desired positions. The government posters being studied are concerned with various aspects of social practices in the domain of public health communication. This paper argues that the representations of these social practices are realized as texts – both visual and verbal – in the posters, as part of a broader recontextualization process where legitimate ways of doing things as stipulated in the government guidelines are recontextualized in these posters and conveyed to members of the public. The aim is two-fold: (1) to provide a systematic analysis of how visual and verbal resources construe evaluation in the government posters and the consequent legitimation discourse as viewers are being persuaded to align with legitimate social practices; (2) to highlight various forms of legitimation, namely, role model authority legitimation, impersonal authority legitimation, and conformity authority legitimation, that are realized in these posters through visual-verbal articulation. (shrink)

Multimodal Critical Discourse Analysis (MCDA) as a sub-discipline of Critical Discourse Analysis (CDA) emerged from the availability of social semiotic frameworks describing multimodal meaning making. However, weaknesses of these frameworks have raised concerns and prompted recent methodological reflections in MCDA. Inspired by these reflections, this paper critically assesses MCDA research on advertising persuasion and identifies a lack of attention in studies to account for the social and ideological impact of advertising. This shortcoming is argued to be attributable to (...) the weak empirical grounding of social semiotic frameworks which in turn is rooted in a ‘discursive gap’ (Bateman, 2019, p. 300) in multimodality as a discipline. The authors then propose methodological solutions involving eye tracking experiments and higher order data analytics which can strengthen the empirical basis of theoretical assumptions made in MCDA research on advertising. Drawing on methodological guidance for empirical multimodal research, the authors discuss specific issues in realizing this proposal, focusing on a strategy addressing operationalization challenges in experimental design and the affordance of a data cluster algorithm that calculates the common viewing path of a group of viewers. (shrink)

This article examines the interaction between grammar and multimodal resources by analyzing reported speech in Korean multiparty face-to-face interaction. The operation of two relevancy rules — minimization and recognition in interaction — is examined together with how the absence or presence of grammar is complemented by multimodal resources of various sorts. For the analysis, three categories are posited depending on who the quoted character is in the talk. In quoting oneself or co-participants in the talk, syntactic resources, prosody, (...) sequential organization and embodied participant frameworks collaborate in the interactional management of the quote without overt realization of grammatical resources traditionally proposed for building quotations in Korean. The multimodal character of quoted speech is most weakly realized when quoting a non-present character, not only because of the limitation in body and spatial resources available in the local context to recover various referential terms, but more importantly, because the interactional project that the parties are engaged in makes a distancing device such as the quotative particle -ko relevant. This study provides evidence for the intricate interaction between multimodal resources and grammar through investigation of an important realm of human interaction, that of reported speech. (shrink)

This paper proposes a methodological synthesis in order to study multimodal media discourse and argumentation in the context of the so-called ‘refugee crisis’ in Greece. It follows the framework of Multimodal Critical Discourse Analysis, integrating this with argumentation studies, with a particular emphasis on the analysis of inference. Our data come from the Greek newspapers Kathimerini and Ta Nea. We contend that the proposed methodological synergy enables scrutiny of (a) racist conceptualizations cultivated by the representation of migrants (...) and refugees in headlines and photographs in newspaper articles along with (b) the argumentative potential that is implicitly sustained in these multimodal representations. (shrink)

The conceptual metaphor “THOUGHT IS FOOD” is exemplified in many verbal expressions. Nevertheless, how food metaphors are realized through the actual dining experience remains unexplored. Based on a food design event called EATAIPEI that took place in the London Design Festival in 2015, one aimed at promoting Taipei as World Design Capital 2016, this article analyzes how the multimodal metaphors of food were creatively represented and elaborated within it. This study proposes an analytical framework that combines insights from (...) cognitive linguistics with an intercultural performative view of food presentation and food metaphor. More specifically, it treats complex metaphors as resulting from the interactions of primary metaphors, proposition schemas and intercultural performance. The multimodal metaphors of food are complex metaphors composed of combined sets of primary metaphors, enriched by schema propositions and embedded within food performance. The experience of eating in the event reactivates the sensorimotor experiences that underpin the complex metaphors. (shrink)

The analysis of legal communication has almost exclusively been the domain of discourse analysts focusing on the ways that the linguistic system is used to realise legal meanings. Multimodal discourse analysis, where visual forms in combination with traditional linguistic expressions co-occur, is now also an area of expanding interest. Taking a Systemic Functional Linguistics “social semiotic” perspective, this paper applies and critiques an analytical framework that has been used for examining intersemiotic complementarity in various types of page-based (...) class='Hi'>multimodal texts by analysing a cartoon satirizing the demands that verdict deliberations can make on a citizen jury seated in a Crown Court. While the multimodal analysis reveals that a straightforward application of this analytical framework is useful in and of itself, it is argued that the framework needs to be extended to also account for further and more complex layers of represented meaning. The analysis is also situated in the wider legal social semiotic, and draws attention to international debates on the efficacy of the jury system itself. (shrink)

This study uses a multimodal discourse analysis (MDA) framework to investigate how physical evidence is linguistically and semiotically constructed in Saudi criminal rulings. Drawing on a corpus of authentic judicial texts encompassing cases of theft, assault, and drug-related offences, the research explores how courts transform material items—such as weapons, jewelry, bruises, and surveillance footage—into narrative elements that substantiate or contest criminal allegations. The analysis focuses on discursive strategies including nominalization, indexical anchoring, evidential framing, and intertextual references to Islamic (...) legal concepts such as “ḥirz” (safeguarded property), “niyya” (intent), “qarā’in” (circumstantial indicators), and “shubha” (doubt). The findings reveal that physical evidence does not possess intrinsic meaning; rather, it is discursively mediated through culturally embedded legal logic and rhetorical structures. Courts assign evidentiary weight by aligning physical evidence with confessions, testimonies, or forensic reports, frequently navigating challenges, such as retracted statements or claims of coercion. The study identifies five recurrent semiotic patterns: iconic/indexical anchoring, referential expansion, visual-verbal translation, denial contestation, and cultural semiotic calibration. These strategies function not only to establish credibility and causality but also to reinforce institutional authority and narrative coherence. (shrink)

The phenomenon of multimodality is central to our everyday interaction. 'Hybrid' modes of communication that combine traditional uses of language with imagery, tagging, hashtags and voice-recognition tools have become the norm. Bringing together concepts of meaning and communication across a range of subject areas, including education, media studies, cultural studies, design and architecture, the authors uncover a multimodal grammar that moves away from rigid and language-centered understandings of meaning. They present the first framework for describing and analysing different (...) forms of meaning across text, image, space, body, sound and speech. Succinct summaries of the main thinkers in the fields of language, communications and semiotics are provided alongside rich examples to illustrate the key arguments. A history of media including the genesis of digital media, Unicode, Emoji, XML and HTML, MP3 and more is covered. This book will stimulate new thinking about the nature of meaning, and life itself, and will serve practitioners and theorists alike. (shrink)

ABSTRACTA common strategy in media communication on climate change is to tailor messages to the existing values of specific target audiences in order to make climate change mitigation attractive to them. The question how such field-specific messages are oriented towards the reality of climate change, however, is often neglected. As an alternative to the common framing approaches to media communication on climate change, this paper advances a discursive and semiotic perspective that takes the epistemological position of a constructivist realism with (...) C.S. Peirce and focuses on the way the abstract object of climate change is represented and transformed in media discourse. With a methodological framework of multimodal critical discourse analysis, which builds on the discourse-historical approach and investigates different discursive strategies, the field-specific representation of climate change in two factual television programmes is critically analysed and the effects of an sufficient orientation towards the object of climate change are discussed. (shrink)

In contrast to symbol-manipulation approaches, Cognitive Linguistics offers a modal rather than an amodal account of meaning in language. From this perspective, the meanings attached to linguistic expressions, in the form of conceptualisations, have various properties in common with visual forms of representation. This makes Cognitive Linguistics a potentially useful framework for identifying and analysing language-image relations in multimodal texts. In this paper, we investigate language-image relations with a specific focus on intersemiotic convergence. Analogous with research on gesture, (...) we extend the notion of co-text images and argue that images and language usages which are proximal to one another in a multimodal text can be expected to exhibit the same or consistent construals of the target scene. We outline some of the dimensions of conceptualisation along which intersemiotic convergence may be enacted in texts, including event-structure, viewpoint, distribution of attention and metaphor. We take as illustrative data photographs and their captions in online news texts covering a range of topics including immigration, political protests, and inter-state conflict. Our analysis suggests the utility of Cognitive Linguistics in allowing new potential sites of intersemiotic convergence to be identified and in proffering an account of language-image relations that is based in language cognition. (shrink)

In contemporary educational contexts there is considerable variation in how argumentation works and what forms and styles it takes. Influencing factors include the educational purpose and task, the level of education, and the discipline or curriculum subject in which it occurs. This paper offers a theoretical framework and a set of multimodal analytical tools which can provide a rich and systematic account of such variation. Using naturalistic data from three different educational sites I illustrate how such a (...) class='Hi'>framework reveals the diverse ways in which students use language and other modes of meaning making as they engage in processes of argumentation. In particular, I consider how new technologies have caused shifts in the distribution of meaning across different semiotic modes (such as visual images, space, colour and graphics) and how this impacts upon both argumentation process and product. The educational implications of such changes are also considered. (shrink)

Interpretive captions play a crucial role in museum contexts by providing verbal information that complements and enhances visual artworks, thereby facilitating communication between visitors and the museum. However, despite their significance, there remains uncertainty about how these captions, generally perceived as secondary to the artworks, vary in their enhancing roles. In response to this gap, this study employs the multimodal corpus SemArt, taking a corpus-driven approach to explore the correlations between interpretive captions and classical European paintings. By applying the (...) Latent Dirichlet Allocation method, themes and keywords were extracted from the interpretive captions and analyzed within a visual communication framework to uncover sub-genre variations and different text-image relations. The findings reveal seven sub-genres within interpretive captions and two types of text-image enhancement relations: content-enhanced and form-enhanced. This research highlights the benefits and challenges of using exploratory quantitative analysis to facilitate multimodal research, paving the way for broader applications of unsupervised machine-learning techniques in digital humanities. (shrink)

XML documents and, more generally, semistructured data, can be seen as labelled graphs. In this paper we set a correspondence between such graphs and the models of a language of hybrid multimodal logic. This allows us to characterize a schema for semistructured data as a formula of hybrid multimodal logic, and instances of the schema as models of this formula. We also investigate how to express in such a logic integrity constraints on semistructured data, in particular some classes (...) of constraints widely considered in the literature. The contribution of this work is twofold: 1) We generalize the notion of schema, by proposing a definition of schema were references are “well typed”. 2) We formalize semistructured data, schema for semistructured data and integrity constraints in a unique framework, namely hybrid multimodal logic. (shrink)

This article addresses an issue in multimodal cultural studies – the inconsistent use of the notions ‘mode’ and ‘modality’. As these notions are frequently employed interchangeably, making a clear distinction between them and positioning them in a coherent system will be helpful. To outline such a system, I envisage a two-layer framework where modes and modalities support each other. The central branch of multimodal semiotics (developing from Gunther Kress’ sociosemiotics towards John Bateman’s comprehensive approach) recognizes ‘mode’ as (...) a pivotal research concept. While ‘mode’ as a semiotic resource is dependent on its materiality, culturally shaped practices and discourse semantics, the neurocognitive characteristics of sensory modalities are often seen as secondary to meaning-making. This article suggests that discussion of the semiotic potential of sensory modalities is complementary to the semiotic theory of multimodality. In order to illustrate this, I will construct an experimental typology of modality relations, which also takes modes into account. This typology distinguishes between supporting, modifying, conflicting, substituting and cross-activating relations. (shrink)

This paper is a multimodal critical discourse study of other-representation in ISIS’s magazine, Dabiq, It focuses on both the micro-level analysis of actor and action representation, and the macro-structure of negative other-depiction in Dabiq from both textual and visual perspectives. Through in-depth examination of linguistic and non-linguistic elements, the study aimed to unfold ISIS’s ideology at the global level, which is to construct its desired reality and eventually to recruit supporters. The analysis was carried out on fifteen issues of (...) Dabiq following a conceptual and analytical framework within Multimodal Critical Discourse Analysis employing Van Leeuwen’s Socio-semantic inventory. The results show a considerable number of systematic utilizations of discursive strategies by ISIS to belittle its enemies and downgrade their practices. Throughout Dabiq, the out-group participants are represented either as passive reactants who are afflicted by ISIS’s attacks or as powerless agents whose actions are trivial and carry no significant weight. Ineffectuality of ‘others’ has further been substantiated by the visual analysis. Deactivation of out-groups is closely tied to ISIS’s propaganda of self-alteration to achieve the double aim of casting fear among their adversaries and brainwashing the audience by twisting the reality, and persuading them that the enemies’ chance of winning this war is dim. (shrink)

The framework of depicting put forward byClark (2016)offers a schematic vantage point from which to examine iconic language use. Confronting the framework with empirical data, we consider some of its key theoretical notions. Crucially, by reconceptualizing the typology of depictions, we identify an overlooked domain in the literature: “speech-embedded nonverbal depictions,” namely cases where meaning is communicated iconically, nonverbally, and without simultaneously co-occurring speech. In addition to contextualizing the phenomenon in relation to existing research, we demonstrate, with examples (...) from American TV talk shows, how such depictions function in real-life language use, offering a brief sketch of their complexities and arguing also for their theoretical significance. (shrink)

Previous studies on legitimation, multimodality and political discourse by researchers, such as Van Leeuwen, Van Dijk and Mackay, have suggested different but supplementary methods of legitimation analysis by providing a number of analytical frameworks. Multimodal legitimation research, however, seems to be in need of a better conflation of the theoretical backgrounds of disciplines, such as narratology. This article focuses on the multimodal discourse of three political advertisements of the political party New Democracy, filmed for the needs of the (...) Greek legislative election of January 2015. What is investigated is the multimodal means by which New Democracy’s president, and Prime Minister at the time, Antonis Samaras attempted to legitimise his candidacy. In this article, I use the six-layer framework proposed by Mackay for multimodal legitimation analyses and I argue that multimodal legitimation research can benefit and get enhanced from the use of narratology and its analytical categories, such as perspective. (shrink)

Research involving children and youth, particularly those from culturally sensitive and vulnerable backgrounds, presents intricate ethical, methodological, and epistemological challenges. While acknowledging parents’ legitimate protective role, unjustified parental gatekeeping often regulates access to minors’ participation and can create tensions between adult-centric authority and minors’ evolving capacity for agency. These ambiguities get further obscured when minors’ self-identified experiences and fluid identities—especially in multilingual and culturally diverse contexts—diverge from parental assumptions or hegemonic discourses surrounding identity and autonomy. In this paper, I critically (...) interrogate these tensions, foreground the limitations of procedural ethics when faced with emergent obstacles, and advocate for adaptive, participant-centered frameworks rooted in dialogic engagement. Drawing on three research projects that I conducted in Canada, I demonstrate how child-engaging, multimodal methodologies can facilitate semiosis and empower minors to articulate their lived realities while safeguarding their emotional safety and agency. In this regard, intersectional reflexivity emerges as a constitutive and vital framework, allowing me to address power asymmetries and ethical dilemmas with situational responsivity. This paper reconceptualizes consent as an iterative and relational process rather than a static obligation by foregrounding minors’ narratives and respecting their welfare. Ultimately, the study contributes to advancing ethically robust and transformative research practices that amplify marginalized voices, dismantle systemic inequities, foster nuanced, inclusive engagements with vulnerable participants, and challenge dominant epistemological hierarchies and centering minors’ perspectives as constitutive of broader social discourse. (shrink)