Dissertations / Theses on the topic 'Multimodal metaphors'

This thesis explores the use of multimodality to communicate retrieved results of Internet search engines. The investigation aimed to investigate suitable multimodal metaphors which would increase the level of usability of Internet search engine interfaces and enhance users` experience in the search activity. The study consisted of three experiments based on questionnaires and Internet search activities with a set of text-based and multimodal interfaces. These interfaces were implemented in two browsing platforms, named AVBRO and AVBRO II. In the first experiment, the efficiency of specific multimodal metaphors to communicate additional information of retrieved results was investigated. The experiment also sought to obtain users` views of these metaphors with a questionnaire. An experimental multimodal interface of the AVBRO platform, which communicated additional information with a combination of three 2D graphs and musical stimuli, was used as a basis for the experiment, together with the Google search engine. The results obtained led to the planning of a second experiment. The aim of this experiment was to obtain and compare the level of usability of four different experimental multimodal interfaces and one traditional text-based interface, all implemented in the AVBRO II platform. Effectiveness, efficiency and users` satisfaction were used as criteria to evaluate the usability of these interfaces. In the third and final experiment the usability analysis of a traditional text-based interface and the two most suitable experimental multimodal interfaces of the AVBRO II platform was further investigated. Learnability, errors rate, efficiency, memorability and users` satisfaction were used as criteria to evaluate the usability of these interfaces. The analysis of the results obtained from these experiments provided the basis for a set of design guidelines for the development of usable interfaces based on a multimodal approach.

This empirical work aims to investigate the impact of using multimodal communication metaphors on e-learning systems’ usability, overall user experience and affective state. The study proposed a triple evaluation approach to avoid the problem of conventional assessment relying only on usability measurements of efficiency, effectiveness and user satisfactions. Usability in that sense refers only to the functionality and pragmatic side of the product and neglects other aspects of the system. Learning is a cognitive and repetitive task, requiring learners’ attention as well as their interest. Therefore, when delivering content, in addition to the pragmatic functionality, an e-learning system should provide a constructive overall user experience and positive affective state. Doing so will ensure user engagement, facilitate the learning process and increase learners’ performance. The impact of using five different communication metaphors was evaluated in three dimensions using the proposed approach. Within the usability dimension, the evaluation criteria involved measuring system efficiency, effectiveness, user satisfaction and learning performance. Within the user experience dimension, the evaluation criteria involved measuring pragmatic aspects of the user experience, the hedonic aspects of user experience in terms of stimulation as well as identification and the overall system attractiveness. Within the affective state dimension a self-assessments manikin technique was used in conjunction with biofeedback measurements, and users’ valence, arousal and dominance were measured. The study found that system attractiveness and the hedonic user experience had a profound impact on users’ learning performance and attitude toward the tested system. Furthermore, they influenced users’ views and judgement of the system and its usability. The communication metaphors were not equal in their influence within the evaluation criteria. Empirically derived guidelines were produced for the use and integration of these metaphors in e-learning systems. The outcome of the study highlights the need to use the triple evaluation approach in the assessment of e-learning interfaces prior to their release for better adoption and acceptance by end users.

This thesis investigated the use of novel combinations of multimodal metaphors in the presentation of learning information to evaluate the effect of these combinations on the usability of e-learning interfaces and on the users' learning performance. The empirical research described in this thesis comprised three experimental phases. In the first phase, an initial experiment was carried out with 30 users to explore and compare the usability and learning performance of facially animated expressive avatars with earcons and speech, and text with graphics metaphors. The second experimental phase involved an experiment conducted with 48 users to investigate their perception of avatar's facial expressions and body gestures when presented in both the absence and presence of interactive e-learning context. In addition, the experiment aimed at evaluating the role that an avatar could play as virtual lecturer in e-learning interfaces by comparing the usability and learning performance of three different modes of interaction: speaking facially expressive virtual lecturer, speaking facially expressive full-body animated virtual lecturer, and two speaking facially expressive virtual lecturers. In the third phase, a total of 24 users experimentally examined a novel approach for the use of earcons and auditory icons in e-learning interfaces to support an animated facially expressive avatar with body gestures during the presentation of the learning material. The obtained results demonstrated the usefulness of the tested metaphors to enhance e-learning usability and to enable users to attain better learning performance. These results provided a set of empirically derived innovative guidelines for the design and use of these metaphors to generate more usable e-learning interfaces. For example, when designing avatars as animated virtual lecturers in e-learning interfaces, specific facial expression and body gestures should be incorporated due to its positive influence in enhancing learners' attitude towards the learning process.

There has been an increasing demand for commercial organisations to foster real-time interaction with customers, because harnessing customer competencies has been shown to be a major contributor towards various benefits, such as growth, innovation and competition. This may drive organisations to embrace the development of multimodal interaction and complement Electronic Customer Knowledge Management Systems (E-CKMS) with metaphors of audio-visual nature. Although the implementation of E-CKMS encounters several challenges, such as lack of trust and information overload, few empirical studies were devoted to assess the role of audio-visual metaphors, and investigate whether these technologies can be put into practice. Therefore, this thesis describes a comparative evaluation study carried out to examine the implication of incorporating multimodal metaphors into E-CKMS interfaces on not only usability of E-CKMS, but also the user's trust, knowledge and acceptance. An experimental E-CKMS platform was implemented with three different modes of interaction: Visual-only E-CKMS (VCKMS) with text and graphics, Multimodal E-CKMS (MCKMS) with speech, earcons and auditory icons and Avatar-enhanced multimodal E-CKMS (ACKMS). The three platforms were evaluated by three independent groups of twenty participants each (total=60) who carried out eight common tasks of increasing complexity and design based on three different styles. Another dependent group of forty-eight participants (n=48) was instructed to interact with the systems under similar usability conditions by performing six common tasks of two styles, and fill a questionnaire devised to measure the aspects of user acceptance. The results therein revealed that ACKMS was more usable and acceptable than both MCKMS and VCKMS, whereas MCKMS was more usable than VCKMS, but less acceptable. Inferential Statistics indicated that these results were statistically significant.

There has been an increasing demand for commercial organisations to foster real-time interaction with customers, because harnessing customer competencies has been shown to be a major contributor towards various benefits, such as growth, innovation and competition. This may drive organisations to embrace the development of multimodal interaction and complement Electronic Customer Knowledge Management Systems (E-CKMS) with metaphors of audio-visual nature. Although the implementation of E-CKMS encounters several challenges, such as lack of trust and information overload, few empirical studies were devoted to assess the role of audio-visual metaphors, and investigate whether these technologies can be put into practice. Therefore, this thesis describes a comparative evaluation study carried out to examine the implication of incorporating multimodal metaphors into E-CKMS interfaces on not only usability of E-CKMS, but also the user¿s trust, knowledge and acceptance. An experimental E-CKMS platform was implemented with three different modes of interaction: Visual-only E-CKMS (VCKMS) with text and graphics, Multimodal E-CKMS (MCKMS) with speech, earcons and auditory icons and Avatar-enhanced multimodal E-CKMS (ACKMS). The three platforms were evaluated by three independent groups of twenty participants each (total=60) who carried out eight common tasks of increasing complexity and design based on three different styles. Another dependent group of forty-eight participants (n=48) was instructed to interact with the systems under similar usability conditions by performing six common tasks of two styles, and fill a questionnaire devised to measure the aspects of user acceptance. The results therein revealed that ACKMS was more usable and acceptable than both MCKMS and VCKMS, whereas MCKMS was more usable than VCKMS, but less acceptable. Inferential Statistics indicated that these results were statistically significant.

Submitted by Rafael Santana (rafael.silvasantana@ufpe.br) on 2017-05-11T17:36:39Z No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Tese_Adiano-BC.pdf: 7401702 bytes, checksum: 6b28f8e30c070212151e0a806655bd7c (MD5)<br>Made available in DSpace on 2017-05-11T17:36:39Z (GMT). No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Tese_Adiano-BC.pdf: 7401702 bytes, checksum: 6b28f8e30c070212151e0a806655bd7c (MD5) Previous issue date: 2016-02-19<br>Este trabalho investiga a realização de metáforas multimodais a partir da interação entre textos escritos e imagens em anúncios publicitários impressos. Para tanto, partese da contribuição da Linguística Sociocognitiva, que encara a linguagem enquanto atividade de conceptualização e concebe a metáfora como elemento do pensamento, das linguagens e das ações (LAKOFF e JOHNSON, 1980 e 1999; LAKOFF, 1987 e 2008). O corpus da pesquisa foi recolhido a partir do acompanhamento, no período de julho de 2013 a junho de 2014, das cinco revistas mensais com maior tiragem nacional: Claudia, Nova Escola, Quatro Rodas, Seleções Reader´s Digest e Superinteressante. Esses títulos foram lidos com o objetivo de verificar a realização multimodal de metáforas. Adotamos as sugestões metodológicas propostas por Schmitt (2005), que têm como objetivo congregar fatores subjetivos e procedimentos metodológicos na análise de metáforas e tenta reconstruir modelos de pensamento, de linguagem e de ação. Ao final da coleta, o corpus foi composto por 30 (trinta) anúncios. As análises mostraram a realização de metáforas multimodais de quatro categorias diferentes: (i) alvo multimodal é fonte visual, (ii) alvo verbal é fonte multimodal, (iii) alvo verbal é fonte visual e (iv) alvo multimodal é fonte multimodal. Essas instanciações cumprem funções cognitivo-discursivas distintas no discurso publicitário. Além disso, os achados evidenciaram a realização de metáforas convencionais (VIDA É VIAGEM, por exemplo) e, também, de metáforas episódicas (processador [de celular] é corredor, por exemplo). Essas realizações não parecem ser aleatórias, mas cumprem, assim como as categorias descritas, propósitos definidos nos textos analisados. As categorias propostas possibilitam a reflexão acerca dos mapeamentos metafóricos e das modalidades que envolvem a sua realização e o seu uso no discurso publicitário.<br>The aim of this work is to investigate the instantiations of multimodal metaphors from the interaction between written texts and images on printed advertisements. For this purpose, we based the research on the Socio-cognitive Linguistics contribution, which postulates language as an activity of conceptualization and conceives metaphor as an element of thought, language and actions (Lakoff and Johnson, 1980 and 1999; Lakoff, 1987 and 2008). The corpus of the research was collected from monitoring, from July 2013 to June 2014, the five largest monthly magazines with national circulation: Claudia, Nova Escola, Quatro Rodas, Seleções Reader's Digest and Superinteressante. These magazines were read in order to check the realization of multimodal metaphors. We adopted methodological suggestions proposed by Schmitt (2005), which aim to gather subjective factors and methodological procedures in the analysis of metaphors and attempts to reconstruct models of thought, speech and action. At the end of the collection, the corpus was composed of thirty (30) printed advertisements. Analyses showed the realization of multimodal metaphors in four different categories: (i) multimodal target is visual source, (ii) verbal target is multimodal source, (iii) verbal target is visual source and (iv) multimodal target is multimodal source. These instantiations meet cognitive-discursive distinct functions in advertising discourse. In addition, the findings showed the performance of conventional metaphors (LIFE IS A JOURNEY, for example) and episodic metaphors (processor [of cellphones] is running, for example). These embodiments do not appear to be random, but they serve for defined purposes in the analyzed texts. The proposed categories enable reflection on the metaphorical mappings and arrangements involving their realization and their use in advertising discourse.

It has been over 35 years since the publication of Reddy's 1979 study of the metaphors for communication, an article that could be said to mark the starting point of Conceptual Metaphor Theory. However, despite the understanding that metaphor and figurative language are fundamental to language, in that time there has been little progress in introducing metaphor into the L2 classroom, especially at lower levels. This thesis argues that learners at this level have figurative resources that have not yet been acknowledged, but could be key elements in developing an L2 metaphor pedagogy. To explore this possibility, a mixed methods investigation of the effect of mutimodal writing prompts based on extended metaphors was conducted with two cohorts of low-intermediate Japanese university students. The mixed methods data analysis revealed not only that the presentation of one extended metaphor could 'activate' metaphorical knowledge of other extended metaphors and induce the production of metaphoric language, but that multimodal material provides an as yet unexploited resource for an L2 metaphor pedagogy.

Impersonal, non-face-to-face contact and text-based interfaces, in the e-Learning segment, present major problems that are encountered by learners, since they are out on vital personal interactions and useful feedback messages, as well as on real-time information about their learning performance. This research programme suggests a multimodal, combined with an edutainment approach, which is expected to improve the communications between users and e-Learning systems. This thesis empirically investigates users’ effectiveness; efficiency and satisfaction, in order to determine the influence of edutainment, (e.g. amusing speech and facial expressions), combined with multimodal metaphors, (e.g. speech, earcon, avatar, etc.), within e-Learning environments. Besides text, speech, visual, and earcon modalities, avatars are incorporated to offer a visual and listening realm, in online learning. The methodology used for this research project comprises a literature review, as well as three experimental platforms. The initial experiment serves as a first step towards investigating the feasibility of completing all the tasks and objectives in the research project, outlined above. The remaining two experiments explore, further, the role of edutainment in enhancing e-Learning user interfaces. The overall challenge is to enhance user-interface usability; to improve the presentation of learning, in e-Learning systems; to improve user enjoyment; to enhance interactivity and learning performance; and, also, to contribute in developing guidelines for multimodal involvement, in the context of edutainment. The results of the experiments presented in this thesis show an improvement in user enjoyment, through satisfaction measurements. In the first experiment, the enjoyment level increased by 11%, in the Edutainment (E) platform, compared to the Non-edutainment (NE) interface. In the second experiment, the Game-Based Learning (GBL) interface obtained 14% greater enhancement than the Virtual Class (VC) interface and 20.85% more than the Storytelling interface; whereas, the percentage obtained by the game incorporated with avatars increased by an extra 3%, compared with the other platforms, in the third experiment. In addition, improvement in both user performance and learning retention were detected through effective and efficiency measurements. In the first experiment, there was no significant difference between mean values of time, for both conditions (E) & (NE) which were not found to be significant, when tested using T-test. In the second experiment, the time spent in condition (GBL) was higher by 7-10 seconds, than in the other conditions. In the third experiment, the mean values of the time taken by the users, in all conditions, were comparable, with an average of 22.8%. With regards to effectiveness, the findings of the first experiment showed, generally, that the mean correct answer for condition (E) was higher by 20%, than the mean for condition (NE). Users in condition (GBL) performed better than the users in the other conditions, in the second experiment. The percentage of correct answers, in the second experiment, was higher by 20% and by 34.7%, in condition (GBL), than in the (VC) and (ST), respectively. Finally, a set of empirically derived guidelines was produced for the design of usable multimodal e-Learning and edutainment interfaces.

碩士<br>國立中山大學<br>外國語文學系研究所<br>104<br>This thesis aims to contribute to the field of Conceptual Multimodal Metaphor Theory by conducting research into the (multimodal) metaphor LOVE IS A (SEA) JOUNEY in Titanic. To begin with, Lakoff’s (1993) theory of Event Structure (ES) with respect to the metaphor LOVE IS A JOURNEY is discussed. As such, the way love is conceptualized as a road journey is systematically investigated first. Nevertheless, in view of the argument that metaphor is a mainly matter of thought and only derivatively represented through verbal modality, Forceville’s (2008a) Multimodal Metaphor Theory (MMT) and Rohdin’s Film Theory (FT) are further adopted for a non-verbal analysis of Lakoff’s discussion about the love-as-a-journey metaphor in love movies. After a close examination of fifty scenes from the three acts in Titanic, a few points are made. First of all, such a deep-rooted love journey metaphor is non-verbally represented in Titanic. And, its “uniqueness” supports Forceville’s idea about non-verbal metaphor: (1) to have a stronger emotional appeal, (2) to have a greater cross-cultural access and (3) to allow for more subtle ways of communication. Additionally, it reasonably defines the way the Three-Act Structure is applied in love movies. Secondly, this study shows that the two film metaphors EVIL IS DARK and EVIL IS DOWN, which were frequently exploited in Act 2 of Titanic, were motivated to create a dramatic effect in the storyline. On the other hand, the metaphors GOOD IS UP and GOOD IS BRIGHT appeared to be strongly suggested whenever something positive was coming up. Overall, it is probable that this metaphorical dichotomy of bright/dark and up/down structures Titanic in its entirety and, therefore, makes the metaphor LOVE IS A (SEA) JOUNEY more dramatically appealing. Thirdly, the song lyrics, threes love metaphors LOVE IS CLOSENESS, LOVE IS FIRE and LOVE IS UNITY, and camera angle and framing are seen to positively contribute to the formative elements of the metaphor LOVE IS A (SEA) JOUNEY in Titanic. For example, the three love metaphors make the non-verbal love journey metaphor more emotionally intensified. This reinforced love emotion might go with the song lyrics to further show Rose’s strong belief in Jack. Undoubtedly, this subtle way of coupling the three love metaphors with the LOVE IS A (SEA) JOURNEY metaphor makes for “an esthetically pleasing experience” (Foreceville, 2013a, p.166). Finally, Lakoff’s (1993) discussion about the metaphor LOVE IS A (SEA) JOURNEY seems to be differently reinterpreted in Titanic. For example, it was always one of the couple who had to fix it out independently (and therefore continued the journey). This way of reinterpretation might run contrary to moviegoers’ expectations if an idealized model of love metaphor does exist. In the end, a proposed model is provided to demonstrate the multimodal analysis of Titanic as a whole. It aims to model the way ES, MMT, and FT interact with each other in the realization of LOVE IS A (SEA) JOURNEY in Titanic.

碩士<br>國立臺北藝術大學<br>博物館研究所<br>107<br>Museum exhibition nowadays not only solely narrates, but also accelerates the audience’s perception of exhibitions through individual’s background and interest (Roppola, 2012). Meanwhile, under the influence of cognitive linguistics, metaphor not only serves as a form of rhetoric, but also the process by which people transform abstract concepts or difficult knowledge into more concrete ways of expression and action. This is an interdisciplinary research on museology and metaphor studies, with the case study of the exhibition “Kau-Puê, Mutual Companionship in Near Future”. The study proposes a conceptual metaphorical analysis to prove how multimodal metaphors could transfer the exhibition content to create a dialogue between curators and audience. This research adapts qualitative methods to collect data from both the curator and the audience. Combining Conceptual Metaphor Theory and Blending Theory, the paper offers a theoretical proposal called the Metaphor Identification Procedure of Exhibition (MIPE), along with the curator’s interview in carrying out exhibition analysis. Moreover, 9 audiences who participated in this case study shared their in-depth perceptions about each display in “Kau-Puê, Mutual Companionship in Near Future”. The researcher uses open coding to analyze the audiences’ descriptive experience, thus simulating a platform for curator and audience to exchange viewpoints through the interpretation of multimodal metaphors applied in exhibition. Three major findings in the study indicate that, first, metonymies are widely used as a display method when exhibition designer tries to give audience contexts about certain object or artworks, be it a location, or a historical scene; whereas multimodal metaphors usually contain difficult messages, atmosphere, or feelings. Second, exhibition narrative is a contact zone connecting the object or artwork displayed, the curator, and the audience. Finally, multimodal metaphors used in displays empower the audience to better understand the exhibition narrative, whilst creating a whole new experience that is intimate, unique and meaningful to each audience.

In Images of Leadership (1991), Bolman and Deal identified four “frames” that school administrators use when making decisions: structural, symbolic, human resource and political. They discovered that the latter two frames, which focus on relationships, partnerships, and communication, were most frequently identified as predicting a school administrator’s success “as both leader and manager”(12). Strikingly, Bolman and Deal found that little emphasis and professional time are afforded to help school administrators learn about these critical frames. While there is ample logistical advice about language use, there is scant research that examines it from a theatrical perspective. The purpose of this autoethnographic study was to examine my literate life as a school administrator through the use of multimodal interaction analysis (Norris, 2004) and dramaturgical metaphors (Goffman, 1959). The study attempted to address the following research questions: (1.) How does my role as a school administrator dramaturgically define the roles I inhabit as I engage in everyday literacy practices in school? and (2.) How do I use language –both verbal and nonverbal language --to negotiate those roles with my various audiences, specifically with teachers and staff, other leaders, students and parents? The participant was myself –in my former role as an assistant principal at a suburban elementary school. Data collection and analysis began in May 2012 and concluded at the end of August 2012. Data for the study was collected through a journal based on questions using dramaturgical terms and a collection of the author’s/participant’s videotaped “performances” with various audiences. The dramaturgical journal was analyzed through Critical Discourse Analysis and deductive coding, while the videotapes were analyzed using Multimodal Interaction Analysis. Poetry was also used throughout the study to include the author’s voice, to recontextualize the experience, and to challenge the traditional prose form. The study revealed the intersection of language and leadership in the life of a school administrator. It also showed how multimodal interaction analysis and dramaturgical metaphors can help educational leaders understand their own literate lives through new lenses and how they can grow from that understanding.

碩士<br>國立中山大學<br>外國語文學系研究所<br>106<br>Drawing on twenty-four covers of the female fashion magazine, International Chinese Edition (ICE) Vogue, this thesis conducts a multimodal analysis. This thesis starts with the discussion of fashion discourse in female fashion magazines. The verbal metaphors used in the selected covers are regarded as an event embedded within event structure rather than separate cognitive metaphors and metonymies. These verbal metaphors are analyzed with Lakoff’s (1993) Event Structure (ES). In addition to the verbal metaphors found in the covers, the visual images are considered metaphors as well, which are analyzed with an overall Forceville’s (2008) theory of pictorial metaphor (PM). After the analysis of the data, several points are made in the current thesis. Firstly, out of fifty-three verbal metaphors found in the selected covers, twenty-six metaphors are under the event of CAUSES ARE FORCES. Five metaphors are under the event of EXTERNAL EVENTS ARE LARGE MOVING OBJECTS. The rest metaphors found are under ACTION IS SELF-PROPELLED MOTION (4), STATES ARE LOCATIONS (2), CHANGES ARE MOVEMENTS (1), and MEANS ARE PATHS (1). The verbally metaphorical expressions in the female fashion magazines are mostly delivered by interpreting the cause of an emotion into a physical force, such as wind, tide, and fire. Secondly, visual metaphors are analyzed with Forceville’s (2008) theory of pictorial metaphor, which divides pictorial metaphors into four substructures: contextual metaphor, hybrid metaphor, pictorial simile, and integrated metaphor. Eighteen pictorial metaphors are discovered. Out of the four substructure under PMT, only integrated metaphor (12), contextual metaphor (6) and hybrid metaphor (1) are utilized in the data. What can be assumed is that integrated metaphor and contextual metaphor are mostly used to conveyed fashion trend in female fashion discourse.