Academic literature on the topic 'Multimodal metaphors'

This thesis explores the use of multimodality to communicate retrieved results of Internet search engines. The investigation aimed to investigate suitable multimodal metaphors which would increase the level of usability of Internet search engine interfaces and enhance users` experience in the search activity. The study consisted of three experiments based on questionnaires and Internet search activities with a set of text-based and multimodal interfaces. These interfaces were implemented in two browsing platforms, named AVBRO and AVBRO II. In the first experiment, the efficiency of specific multimodal metaphors to communicate additional information of retrieved results was investigated. The experiment also sought to obtain users` views of these metaphors with a questionnaire. An experimental multimodal interface of the AVBRO platform, which communicated additional information with a combination of three 2D graphs and musical stimuli, was used as a basis for the experiment, together with the Google search engine. The results obtained led to the planning of a second experiment. The aim of this experiment was to obtain and compare the level of usability of four different experimental multimodal interfaces and one traditional text-based interface, all implemented in the AVBRO II platform. Effectiveness, efficiency and users` satisfaction were used as criteria to evaluate the usability of these interfaces. In the third and final experiment the usability analysis of a traditional text-based interface and the two most suitable experimental multimodal interfaces of the AVBRO II platform was further investigated. Learnability, errors rate, efficiency, memorability and users` satisfaction were used as criteria to evaluate the usability of these interfaces. The analysis of the results obtained from these experiments provided the basis for a set of design guidelines for the development of usable interfaces based on a multimodal approach.

This empirical work aims to investigate the impact of using multimodal communication metaphors on e-learning systems’ usability, overall user experience and affective state. The study proposed a triple evaluation approach to avoid the problem of conventional assessment relying only on usability measurements of efficiency, effectiveness and user satisfactions. Usability in that sense refers only to the functionality and pragmatic side of the product and neglects other aspects of the system. Learning is a cognitive and repetitive task, requiring learners’ attention as well as their interest. Therefore, when delivering content, in addition to the pragmatic functionality, an e-learning system should provide a constructive overall user experience and positive affective state. Doing so will ensure user engagement, facilitate the learning process and increase learners’ performance. The impact of using five different communication metaphors was evaluated in three dimensions using the proposed approach. Within the usability dimension, the evaluation criteria involved measuring system efficiency, effectiveness, user satisfaction and learning performance. Within the user experience dimension, the evaluation criteria involved measuring pragmatic aspects of the user experience, the hedonic aspects of user experience in terms of stimulation as well as identification and the overall system attractiveness. Within the affective state dimension a self-assessments manikin technique was used in conjunction with biofeedback measurements, and users’ valence, arousal and dominance were measured. The study found that system attractiveness and the hedonic user experience had a profound impact on users’ learning performance and attitude toward the tested system. Furthermore, they influenced users’ views and judgement of the system and its usability. The communication metaphors were not equal in their influence within the evaluation criteria. Empirically derived guidelines were produced for the use and integration of these metaphors in e-learning systems. The outcome of the study highlights the need to use the triple evaluation approach in the assessment of e-learning interfaces prior to their release for better adoption and acceptance by end users.

This thesis investigated the use of novel combinations of multimodal metaphors in the presentation of learning information to evaluate the effect of these combinations on the usability of e-learning interfaces and on the users' learning performance. The empirical research described in this thesis comprised three experimental phases. In the first phase, an initial experiment was carried out with 30 users to explore and compare the usability and learning performance of facially animated expressive avatars with earcons and speech, and text with graphics metaphors. The second experimental phase involved an experiment conducted with 48 users to investigate their perception of avatar's facial expressions and body gestures when presented in both the absence and presence of interactive e-learning context. In addition, the experiment aimed at evaluating the role that an avatar could play as virtual lecturer in e-learning interfaces by comparing the usability and learning performance of three different modes of interaction: speaking facially expressive virtual lecturer, speaking facially expressive full-body animated virtual lecturer, and two speaking facially expressive virtual lecturers. In the third phase, a total of 24 users experimentally examined a novel approach for the use of earcons and auditory icons in e-learning interfaces to support an animated facially expressive avatar with body gestures during the presentation of the learning material. The obtained results demonstrated the usefulness of the tested metaphors to enhance e-learning usability and to enable users to attain better learning performance. These results provided a set of empirically derived innovative guidelines for the design and use of these metaphors to generate more usable e-learning interfaces. For example, when designing avatars as animated virtual lecturers in e-learning interfaces, specific facial expression and body gestures should be incorporated due to its positive influence in enhancing learners' attitude towards the learning process.

There has been an increasing demand for commercial organisations to foster real-time interaction with customers, because harnessing customer competencies has been shown to be a major contributor towards various benefits, such as growth, innovation and competition. This may drive organisations to embrace the development of multimodal interaction and complement Electronic Customer Knowledge Management Systems (E-CKMS) with metaphors of audio-visual nature. Although the implementation of E-CKMS encounters several challenges, such as lack of trust and information overload, few empirical studies were devoted to assess the role of audio-visual metaphors, and investigate whether these technologies can be put into practice. Therefore, this thesis describes a comparative evaluation study carried out to examine the implication of incorporating multimodal metaphors into E-CKMS interfaces on not only usability of E-CKMS, but also the user's trust, knowledge and acceptance. An experimental E-CKMS platform was implemented with three different modes of interaction: Visual-only E-CKMS (VCKMS) with text and graphics, Multimodal E-CKMS (MCKMS) with speech, earcons and auditory icons and Avatar-enhanced multimodal E-CKMS (ACKMS). The three platforms were evaluated by three independent groups of twenty participants each (total=60) who carried out eight common tasks of increasing complexity and design based on three different styles. Another dependent group of forty-eight participants (n=48) was instructed to interact with the systems under similar usability conditions by performing six common tasks of two styles, and fill a questionnaire devised to measure the aspects of user acceptance. The results therein revealed that ACKMS was more usable and acceptable than both MCKMS and VCKMS, whereas MCKMS was more usable than VCKMS, but less acceptable. Inferential Statistics indicated that these results were statistically significant.

There has been an increasing demand for commercial organisations to foster real-time interaction with customers, because harnessing customer competencies has been shown to be a major contributor towards various benefits, such as growth, innovation and competition. This may drive organisations to embrace the development of multimodal interaction and complement Electronic Customer Knowledge Management Systems (E-CKMS) with metaphors of audio-visual nature. Although the implementation of E-CKMS encounters several challenges, such as lack of trust and information overload, few empirical studies were devoted to assess the role of audio-visual metaphors, and investigate whether these technologies can be put into practice. Therefore, this thesis describes a comparative evaluation study carried out to examine the implication of incorporating multimodal metaphors into E-CKMS interfaces on not only usability of E-CKMS, but also the user¿s trust, knowledge and acceptance. An experimental E-CKMS platform was implemented with three different modes of interaction: Visual-only E-CKMS (VCKMS) with text and graphics, Multimodal E-CKMS (MCKMS) with speech, earcons and auditory icons and Avatar-enhanced multimodal E-CKMS (ACKMS). The three platforms were evaluated by three independent groups of twenty participants each (total=60) who carried out eight common tasks of increasing complexity and design based on three different styles. Another dependent group of forty-eight participants (n=48) was instructed to interact with the systems under similar usability conditions by performing six common tasks of two styles, and fill a questionnaire devised to measure the aspects of user acceptance. The results therein revealed that ACKMS was more usable and acceptable than both MCKMS and VCKMS, whereas MCKMS was more usable than VCKMS, but less acceptable. Inferential Statistics indicated that these results were statistically significant.

Submitted by Rafael Santana (rafael.silvasantana@ufpe.br) on 2017-05-11T17:36:39Z No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Tese_Adiano-BC.pdf: 7401702 bytes, checksum: 6b28f8e30c070212151e0a806655bd7c (MD5)<br>Made available in DSpace on 2017-05-11T17:36:39Z (GMT). No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Tese_Adiano-BC.pdf: 7401702 bytes, checksum: 6b28f8e30c070212151e0a806655bd7c (MD5) Previous issue date: 2016-02-19<br>Este trabalho investiga a realização de metáforas multimodais a partir da interação entre textos escritos e imagens em anúncios publicitários impressos. Para tanto, partese da contribuição da Linguística Sociocognitiva, que encara a linguagem enquanto atividade de conceptualização e concebe a metáfora como elemento do pensamento, das linguagens e das ações (LAKOFF e JOHNSON, 1980 e 1999; LAKOFF, 1987 e 2008). O corpus da pesquisa foi recolhido a partir do acompanhamento, no período de julho de 2013 a junho de 2014, das cinco revistas mensais com maior tiragem nacional: Claudia, Nova Escola, Quatro Rodas, Seleções Reader´s Digest e Superinteressante. Esses títulos foram lidos com o objetivo de verificar a realização multimodal de metáforas. Adotamos as sugestões metodológicas propostas por Schmitt (2005), que têm como objetivo congregar fatores subjetivos e procedimentos metodológicos na análise de metáforas e tenta reconstruir modelos de pensamento, de linguagem e de ação. Ao final da coleta, o corpus foi composto por 30 (trinta) anúncios. As análises mostraram a realização de metáforas multimodais de quatro categorias diferentes: (i) alvo multimodal é fonte visual, (ii) alvo verbal é fonte multimodal, (iii) alvo verbal é fonte visual e (iv) alvo multimodal é fonte multimodal. Essas instanciações cumprem funções cognitivo-discursivas distintas no discurso publicitário. Além disso, os achados evidenciaram a realização de metáforas convencionais (VIDA É VIAGEM, por exemplo) e, também, de metáforas episódicas (processador [de celular] é corredor, por exemplo). Essas realizações não parecem ser aleatórias, mas cumprem, assim como as categorias descritas, propósitos definidos nos textos analisados. As categorias propostas possibilitam a reflexão acerca dos mapeamentos metafóricos e das modalidades que envolvem a sua realização e o seu uso no discurso publicitário.<br>The aim of this work is to investigate the instantiations of multimodal metaphors from the interaction between written texts and images on printed advertisements. For this purpose, we based the research on the Socio-cognitive Linguistics contribution, which postulates language as an activity of conceptualization and conceives metaphor as an element of thought, language and actions (Lakoff and Johnson, 1980 and 1999; Lakoff, 1987 and 2008). The corpus of the research was collected from monitoring, from July 2013 to June 2014, the five largest monthly magazines with national circulation: Claudia, Nova Escola, Quatro Rodas, Seleções Reader's Digest and Superinteressante. These magazines were read in order to check the realization of multimodal metaphors. We adopted methodological suggestions proposed by Schmitt (2005), which aim to gather subjective factors and methodological procedures in the analysis of metaphors and attempts to reconstruct models of thought, speech and action. At the end of the collection, the corpus was composed of thirty (30) printed advertisements. Analyses showed the realization of multimodal metaphors in four different categories: (i) multimodal target is visual source, (ii) verbal target is multimodal source, (iii) verbal target is visual source and (iv) multimodal target is multimodal source. These instantiations meet cognitive-discursive distinct functions in advertising discourse. In addition, the findings showed the performance of conventional metaphors (LIFE IS A JOURNEY, for example) and episodic metaphors (processor [of cellphones] is running, for example). These embodiments do not appear to be random, but they serve for defined purposes in the analyzed texts. The proposed categories enable reflection on the metaphorical mappings and arrangements involving their realization and their use in advertising discourse.