To see the other types of publications on this topic, follow the link: Attention based models.
Journal articles on the topic 'Attention based models'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Attention based models.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Zang, Yubin, Zhenming Yu, Kun Xu, Minghua Chen, Sigang Yang, and Hongwei Chen. "Fiber communication receiver models based on the multi-head attention mechanism." Chinese Optics Letters 21, no. 3 (2023): 030602. http://dx.doi.org/10.3788/col202321.030602.
Full text
2
Qin, Chu-Xiong, and Dan Qu. "Towards Understanding Attention-Based Speech Recognition Models." IEEE Access 8 (2020): 24358–69. http://dx.doi.org/10.1109/access.2020.2970758.
Full text
3
Cha, Peter, Paul Ginsparg, Felix Wu, Juan Carrasquilla, Peter L. McMahon, and Eun-Ah Kim. "Attention-based quantum tomography." Machine Learning: Science and Technology 3, no. 1 (2021): 01LT01. http://dx.doi.org/10.1088/2632-2153/ac362b.
Full textAbstract:
Abstract With rapid progress across platforms for quantum systems, the problem of many-body quantum state reconstruction for noisy quantum states becomes an important challenge. There has been a growing interest in approaching the problem of quantum state reconstruction using generative neural network models. Here we propose the ‘attention-based quantum tomography’ (AQT), a quantum state reconstruction using an attention mechanism-based generative network that learns the mixed state density matrix of a noisy quantum state. AQT is based on the model proposed in ‘Attention is all you need’ by Vaswani et al (2017 NIPS) that is designed to learn long-range correlations in natural language sentences and thereby outperform previous natural language processing (NLP) models. We demonstrate not only that AQT outperforms earlier neural-network-based quantum state reconstruction on identical tasks but that AQT can accurately reconstruct the density matrix associated with a noisy quantum state experimentally realized in an IBMQ quantum computer. We speculate the success of the AQT stems from its ability to model quantum entanglement across the entire quantum system much as the attention model for NLP captures the correlations among words in a sentence.
4
Fallahnejad, Zohreh, and Hamid Beigy. "Attention-based skill translation models for expert finding." Expert Systems with Applications 193 (May 2022): 116433. http://dx.doi.org/10.1016/j.eswa.2021.116433.
Full text
5
Steelman, Kelly S., Jason S. McCarley, and Christopher D. Wickens. "Theory-based Models of Attention in Visual Workspaces." International Journal of Human–Computer Interaction 33, no. 1 (2016): 35–43. http://dx.doi.org/10.1080/10447318.2016.1232228.
Full text
6
Thapa, Krishu K., Bhupinderjeet Singh, Supriya Savalkar, Alan Fern, Kirti Rajagopalan, and Ananth Kalyanaraman. "Attention-Based Models for Snow-Water Equivalent Prediction." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 21 (2024): 22969–75. http://dx.doi.org/10.1609/aaai.v38i21.30337.
Full textAbstract:
Snow Water-Equivalent (SWE)—the amount of water available if snowpack is melted—is a key decision variable used by water management agencies to make irrigation, flood control, power generation, and drought management decisions. SWE values vary spatiotemporally—affected by weather, topography, and other environmental factors. While daily SWE can be measured by Snow Telemetry (SNOTEL) stations with requisite instrumentation, such stations are spatially sparse requiring interpolation techniques to create spatiotemporal complete data. While recent efforts have explored machine learning (ML) for SWE prediction, a number of recent ML advances have yet to be considered. The main contribution of this paper is to explore one such ML advance, attention mechanisms, for SWE prediction. Our hypothesis is that attention has a unique ability to capture and exploit correlations that may exist across locations or the temporal spectrum (or both). We present a generic attention-based modeling framework for SWE prediction and adapt it to capture spatial attention and temporal attention. Our experimental results on 323 SNOTEL stations in the Western U.S. demonstrate that our attention-based models outperform other machine-learning approaches. We also provide key results highlighting the differences between spatial and temporal attention in this context and a roadmap toward deployment for generating spatially-complete SWE maps.
7
王, 紫阳. "Wind Speed Prediction Based on Attention-Combined Models." Artificial Intelligence and Robotics Research 14, no. 02 (2025): 389–96. https://doi.org/10.12677/airr.2025.142038.
Full text
8
AlOmar, Ban, Zouheir Trabelsi, and Firas Saidi. "Attention-Based Deep Learning Modelling for Intrusion Detection." European Conference on Cyber Warfare and Security 22, no. 1 (2023): 22–32. http://dx.doi.org/10.34190/eccws.22.1.1172.
Full textAbstract:
Cyber-attacks are becoming increasingly sophisticated, posing more significant challenges to traditional intrusion detection methods. The inability to prevent intrusions could compromise the credibility of security services, thereby putting data confidentiality, integrity, and availability at risk. In response to this problem, research has been conducted to apply deep learning (DL) models to intrusion detection, leveraging the new era of AI and the proven efficiency of DL in many fields. This study proposes a new intrusion detection system (IDS) based on DL, utilizing attention-based long short-term memory (AT-LSTM) and attention-based bidirectional LSTM (AT-BiLSTM) models. The time-series nature of network traffic data, which changes continuously over time, makes LSTM and BiLSTM particularly effective in handling intrusion detection. These models can capture long-term dependencies in the sequence of events, learn the patterns of normal network behaviour, and detect deviations from this behaviour that may indicate an intrusion. Also, the attention mechanism in the proposed models lets them make predictions based on the most important parts of the network traffic data. This is important for finding intrusions because network traffic data can have many different features, not all of which are important for finding an attack. The attention mechanism lets the models learn which features are most important for making accurate predictions, which improves their performance and efficiency. The UNSW-NB15 benchmark dataset is used in the study to measure and compare the effectiveness and reliability of the proposed system. This dataset contains normal and attack traffic data with a significant class imbalance. To address this issue, the study employs the Synthetic Minority Over-sampling Technique (SMOTE) to balance the dataset, thus reducing the risk of overfitting to the majority class and improving the model's performance in detecting attacks. The performance evaluation results demonstrate that the proposed models achieved a detection rate of over 93%, indicating high precision in detecting intrusions. By harnessing the power of deep learning, these models can learn and adapt to new threats over time, thus ensuring data confidentiality, integrity, and availability in today's interconnected world.
9
Tangsali, Rahul, Swapnil Chhatre, Soham Naik, Pranav Bhagwat, and Geetanjali Kale. "Evaluating Performances of Attention-Based Merge Architecture Models for Image Captioning in Indian Languages." Journal of Image and Graphics 11, no. 3 (2023): 294–301. http://dx.doi.org/10.18178/joig.11.3.294-301.
Full textAbstract:
Image captioning is a growing topic of research in which numerous advancements have been made in the past few years. Deep learning methods have been used extensively for generating textual descriptions of image data. In addition, attention-based image captioning mechanisms have also been proposed, which give state-ofthe- art results in image captioning. However, many applications and analyses of these methodologies have not been made in the case of languages from the Indian subcontinent. This paper presents attention-based merge architecture models to achieve accurate captions of images in four Indian languages- Marathi, Kannada, Malayalam, and Tamil. The widely known Flickr8K dataset was used for this project. Pre-trained Convolutional Neural Network (CNN) models and language decoder attention models were implemented, which serve as the components of the mergearchitecture proposed here. Finally, the accuracy of the generated captions was compared against the gold captions using Bilingual Evaluation Understudy (BLEU) as an evaluation metric. It was observed that the merge architectures consisting of InceptionV3 give the best results for the languages we test on, the scores discussed in the paper. Highest BLEU-1 scores obtained for each language were: 0.4939 for Marathi, 0.4557 for Kannada, 0.5082 for Malayalam, and 0.5201 for Tamil. Our proposed architectures gave much higher scores than other architectures implemented for these languages.
10
Kong, Phutphalla, Matei Mancas, Bernard Gosselin, and Kimtho Po. "DeepRare: Generic Unsupervised Visual Attention Models." Electronics 11, no. 11 (2022): 1696. http://dx.doi.org/10.3390/electronics11111696.
Full textAbstract:
Visual attention selects data considered as “interesting” by humans, and it is modeled in the field of engineering by feature-engineered methods finding contrasted/surprising/unusual image data. Deep learning drastically improved the models efficiency on the main benchmark datasets. However, Deep Neural Networks-based (DNN-based) models are counterintuitive: surprising or unusual data are by definition difficult to learn because of their low occurrence probability. In reality, DNN-based models mainly learn top-down features such as faces, text, people, or animals which usually attract human attention, but they have low efficiency in extracting surprising or unusual data in the images. In this article, we propose a new family of visual attention models called DeepRare and especially DeepRare2021 (DR21), which uses the power of DNNs’ feature extraction and the genericity of feature-engineered algorithms. This algorithm is an evolution of a previous version called DeepRare2019 (DR19) based on this common framework. DR21 (1) does not need any additional training other than the default ImageNet training, (2) is fast even on CPU, (3) is tested on four very different eye-tracking datasets showing that DR21 is generic and is always within the top models on all datasets and metrics while no other model exhibits such a regularity and genericity. Finally, DR21 (4) is tested with several network architectures such as VGG16 (V16), VGG19 (V19), and MobileNetV2 (MN2), and (5) it provides explanation and transparency on which parts of the image are the most surprising at different levels despite the use of a DNN-based feature extractor.
11
Fei Wang, Fei Wang, and Haijun Zhang Fei Wang. "Multiscale Convolutional Attention-based Residual Network Expression Recognition." 網際網路技術學刊 24, no. 5 (2023): 1169–75. http://dx.doi.org/10.53106/160792642023092405015.
Full textAbstract:
<p>Expression recognition has wide application in the fields of distance education and clinical medicine. In response to the problems of insufficient feature extraction ability of expression recognition models in current research, and the deeper the depth of the model, the more serious the loss of useful information, a residual network model with multi-scale convolutional attention is proposed. This model mainly takes the residual network as the main body, adds normalization layer and channel attention mechanism, so as to extract useful image information at multiple scales, and incorporates the Inception module and channel attention module into the residual network to enhance the feature extraction ability of the model and to prevent the loss of more useful information due to too deep network, and to improve the generalization performance of the model. From results of lots of experiments we can see that the recognition accuracy of the model in FER+ and CK+ datasets reaches 87.80% and 99.32% respectively, with better recognition performance and robustness.</p> <p>&nbsp;</p>
12
Hashemi, Seyyed Mohammad Reza. "A Survey of Visual Attention Models." Ciência e Natura 37 (December 19, 2015): 297. http://dx.doi.org/10.5902/2179460x20786.
Full textAbstract:
The present paper surveys visual attention models, showing factors’ categorization. It also studies bottom-up models in comparison to top-to-down, spatial models compared to spatial-temporal ones, obvious attention against the hidden one, and space-based models against the object-based ones. It categorizes some challenging model issues, including biological calculations, correlation with the set of eye-movement data, as well as bottom-up and top-to-down topics, explaining each in details.
13
Lee, Soohyun, and Jongyoul Park. "Attention Map-Based Automatic Masking for Object Swapping in Diffusion Models." Journal of KIISE 52, no. 4 (2025): 284–92. https://doi.org/10.5626/jok.2025.52.4.284.
Full text
14
Yue, Wang, and Li Lei. "Sentiment Analysis using a CNN-BiLSTM Deep Model Based on Attention Classification." Information 26, no. 3 (2023): 117–62. http://dx.doi.org/10.47880/inf2603-02.
Full textAbstract:
With the rapid development of the Internet, the number of social media and e-commerce platforms increased dramatically. Users from all over world share their comments and sentiments on the Internet become a new tradition. Applying natural language processing technology to analyze the text on the Internet for mining the emotional tendencies has become the main way in the social public opinion monitoring and the after-sale feedback of manufactory. Thus, the study on text sentiment analysis has shown important social significance and commercial value. Sentiment analysis is a hot research topic in the field of natural language processing and data mining in recent ten years. The paper starts with the topic of "Sentiment Analysis using a CNN-BiLSTM deep model based on attention mechanism classification". First, it conducts an in-depth investigation on the current research status and commonly used algorithms at home and abroad, and briefly introduces and analyzes the current mainstream sentiment analysis methods. As a direction of machine learning, deep learning has become a hot research topic in emotion classification in the field of natural language processing. This paper uses deep learning models to study the sentiment classification problem of short and long text sentiment classification tasks. The main research contents are as follows. Firstly, Traditional neural network based short text classification algorithms for sentiment classification is easy to find the errors. The feature dimension is too high, and the feature information of the pool layer is lost, which leads to the loss of the details of the emotion vocabulary. To solve this problem, the Word Vector Model (Word2vec), Bidirectional Long-term and Short-term Memory networks (BiLSTM) and convolutional neural network (CNN) are combined in Quora dataset. The experiment shows that the accuracy of CNN-BiLSTM model associated with Word2vec word embedding achieved 91.48%. This proves that the hybrid network model performs better than the single structure neural network in short text. Convolutional neural network (CNN) models use convolutional layers and maximum pooling or max-overtime pooling layers to extract higher-level features, while LSTM models can capture long- term dependencies between words hence are better used for text classification. However, even with the hybrid approach that leverages the powers of these two deep-learning models, the number of features to remember for classification remains huge, hence hindering the training process. Secondly, we propose an attention based CNN-BiLSTM hybrid model that capitalize on the advantages of LSTM and CNN with an additional attention mechanism in IMDB movie reviews dataset. In the experiment, under the control of single variable of Data volume and Epoch, the proposed hybrid model was compared with the results of various indicators including recall, precision, F1 score and accuracy of CNN, LSTM and CNN-LSTM in long text. When the data size was 13 k, the proposed model had the highest accuracy at 0.908, and the F1 score also showed the highest performance at 0.883. When the epoch value for obtaining the optimal accuracy of each model was 10 for CNN, 14 for LSTM, 5 for MLP and 15 epochs for CNN-LSTM, which took the longest learning time. The F1 score also showed the best performance of the proposed model at 0.906, and accuracy of the proposed model was the highest at 0.929. Finally, the experimental results show that the bidirectional long- and short-term memory convolutional neural network (BiLSTM-CNN) model based on attention mechanism can effectively improve the performance of sentiment classification of data sets when processing long-text sentiment classification tasks. Keywords: sentiment analysis, CNN, BiLSTM, attention mechanism, text classification
15
Sharada, Gupta, and N. Eshwarappa Murundi. "Breast cancer detection through attention based feature integration model." IAES International Journal of Artificial Intelligence (IJ-AI) 13, no. 2 (2024): 2254–64. https://doi.org/10.11591/ijai.v13.i2.pp2254-2264.
Full textAbstract:
Breast cancer is detected by screening mammography wherein X-rays are used to produce images of the breast. Mammograms for screening can detect breast cancer early. This research focuses on the challenges of using multi-view mammography to diagnose breast cancer. By examining numerous perspectives of an image, an attention-based feature-integration mechanism (AFIM) model that concentrates on local abnormal areas associated with cancer and displays the essential features considered for evaluation, analyzing cross-view data. This is segmented into two views the bi-lateral attention module (BAM) module integrates the left and right activation maps for a similar projection is used to create a spatial attention map that highlights the impact of asymmetries. Here the module's focus is on data gathering through mediolateral oblique (MLO) and bilateral craniocaudal (CC) for each breast to develop an attention module. The proposed AFIM model generates using spatial attention maps obtained from the identical image through other breasts to identify bilaterally uneven areas and class activation map (CAM) generated from two similar breast images to emphasize the feature channels connected to a single lesion in a breast. AFIM model may easily be included in ResNet-style architectures to develop multi-view classification models.
16
Singh, Sushant, and Ausif Mahmood. "CacheFormer: High-Attention-Based Segment Caching." AI 6, no. 4 (2025): 85. https://doi.org/10.3390/ai6040085.
Full textAbstract:
Efficiently handling long contexts in transformer-based language models with low perplexity is an active area of research. Numerous recent approaches like Linformer, Longformer, Performer, and Structured state space models (SSMs), have not fully resolved this problem. All these models strive to reduce the quadratic time complexity of the attention mechanism while minimizing the loss in quality due to the effective compression of the long context. Inspired by the cache and virtual memory principle in computers, where in case of a cache miss, not only the needed data are retrieved from the memory, but the adjacent data are also obtained, we apply this concept to handling long contexts by dividing it into small segments. In our design, we retrieve the nearby segments in an uncompressed form when high segment-level attention occurs at the compressed level. Our enhancements for handling long context include aggregating four attention mechanisms consisting of short sliding window attention, long compressed segmented attention, dynamically retrieving top-k high-attention uncompressed segments, and overlapping segments in long segment attention to avoid segment fragmentation. These enhancements result in an architecture that outperforms existing SOTA architectures with an average perplexity improvement of 8.5% over similar model sizes.
17
Wenjuan Xiao, Wenjuan Xiao, and Xiaoming Wang Wenjuan Xiao. "Attention Mechanism Based Spatial-Temporal Graph Convolution Network for Traffic Prediction." 電腦學刊 35, no. 4 (2024): 093–108. http://dx.doi.org/10.53106/199115992024083504007.
Full textAbstract:
<p>Considering the complexity of traffic systems and the challenges brought by various factors in traffic prediction, we propose a spatial-temporal graph convolutional neural network based on attention mechanism (AMSTGCN) to adapt to these dynamic changes and improve prediction accuracy. The model combines the spatial feature extraction capability of graph attention network (GAT) and the dynamic correlation learning capability of attention mechanism. By introducing the attention mechanism, the network can adaptively focus on the dependencies between different time steps and different nodes, effectively mining the dynamic spatial-temporal relationships in the traffic data. Specifically, we adopt an improved version of graph attention network (GAT_v2) in the spatial dimension, which allows the model to capture more complex dynamic spatial correlations. Furthermore, in the temporal dimension, we combine gated recurrent unit (GRU) structure with an attention mechanism to enhance the model&rsquo;s ability to process sequential data and predict traffic flow changes over prolonged periods. To validate the effectiveness of the proposed method, extensive experiments were conducted on public traffic datasets, where AMSTGCN was compared with five different benchmark models. Experimental results demonstrate that AMSTGCN exhibits superior performance on both short-term and long-term prediction tasks and outperforms other models on multiple evaluation metrics, validating its potential and practical value in the field of traffic prediction.</p> <p>&nbsp;</p>
18
Nazari, Sana, and Rafael Garcia. "Going Smaller: Attention-based models for automated melanoma diagnosis." Computers in Biology and Medicine 185 (February 2025): 109492. https://doi.org/10.1016/j.compbiomed.2024.109492.
Full text
19
Zhou, Qifeng, Xiang Liu, and Qing Wang. "Interpretable duplicate question detection models based on attention mechanism." Information Sciences 543 (January 2021): 259–72. http://dx.doi.org/10.1016/j.ins.2020.07.048.
Full text
20
Israr, Huma, Safdar Abbas Khan, Muhammad Ali Tahir, Muhammad Khuram Shahzad, Muneer Ahmad, and Jasni Mohamad Zain. "Neural Machine Translation Models with Attention-Based Dropout Layer." Computers, Materials & Continua 75, no. 2 (2023): 2981–3009. http://dx.doi.org/10.32604/cmc.2023.035814.
Full text
21
Yang, Zhifei, Wenmin Li, Fei Gao, and Qiaoyan Wen. "FAPA: Transferable Adversarial Attacks Based on Foreground Attention." Security and Communication Networks 2022 (October 29, 2022): 1–8. http://dx.doi.org/10.1155/2022/4447307.
Full textAbstract:
Deep learning models are vulnerable to attacks by adversarial examples. However, current studies are mainly limited to generating adversarial examples for specific models, and the migration of adversarial examples between different models is rarely studied. At the same time, in only studies, it is not considered that adding disturbance to the position of the image can improve the migration of adversarial examples better. As the main part of the picture, the model should give more weight to the foreground information in the recognition. Will adding more perturbations to the foreground information of the image result in a higher transfer attack rate? This paper focuses on the above problems, and proposes the FAPA algorithm, which first selects the foreground information of the image through the DINO framework, then uses the foreground information to generate M, and then uses PNA to generate the perturbation required for the whole picture. In order to show that our method attaches importance to the foreground information, we give a greater weight to the perturbation corresponding to the foreground information, and a smaller weight to the rest of the image. Finally, we optimize the generated perturbation through the gradient generated by the dual attack framework. In order to demonstrate the effectiveness of our method, we have conducted relevant comparative experiments. During the experiment, we used the three white-box ViTs models to attack the six black-box ViTs models and the three black-box CNNs models. In the transferable attack of ViTs models, the average attack success rate of our algorithm reaches 64.19%, which is much higher than 21.12% of the FGSM algorithm. In the transferable attack of CNN models, the average attack success rate of our algorithm reaches 48.07%, which is also higher than 18.65% of the FGSM algorithm. By integrating ViTs and CNNs models, the attack success rate of transfer of our algorithm reaches 56.13%, which is higher than 1.18% of the dual attack framework we refer to.
22
He, Ruifeng, Mingtian Xie, and Aixing He. "Video anomaly detection based on hybrid attention mechanism." Applied and Computational Engineering 57, no. 1 (2024): 212–17. http://dx.doi.org/10.54254/2755-2721/57/20241336.
Full textAbstract:
To improve the ability of video anomaly detection models to extract normal behavior features of samples and suppress abnormal behaviors, this paper proposes an unsupervised video anomaly detection model, which takes advantage of spatio-temporal feature fusion, storage module, attention mechanism, and 3D autoencoder model. The model utilizes autoencoder to capture scene feature maps to enhance anomaly feature extraction. These maps are merged with the original video frames, forming fundamental units constituting continuous sequences serving as the model's input. Moreover, the attention mechanism is integrated into the 3D convolutional neural network to strengthen the network's capability in extracting channel and spatial features from videos. Experimental validation is performed on a publicly accessible campus dataset, illustrating the model's superior accuracy in anomaly detection.
23
Rosenberg, Monica D., Wei-Ting Hsu, Dustin Scheinost, R. Todd Constable, and Marvin M. Chun. "Connectome-based Models Predict Separable Components of Attention in Novel Individuals." Journal of Cognitive Neuroscience 30, no. 2 (2018): 160–73. http://dx.doi.org/10.1162/jocn_a_01197.
Full textAbstract:
Although we typically talk about attention as a single process, it comprises multiple independent components. But what are these components, and how are they represented in the functional organization of the brain? To investigate whether long-studied components of attention are reflected in the brain's intrinsic functional organization, here we apply connectome-based predictive modeling (CPM) to predict the components of Posner and Petersen's influential model of attention: alerting (preparing and maintaining alertness and vigilance), orienting (directing attention to a stimulus), and executive control (detecting and resolving cognitive conflict) [Posner, M. I., & Petersen, S. E. The attention system of the human brain. Annual Review of Neuroscience, 13, 25–42, 1990]. Participants performed the Attention Network Task (ANT), which measures these three factors, and rested during fMRI scanning. CPMs tested with leave-one-subject-out cross-validation successfully predicted novel individual's overall ANT accuracy, RT variability, and executive control scores from functional connectivity observed during ANT performance. CPMs also generalized to predict participants' alerting scores from their resting-state functional connectivity alone, demonstrating that connectivity patterns observed in the absence of an explicit task contain a signature of the ability to prepare for an upcoming stimulus. Suggesting that significant variance in ANT performance is also explained by an overall sustained attention factor, the sustained attention CPM, a model defined in prior work to predict sustained attentional abilities, predicted accuracy, RT variability, and executive control from task-based data and predicted RT variability from resting-state data. Our results suggest that, whereas executive control may be closely related to sustained attention, the infrastructure that supports alerting is distinct and can be measured at rest. In the future, CPM may be applied to elucidate additional independent components of attention and relationships between the functional brain networks that predict them.
24
Guo, Yuxi. "Interpretability analysis in transformers based on attention visualization." Applied and Computational Engineering 76, no. 1 (2024): 92–102. http://dx.doi.org/10.54254/2755-2721/76/20240571.
Full textAbstract:
Self-attention is the core idea of the transformer, a kind of special structure for models to understand sentences and texts. Transformer is growing fast, but the model's internal unknowns are still out of control. In this work, the research visualizes self-attention and observes those self-attentions in some transformers. Through observation, there are five types of self-attention connections. The research classifies them as Parallel self-attention head, Radioactive self-attention head, Homogeneous self-attention head, X-type self-attention head, and Compound self-attention head. The Parallel self-attention head is the most important. The combination of different types will affect the performance of the transformer. Visualizations can indicate the location of different types. The results show that some homogeneous heads should be more varied in that case the model will perform better. A new training method is called local head training method, and the local training method may be useful during training transformer. The purpose of this study is to lay the foundation for model biology, to take other perspectives to understand transformers, and to fine-tune training methods.
25
Wang, Lei, Ed X. Wu, and Fei Chen. "EEG-based auditory attention decoding using speech-level-based segmented computational models." Journal of Neural Engineering 18, no. 4 (2021): 046066. http://dx.doi.org/10.1088/1741-2552/abfeba.
Full text
26
Kramer, Arthur F., and Andrew Jacobson. "A comparison of Space-Based and Object-Based Models of Visual Attention." Proceedings of the Human Factors Society Annual Meeting 34, no. 19 (1990): 1489–93. http://dx.doi.org/10.1177/154193129003401915.
Full text
27
Rayeesa, Mehmood, Bashir Rumaan, and J. Giri Kaiser. "Deep Generative Models: A Review." Indian Journal of Science and Technology 16, no. 7 (2023): 460–67. https://doi.org/10.17485/IJST/v16i7.2296.
Full textAbstract:
ABSTRACT <strong>Objectives:</strong> To provide insight into deep generative models and review the most prominent and efficient deep generative models, including Variational Auto-encoder (VAE) and Generative Adversarial Networks (GANs). <strong>Methods:</strong> We provide a comprehensive overview of VAEs and GANs along with their advantages and disadvantages. This paper also surveys the recently introduced Attention-based GANs and the most recently introduced Transformer based GANs. <strong>Findings:</strong> GANs have been intensively researched because of their significant advantages over VAE. Furthermore, GANs are powerful generative models that have been widely employed in a variety of fields. Though GANs have a number of advantages over VAEs, but, despite their immense popularity and success, training GANs is still difficult and has experienced a lot of setbacks. These failures include mode collapse, where the generator produces the same set of outputs for various inputs, ultimately resulting in the loss of diversity; non-convergence due to oscillatory and diverging behaviors of the generator and discriminator during the training phase; and vanishing or exploding gradients, where learning either ceases to occur or occurs very slowly. Recently, some attention-based GANs and Transformer-based GANs have also been proposed for high-fidelity image generation. <strong>Novelty:</strong> Unlike previous survey articles, which often focus on all DGMs and dive into their complicated aspects, this work focuses on the most prominent DGMs, VAEs, and GANs and provides a theoretical understanding of them. Furthermore, because GAN is now the most extensively used DGM being studied by the academic community, the literature on it needs to be explored more. Moreover, while numerous articles on GANs are available, none have analyzed the most recent attention-based GANs and Transformer-based GANs. So, in this study, we review the recently introduced attention-based GANs and Transformer-based GANs, the literature related to which has not been reviewed by any survey paper. <strong>Keywords:</strong> Variational Autoencoder; Generative Adversarial Networks; Autoencoder; Transformer; Self-Attention
28
Yeom, Hong-gi, and Kyung-min An. "A Simplified Query-Only Attention for Encoder-Based Transformer Models." Applied Sciences 14, no. 19 (2024): 8646. http://dx.doi.org/10.3390/app14198646.
Full textAbstract:
Transformer models have revolutionized fields like Natural Language Processing (NLP) by enabling machines to accurately understand and generate human language. However, these models’ inherent complexity and limited interpretability pose barriers to their broader adoption. To address these challenges, we propose a simplified query-only attention mechanism specifically for encoder-based transformer models to reduce complexity and improve interpretability. Unlike conventional attention mechanisms, which rely on query (Q), key (K), and value (V) vectors, our method uses only the Q vector for attention calculation. This approach reduces computational complexity while maintaining the model’s ability to capture essential relationships, enhancing interpretability. We evaluated the proposed query-only attention on an EEG conformer model, a state-of-the-art architecture for EEG signal classification. We demonstrated that it performs comparably to the original QKV attention mechanism, while simplifying the model’s architecture. Our findings suggest that query-only attention offers a promising direction for the development of more efficient and interpretable transformer-based models, with potential applications across various domains beyond NLP.
29
Hanafi, Hanafi, Andri Pranolo, Yingchi Mao, Taqwa Hariguna, Leonel Hernandez, and Nanang Fitriana Kurniawan. "IDSX-Attention: Intrusion detection system (IDS) based hybrid MADE-SDAE and LSTM-Attention mechanism." International Journal of Advances in Intelligent Informatics 9, no. 1 (2023): 121. http://dx.doi.org/10.26555/ijain.v9i1.942.
Full textAbstract:
An Intrusion Detection System (IDS) is essential for automatically monitoring cyber-attack activity. Adopting machine learning to develop automatic cyber attack detection has become an important research topic in the last decade. Deep learning is a popular machine learning algorithm recently applied in IDS applications. The adoption of complex layer algorithms in the term of deep learning has been applied in the last five years to increase IDS detection effectiveness. Unfortunately, most deep learning models generate a large number of false negatives, leading to dominant mistake detection that can affect the performance of IDS applications. This paper aims to integrate a statistical model to remove outliers in pre-processing, SDAE, responsible for reducing data dimensionality, and LSTM-Attention, responsible for producing attack classification tasks. The model was implemented into the NSL-KDD dataset and evaluated using Accuracy, F1, Recall, and Confusion metrics measures. The results showed that the proposed IDSX-Attention outperformed the baseline model, SDAE, LSTM, PCA-LSTM, and Mutual Information (MI)-LSTM, achieving more than a 2% improvement on average. This study demonstrates the potential of the proposed IDSX-Attention, particularly as a deep learning approach, in enhancing the effectiveness of IDS and addressing the challenges in cyber threat detection. It highlights the importance of integrating statistical models, deep learning, and dimensionality reduction mechanisms to improve IDS detection. Further research can explore the integration of other deep learning algorithms and datasets to validate the proposed model's effectiveness and improve the performance of IDS.
30
Sun, Wenhao, Xue-Mei Dong, Benlei Cui, and Jingqun Tang. "Attentive Eraser: Unleashing Diffusion Model’s Object Removal Potential via Self-Attention Redirection Guidance." Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 19 (2025): 20734–42. https://doi.org/10.1609/aaai.v39i19.34285.
Full textAbstract:
Recently, diffusion models have emerged as promising newcomers in the field of generative models, shining brightly in image generation. However, when employed for object removal tasks, they still encounter issues such as generating random artifacts and the incapacity to repaint foreground object areas with appropriate content after removal. To tackle these problems, we propose Attentive Eraser, a tuning-free method to empower pre-trained diffusion models for stable and effective object removal. Firstly, in light of the observation that the self-attention maps influence the structure and shape details of the generated images, we propose Attention Activation and Suppression (ASS), which re-engineers the self-attention mechanism within the pre-trained diffusion models based on the given mask, thereby prioritizing the background over the foreground object during the reverse generation process. Moreover, we introduce Self-Attention Redirection Guidance (SARG), which utilizes the self-attention redirected by ASS to guide the generation process, effectively removing foreground objects within the mask while simultaneously generating content that is both plausible and coherent. Experiments demonstrate the stability and effectiveness of Attentive Eraser in object removal across a variety of pre-trained diffusion models, outperforming even training-based methods. Furthermore, Attentive Eraser can be implemented in various diffusion model architectures and checkpoints, enabling excellent scalability.
31
Kristensen, Terje. "Towards Spike based Models of Visual Attention in the Brain." International Journal of Adaptive, Resilient and Autonomic Systems 6, no. 2 (2015): 117–38. http://dx.doi.org/10.4018/ijaras.2015070106.
Full textAbstract:
A numerical solution of Hodgkin Huxley equations is presented to simulate the spiking behavior of a biological neuron. The solution is illustrated by building a graphical chart interface to finely tune the behavior of the neuron under different stimulations. In addition, a Multi-Agent System (MAS) has been developed to simulate the Visual Attention Network Model of the brain. Tasks are assigned to the agents according to the Attention Network Theory, developed by neuroscientists. A sequential communication model based on simple objects has been constructed, aiming to show the relations and the workflow between the different visual attention networks. Each agent is being used as an analogy to a role or function of the visual attention systems in the brain. Some experimental results based on this model have been presented in an earlier paper. The two approaches are at the moment not integrated. The long term goal is to develop an integrated parallel layered object model of the visual attention process, as a tool for simulating neuron interactions described by Hodgkin Huxley's equations or the Leaky-Integrate-and-Fire model.
32
Chandrasekaran, Ganesh, Mandalapu Kalpana Chowdary, Jyothi Chinna Babu, Ajmeera Kiran, Kotthuru Anil Kumar, and Seifedine Kadry. "Deep learning-based attention models for sarcasm detection in text." International Journal of Electrical and Computer Engineering (IJECE) 14, no. 6 (2024): 6786. http://dx.doi.org/10.11591/ijece.v14i6.pp6786-6796.
Full textAbstract:
Finding sarcastic statements has recently drawn a lot of curiosity in social media, mainly because sarcastic tweets may include favorable phrases that fill in unattractive or undesirable attributes. As the internet becomes increasingly ingrained in our daily lives, many multimedia information is being produced online. Much of the information recorded mostly on the internet is textual data. It is crucial to comprehend people's sentiments. However, sarcastic content will hinder the effectiveness of sentiment analysis systems. Correctly identifying sarcasm and correctly predicting people's motives are extremely important. Sarcasm is particularly hard to recognize, both by humans and by machines. We employ the deep bi-directional long-short memory (Bi-LSTM) and a hybrid architecture of the convolution neural network+Bi-LSTM (CNN+Bi-LSTM) with attention networks for identifying sarcastic remarks in a corpus. Using the SarcasmV2 dataset, we test the efficacy of deep learning methods BiLSTM, and CNN+BiLSTM with attention network) for the task of identifying text sarcasm. The suggested approach incorporating deep networks is consistent with various recent and advanced techniques for sarcasm detection. With attention processes, the improved CNN+Bi-LSTM model achieved an accuracy rate of 91.76%, which is a notable increase over earlier research.
33
Sun, Xinhao. "Application of Attention-Based LSTM Hybrid Models for Stock Price Prediction." Advances in Economics, Management and Political Sciences 104, no. 1 (2024): 46–60. http://dx.doi.org/10.54254/2754-1169/104/2024ed0152.
Full textAbstract:
The stock market plays a pivotal role in the national economy, while the application of artificial intelligence (AI) in stock price prediction has gained traction. This paper evalu-ates the performance of five advanced deep learning (DL) models: Long Short-Term Memory (LSTM), Self-attention, Convolutional Neural Network-LSTM with attention (CNN-LSTM-attention), Gated Recurrent Unit-LSTM with attention (GRU-LSTM-attention), and CNN-Bidirectional LSTM-GRU with attention (CNN-BiLSTM-GRU-attention), utilizing a decade of data on Amazons closing prices. Our results show that the CNN-BiLSTM-GRU-attention model exhibits superior performance, achieving a root mean square error (RMSE) of 1.054589 and a coefficient of determination (R2) of 0.970123, indicative of its proficiency in handling intricate financial data. This papers significance lies in its validation of the effectiveness of attention-based ensemble models in stock market prediction, as well as the introduction of the innovative application of the CNN-BiLSTM-GRU-attention model in financial forecast-ing, which holds potential for wide-ranging applicability.
34
Kamalov, Firuz, Inga Zicmane, Murodbek Safaraliev, Linda Smail, Mihail Senyuk, and Pavel Matrenin. "Attention-Based Load Forecasting with Bidirectional Finetuning." Energies 17, no. 18 (2024): 4699. http://dx.doi.org/10.3390/en17184699.
Full textAbstract:
Accurate load forecasting is essential for the efficient and reliable operation of power systems. Traditional models primarily utilize unidirectional data reading, capturing dependencies from past to future. This paper proposes a novel approach that enhances load forecasting accuracy by fine tuning an attention-based model with a bidirectional reading of time-series data. By incorporating both forward and backward temporal dependencies, the model gains a more comprehensive understanding of consumption patterns, leading to improved performance. We present a mathematical framework supporting this approach, demonstrating its potential to reduce forecasting errors and improve robustness. Experimental results on real-world load datasets indicate that our bidirectional model outperforms state-of-the-art conventional unidirectional models, providing a more reliable tool for short and medium-term load forecasting. This research highlights the importance of bidirectional context in time-series forecasting and its practical implications for grid stability, economic efficiency, and resource planning.
35
Zheng, Guoqiang, Tianle Zhao, and Yaohui Liu. "Cloud Removal in the Tibetan Plateau Region Based on Self-Attention and Local-Attention Models." Sensors 24, no. 23 (2024): 7848. https://doi.org/10.3390/s24237848.
Full textAbstract:
Optical remote sensing images have a wide range of applications but are often affected by cloud cover, which interferes with subsequent analysis. Therefore, cloud removal has become indispensable in remote sensing data processing. The Tibetan Plateau, as a sensitive region to climate change, plays a crucial role in the East Asian water cycle and regional climate due to its snow cover. However, the rich ice and snow resources, rapid snow condition changes, and active atmospheric convection in the plateau as well as its surrounding mountainous areas, make optical remote sensing prone to cloud interference. This is particularly significant when monitoring snow cover changes, where cloud removal becomes essential considering the complex terrain and unique snow characteristics of the Tibetan Plateau. This paper proposes a novel Multi-Scale Attention-based Cloud Removal Model (MATT). The model integrates global and local information by incorporating multi-scale attention mechanisms and local interaction modules, enhancing the contextual semantic relationships and improving the robustness of feature representation. To improve the segmentation accuracy of cloud- and snow-covered regions, a cloud mask is introduced in the local-attention module, combined with the local interaction module to modulate and reconstruct fine-grained details. This enables the simultaneous representation of both fine-grained and coarse-grained features at the same level. With the help of multi-scale fusion modules and selective attention modules, MATT demonstrates excellent performance on both the Sen2_MTC_New and XZ_Sen2_Dataset datasets. Particularly on the XZ_Sen2_Dataset, it achieves outstanding results: PSNR = 29.095, SSIM = 0.897, FID = 125.328, and LPIPS = 0.356. The model shows strong cloud removal capabilities in cloud- and snow-covered areas in mountainous regions while effectively preserving snow information, and providing significant support for snow cover change studies.
36
Alsayadi, Hamzah A., Abdelaziz A. Abdelhamid, Islam Hegazy, and Zaki T. Fayed. "Non-diacritized Arabic speech recognition based on CNN-LSTM and attention-based models." Journal of Intelligent & Fuzzy Systems 41, no. 6 (2021): 6207–19. http://dx.doi.org/10.3233/jifs-202841.
Full textAbstract:
Arabic language has a set of sound letters called diacritics, these diacritics play an essential role in the meaning of words and their articulations. The change in some diacritics leads to a change in the context of the sentence. However, the existence of these letters in the corpus transcription affects the accuracy of speech recognition. In this paper, we investigate the effect of diactrics on the Arabic speech recognition based end-to-end deep learning. The applied end-to-end approach includes CNN-LSTM and attention-based technique presented in the state-of-the-art framework namely, Espresso using Pytorch. In addition, and to the best of our knowledge, the approach of CNN-LSTM with attention-based has not been used in the task of Arabic Automatic speech recognition (ASR). To fill this gap, this paper proposes a new approach based on CNN-LSTM with attention based method for Arabic ASR. The language model in this approach is trained using RNN-LM and LSTM-LM and based on nondiacritized transcription of the speech corpus. The Standard Arabic Single Speaker Corpus (SASSC), after omitting the diacritics, is used to train and test the deep learning model. Experimental results show that the removal of diacritics decreased out-of-vocabulary and perplexity of the language model. In addition, the word error rate (WER) is significantly improved when compared to diacritized data. The achieved average reduction in WER is 13.52%.
37
Guang, Jiahe, Xingrui He, Zeng Li, and Shiyu He. "Road Pothole Detection Model Based on Local Attention Resnet18-CNN-LSTM." Theoretical and Natural Science 42, no. 1 (2024): 131–38. http://dx.doi.org/10.54254/2753-8818/42/20240669.
Full textAbstract:
Abstract. In response to the low detection accuracy and slow speed of existing road pothole detection methods, a road pothole classification detection model based on local attention Resnet18-CNN-LSTM (Long Short-Term Memory network) is proposed. On the basis of Resnet18, a local attention mechanism and a CNN-LSTM combined model are added to propose a road pothole detection model based on local attention Resnet18-CNN-LSTM. The local attention mechanism is used to accurately extract specific target feature values, CNN is used to extract the spatial features of the input data, and LSTM enhances the detection model's extraction of sequential features and performs classification, thereby improving the accuracy of the road and pothole model. Experimental results show that on the training set, the accuracy of the local attention mechanism-based ResNet18-CNN-LSTM model reached 99.2188%, which is an increase of 0.7813% and 2.3438% compared to the ResNet34-CNN-LSTM and ResNet50-CNN-LSTM models under the same conditions, respectively. On the test set, the model's accuracy was 93.4437%, an increase of 0.5437% and 1.9867% compared to the ResNet34-CNN-LSTM and ResNet50-CNN-LSTM models, respectively. After dealing with overfitting issues through early stopping, the detection accuracy of this model has significantly improved compared to the detection models based on ResNet34 and ResNet50, with an increase of 1.2% and 1.49% respectively. The model shows faster processing speed in identification time, effectively retains the correlation and sequence features of the data, overcomes the problem of gradient disappearance in deep networks, and thereby enhances the extraction capability of local target features of road pothole images. The above results indicate that the local attention mechanism-based ResNet18-CNN-LSTM model shows superior performance in road pothole detection.
38
Zhou, Jiawei. "Predicting Stock Price by Using Attention-Based Hybrid LSTM Model." Asian Journal of Basic Science & Research 06, no. 02 (2024): 145–58. http://dx.doi.org/10.38177/ajbsr.2024.6211.
Full textAbstract:
The financial markets are inherently complex and dynamic, characterized by high volatility and the influence of numerous factors. Predicting stock price trends is a challenging endeavor that has been significantly advanced by the advent of machine learning techniques. This study investigates the effectiveness of various models, including Long Short-Term Memory (LSTM), XGBoost, Support Vector Machine (SVM), and hybrid models combining LSTM with XGBoost and SVM, in forecasting stock prices. Our results indicate that the LSTM model outperforms others, demonstrating superior predictive accuracy. Among the hybrid models, LSTM combined with XGBoost shows the best performance. Despite these findings, the study identifies several areas for further improvement, such as enhanced feature engineering, advanced hybrid models, refined attention mechanisms, improved model interpretability, data augmentation, and real-time prediction capabilities. This research contributes valuable insights into the application of hybrid attention-based LSTM models in financial forecasting, highlighting their potential and areas for future enhancement.
39
Shin, Yehjin, Jeongwhan Choi, Hyowon Wi, and Noseong Park. "An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 8 (2024): 8984–92. http://dx.doi.org/10.1609/aaai.v38i8.28747.
Full textAbstract:
Sequential recommendation (SR) models based on Transformers have achieved remarkable successes. The self-attention mechanism of Transformers for computer vision and natural language processing suffers from the oversmoothing problem, i.e., hidden representations becoming similar to tokens. In the SR domain, we, for the first time, show that the same problem occurs. We present pioneering investigations that reveal the low-pass filtering nature of self-attention in the SR, which causes oversmoothing. To this end, we propose a novel method called Beyond Self-Attention for Sequential Recommendation (BSARec), which leverages the Fourier transform to i) inject an inductive bias by considering fine-grained sequential patterns and ii) integrate low and high-frequency information to mitigate oversmoothing. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models. We test our proposed approach through extensive experiments on 6 benchmark datasets. The experimental results demonstrate that our model outperforms 7 baseline methods in terms of recommendation performance. Our code is available at https://github.com/yehjin-shin/BSARec.
40
Wang, Yuming, Yu Li, and Hua Zou. "Masked Face Recognition System Based on Attention Mechanism." Information 14, no. 2 (2023): 87. http://dx.doi.org/10.3390/info14020087.
Full textAbstract:
With the continuous development of deep learning, the face recognition field has also developed rapidly. However, with the massive popularity of COVID-19, face recognition with masks is a problem that is now about to be tackled in practice. In recognizing a face wearing a mask, the mask obscures most of the facial features of the face, resulting in the general face recognition model only capturing part of the facial information. Therefore, existing face recognition models are usually ineffective in recognizing faces wearing masks. This article addresses this problem in the existing face recognition model and proposes an improvement of Facenet. We use ConvNeXt-T as the backbone of the network model and add the ECA (Efficient Channel Attention) mechanism. This enhances the feature extraction of the unobscured part of the face to obtain more useful information, while avoiding dimensionality reduction and not increasing the model complexity. We design new face recognition models by investigating the effects of different attention mechanisms on face mask recognition models and the effects of different data set ratios on experimental results. In addition, we construct a large set of faces wearing masks so that we can efficiently and quickly train the model. Through experiments, our model proved to be 99.76% accurate for real faces wearing masks. A combined accuracy of 99.48% for extreme environments such as too high or lousy contrast and brightness.
41
Kamal, Saurabh, Sahil Sharma, Vijay Kumar, Hammam Alshazly, Hany S. Hussein, and Thomas Martinetz. "Trading Stocks Based on Financial News Using Attention Mechanism." Mathematics 10, no. 12 (2022): 2001. http://dx.doi.org/10.3390/math10122001.
Full textAbstract:
Sentiment analysis of news headlines is an important factor that investors consider when making investing decisions. We claim that the sentiment analysis of financial news headlines impacts stock market values. Hence financial news headline data are collected along with the stock market investment data for a period of time. Using Valence Aware Dictionary and Sentiment Reasoning (VADER) for sentiment analysis, the correlation between the stock market values and sentiments in news headlines is established. In our experiments, the data on stock market prices are collected from Yahoo Finance and Kaggle. Financial news headlines are collected from the Wall Street Journal, Washington Post, and Business-Standard website. To cope with such a massive volume of data and extract useful information, various embedding methods, such as Bag-of-words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF), are employed. These are then fed into machine learning models such as Naive Bayes and XGBoost as well as deep learning models such as Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM). Various natural language processing, andmachine and deep learning algorithms are considered in our study to achieve the desired outcomes and to attain superior accuracy than the current state-of-the-art. Our experimental study has shown that CNN (80.86%) and LSTM (84%) are the best performing models in relation to machine learning models, such as Support Vector Machine (SVM) (50.3%), Random Forest (67.93%), and Naive Bayes (59.79%). Moreover, two novel methods, BERT and RoBERTa, were applied with the expectation of better performance than all the other models, and they did exceptionally well by achieving an accuracy of 90% and 88%, respectively.
42
Si, Nianwen, Wenlin Zhang, Dan Qu, Xiangyang Luo, Heyu Chang, and Tong Niu. "Spatial-Channel Attention-Based Class Activation Mapping for Interpreting CNN-Based Image Classification Models." Security and Communication Networks 2021 (May 31, 2021): 1–13. http://dx.doi.org/10.1155/2021/6682293.
Full textAbstract:
Convolutional neural network (CNN) has been applied widely in various fields. However, it is always hindered by the unexplainable characteristics. Users cannot know why a CNN-based model produces certain recognition results, which is a vulnerability of CNN from the security perspective. To alleviate this problem, in this study, the three existing feature visualization methods of CNN are analyzed in detail firstly, and a unified visualization framework for interpreting the recognition results of CNN is presented. Here, class activation weight (CAW) is considered as the most important factor in the framework. Then, the different types of CAWs are further analyzed, and it is concluded that a linear correlation exists between them. Finally, on this basis, a spatial-channel attention-based class activation mapping (SCA-CAM) method is proposed. This method uses different types of CAWs as attention weights and combines spatial and channel attentions to generate class activation maps, which is capable of using richer features for interpreting the results of CNN. Experiments on four different networks are conducted. The results verify the linear correlation between different CAWs. In addition, compared with the existing methods, the proposed method SCA-CAM can effectively improve the visualization effect of the class activation map with higher flexibility on network structure.
43
Zhang, Mengya, Yuan Zhang, and Qinghui Zhang. "Attention-Mechanism-Based Models for Unconstrained Face Recognition with Mask Occlusion." Electronics 12, no. 18 (2023): 3916. http://dx.doi.org/10.3390/electronics12183916.
Full textAbstract:
Masks cover most areas of the face, resulting in a serious loss of facial identity information; thus, how to alleviate or eliminate the negative impact of occlusion is a significant problem in the field of unconstrained face recognition. Inspired by the successful application of attention mechanisms and capsule networks in computer vision, we propose ECA-Inception-Resnet-Caps, which is a novel framework based on Inception-Resnet-v1 for learning discriminative face features in unconstrained mask-wearing conditions. Firstly, Squeeze-and-Excitation (SE) modules and Efficient Channel Attention (ECA) modules are applied to Inception-Resnet-v1 to increase the attention on unoccluded face areas, which is used to eliminate the negative impact of occlusion during feature extraction. Secondly, the effects of the two attention mechanisms on the different modules in Inception-Resnet-v1 are compared and analyzed, which is the foundation for further constructing the ECA-Inception-Resnet-Caps framework. Finally, ECA-Inception-Resnet-Caps is obtained by improving Inception-Resnet-v1 with capsule modules, which is explored to increase the interpretability and generalization of the model after reducing the negative impact of occlusion. The experimental results demonstrate that both attention mechanisms and the capsule network can effectively enhance the performance of Inception-Resnet-v1 for face recognition in occlusion tasks, with the ECA-Inception-Resnet-Caps model being the most effective, achieving an accuracy of 94.32%, which is 1.42% better than the baseline model.
44
Xue, Mengfan, Minghao Chen, Dongliang Peng, Yunfei Guo, and Huajie Chen. "One Spatio-Temporal Sharpening Attention Mechanism for Light-Weight YOLO Models Based on Sharpening Spatial Attention." Sensors 21, no. 23 (2021): 7949. http://dx.doi.org/10.3390/s21237949.
Full textAbstract:
Attention mechanisms have demonstrated great potential in improving the performance of deep convolutional neural networks (CNNs). However, many existing methods dedicate to developing channel or spatial attention modules for CNNs with lots of parameters, and complex attention modules inevitably affect the performance of CNNs. During our experiments of embedding Convolutional Block Attention Module (CBAM) in light-weight model YOLOv5s, CBAM does influence the speed and increase model complexity while reduce the average precision, but Squeeze-and-Excitation (SE) has a positive impact in the model as part of CBAM. To replace the spatial attention module in CBAM and offer a suitable scheme of channel and spatial attention modules, this paper proposes one Spatio-temporal Sharpening Attention Mechanism (SSAM), which sequentially infers intermediate maps along channel attention module and Sharpening Spatial Attention (SSA) module. By introducing sharpening filter in spatial attention module, we propose SSA module with low complexity. We try to find a scheme to combine our SSA module with SE module or Efficient Channel Attention (ECA) module and show best improvement in models such as YOLOv5s and YOLOv3-tiny. Therefore, we perform various replacement experiments and offer one best scheme that is to embed channel attention modules in backbone and neck of the model and integrate SSAM into YOLO head. We verify the positive effect of our SSAM on two general object detection datasets VOC2012 and MS COCO2017. One for obtaining a suitable scheme and the other for proving the versatility of our method in complex scenes. Experimental results on the two datasets show obvious promotion in terms of average precision and detection performance, which demonstrates the usefulness of our SSAM in light-weight YOLO models. Furthermore, visualization results also show the advantage of enhancing positioning ability with our SSAM.
45
Amin, Rashid. "Urdu Sentiment Analysis Using Deep Attention-based Technique." Foundation University Journal of Engineering and Applied Sciences <br><i style="color:yellow;">(HEC Recognized Y Category , ISSN 2706-7351)</i> 3, no. 1 (2022): 7. http://dx.doi.org/10.33897/fujeas.v3i1.564.
Full textAbstract:
Sentiment analysis (SA) is a process that aims to classify text into positive, negative, or neutral categories. It has recently gained the research community's attention because of the abundance of opinion data to be processed for better understanding and decision-making. Deep learning techniques have recently shown tremendous performance, with a high tendency to reveal the underlying semantic meaning of text inputs. Since deep learning techniques are seen as black boxes, their effectiveness comes in the form of interpretability. The major goal of this article is to create an Urdu SA model that can comprehend review semantics without the need for language resources. The proposed model is tested on reviews to extract significant words using various scenarios and architectures. By emphasizing the most informative terms to the class label, the results demonstrated the suggested model's capacity to interpret a given review. Furthermore, the suggested models provide a visualization option for an intelligible explanation of the result. The impact of using transfer learning on the problem of Urdu SA is also investigated in this article.
46
Zhou, Lixin, Zhenyu Zhang, Laijun Zhao, and Pingle Yang. "Attention-based BiLSTM models for personality recognition from user-generated content." Information Sciences 596 (June 2022): 460–71. http://dx.doi.org/10.1016/j.ins.2022.03.038.
Full text
47
Zhang, Lin, Huapeng Qin, Junqi Mao, Xiaoyan Cao, and Guangtao Fu. "High temporal resolution urban flood prediction using attention-based LSTM models." Journal of Hydrology 620 (May 2023): 129499. http://dx.doi.org/10.1016/j.jhydrol.2023.129499.
Full text
48
Wang, Zhenyi, Pengfei Yang, Linwei Hu, et al. "SLAPP: Subgraph-level attention-based performance prediction for deep learning models." Neural Networks 170 (February 2024): 285–97. http://dx.doi.org/10.1016/j.neunet.2023.11.043.
Full text
49
Hao, Cuiping, and Ting Yang. "Deep Collaborative Online Learning Resource Recommendation Based on Attention Mechanism." Scientific Programming 2022 (March 24, 2022): 1–10. http://dx.doi.org/10.1155/2022/3199134.
Full textAbstract:
In view of the lack of hierarchical and systematic resource recommendation caused by rich online learning resources and many learning platforms, an attention-based ADCF online learning resource recommendation model is proposed by introducing the attention mechanism into a deep collaborative DCF model. Experimental results show that the proposed ADCF model enables an accurate recommendation of online learning resources, reaching 0.626 and 0.339 on the HR and NDCG metrics, respectively, compared to the DCF models before improved, up by 1.31% and 1.25%, and the proposed ADCF models by 1.79%, 2.17%, and 2.32%, respectively, compared to the IUNeu and NeuCF models.
50
Ashtari, Amirsaman, Chang Wook Seo, Cholmin Kang, Sihun Cha, and Junyong Noh. "Reference Based Sketch Extraction via Attention Mechanism." ACM Transactions on Graphics 41, no. 6 (2022): 1–16. http://dx.doi.org/10.1145/3550454.3555504.
Full textAbstract:
We propose a model that extracts a sketch from a colorized image in such a way that the extracted sketch has a line style similar to a given reference sketch while preserving the visual content identically to the colorized image. Authentic sketches drawn by artists have various sketch styles to add visual interest and contribute feeling to the sketch. However, existing sketch-extraction methods generate sketches with only one style. Moreover, existing style transfer models fail to transfer sketch styles because they are mostly designed to transfer textures of a source style image instead of transferring the sparse line styles from a reference sketch. Lacking the necessary volumes of data for standard training of translation systems, at the core of our GAN-based solution is a self-reference sketch style generator that produces various reference sketches with a similar style but different spatial layouts. We use independent attention modules to detect the edges of a colorized image and reference sketch as well as the visual correspondences between them. We apply several loss terms to imitate the style and enforce sparsity in the extracted sketches. Our sketch-extraction method results in a close imitation of a reference sketch style drawn by an artist and outperforms all baseline methods. Using our method, we produce a synthetic dataset representing various sketch styles and improve the performance of auto-colorization models, in high demand in comics. The validity of our approach is confirmed via qualitative and quantitative evaluations.