To see the other types of publications on this topic, follow the link: BERT.
Journal articles on the topic 'BERT'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'BERT.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Bicca, Aline Brugalli, and Lezilda Carvalho Torgan. "Novos registros de Eunotia Ehrenberg (Eunotiaceae-Bacillariophyta) para o Estado do Rio Grande do Sul e Brasil." Acta Botanica Brasilica 23, no. 2 (2009): 427–35. http://dx.doi.org/10.1590/s0102-33062009000200014.
Full textAbstract:
O trabalho tem como objetivo apresentar as características morfológicas, e/ou estruturais e métricas de 12 espécies de Eunotia (E. batavica A. Berg, E. deficiens Metz., Lange-Bert & García-Rodr., E. genuflexa Nörpel-Sch., E. hepaticola Lang-Bert. & Wydrz., E. herzogii Krasske, E. mucophila (Lange-Bert., Nörpel-Sch. & Alles) Lange-Bert., E. pileus Ehr., E. pirla Carter & Flower, E. schwabei Krasske, E. subarcuatoides Alles, Nörpel-Sch. & Lange-Bert., E. transfuga Metz. & Lange-Bert. e E. yanomami Metz. & Lange-Bert.) encontradas nas áreas da Lagoa do Casamento e dos Butiazais de Tapes, entre as coordenadas 30º10'-30º40'S e 50º30'-51º30'W, na Planície Costeira do Rio Grande do Sul. São incluídos comentários sobre a distribuição e os ambientes onde os táxons foram encontrados.
2
Smith, Rod. "Bert." Baffler 6 (November 1994): 90. http://dx.doi.org/10.1162/bflr.1994.6.90.
Full text
3
Nicolae, Dragoş Constantin, Rohan Kumar Yadav, and Dan Tufiş. "A Lite Romanian BERT: ALR-BERT." Computers 11, no. 4 (2022): 57. http://dx.doi.org/10.3390/computers11040057.
Full textAbstract:
Large-scale pre-trained language representation and its promising performance in various downstream applications have become an area of interest in the field of natural language processing (NLP). There has been huge interest in further increasing the model’s size in order to outperform the best previously obtained performances. However, at some point, increasing the model’s parameters may lead to reaching its saturation point due to the limited capacity of GPU/TPU. In addition to this, such models are mostly available in English or a shared multilingual structure. Hence, in this paper, we propose a lite BERT trained on a large corpus solely in the Romanian language, which we called “A Lite Romanian BERT (ALR-BERT)”. Based on comprehensive empirical results, ALR-BERT produces models that scale far better than the original Romanian BERT. Alongside presenting the performance on downstream tasks, we detail the analysis of the training process and its parameters. We also intend to distribute our code and model as an open source together with the downstream task.
4
Yu, Daegon, Yongyeon Kim, Sangwoo Han, and Byung-Won On. "CLES-BERT: Contrastive Learning-based BERT Model for Automated Essay Scoring." Journal of Korean Institute of Information Technology 21, no. 4 (2023): 31–43. http://dx.doi.org/10.14801/jkiit.2023.21.4.31.
Full text
5
Angger Saputra, Revelin, and Yuliant Sibaroni. "Multilabel Hate Speech Classification in Indonesian Political Discourse on X using Combined Deep Learning Models with Considering Sentence Length." Jurnal Ilmu Komputer dan Informasi 18, no. 1 (2025): 113–25. https://doi.org/10.21609/jiki.v18i1.1440.
Full textAbstract:
Hate speech, as public expression of hatred or offensive discourse targeting race, religion, gender, or sexual orientation, is widespread on social media. This study assesses BERT-based models for multi-label hate speech detection, emphasizing how text length impacts model performance. Models tested include BERT, BERT-CNN, BERT-LSTM, BERT-BiLSTM, and BERT with two LSTM layers. Overall, BERT-BiLSTM achieved the highest (82.00%) and best performance on longer texts (83.20% ) with high and , highlighting its ability to capture nuanced context. BERT-CNN excelled in shorter texts, achieving the highest (79.80%) and an of 79.10%, indicating its effectiveness in extracting features in brief content. BERT-LSTM showed balanced and across text lengths, while BERT-BiLSTM, although high in r, had slightly lower on short texts due to its reliance on broader context. These results highlight the importance of model selection based on text characteristics: BERT-BiLSTM is ideal for nuanced analysis in longer texts, while BERT-CNN better captures key features in shorter content.
6
Xu, Huatao, Pengfei Zhou, Rui Tan, Mo Li, and Guobin Shen. "LIMU-BERT." GetMobile: Mobile Computing and Communications 26, no. 3 (2022): 39–42. http://dx.doi.org/10.1145/3568113.3568124.
Full textAbstract:
Deep learning greatly empowers Inertial Measurement Unit (IMU) sensors for a wide range of sensing applications. Most existing works require substantial amounts of wellcurated labeled data to train IMU-based sensing models, which incurs high annotation and training costs. Compared with labeled data, unlabeled IMU data are abundant and easily accessible. This article presents a novel representation learning model that can make use of unlabeled IMU data and extract generalized rather than task-specific features. With the representations learned via our model, task-specific models trained with limited labeled samples can achieve superior performances in typical IMU sensing applications, such as Human Activity Recognition (HAR).
7
Duffy, Dennis. "Knighting Bert." Ontario History 104, no. 2 (2012): 28. http://dx.doi.org/10.7202/1065436ar.
Full text
8
Fournier, Véronique. "Anne Bert." Cerveau & Psycho N° 93, no. 10 (2017): 62–64. http://dx.doi.org/10.3917/cerpsy.093.0062.
Full text
9
Cazalaà, Jean-Bernard. "Paul Bert." Anesthesiology 117, no. 6 (2012): 1244. http://dx.doi.org/10.1097/aln.0b013e31827ce191.
Full text
10
Skitol, Robert. "Bert Foer." Antitrust Bulletin 60, no. 2 (2015): 88–90. http://dx.doi.org/10.1177/0003603x15584607.
Full text
11
Mees, Barend M. E. "Bert Eikelboom." European Journal of Vascular and Endovascular Surgery 57, no. 3 (2019): 464. http://dx.doi.org/10.1016/j.ejvs.2018.12.001.
Full text
12
Erbrink, Jacobien. "Bert Keizer." Tijdschrift voor Ouderengeneeskunde 35, no. 3 (2010): 100–101. http://dx.doi.org/10.1007/bf03089855.
Full text
13
Torri Saldanha Coelho, Renata. "BERT HELLINGER." Alamedas 10, no. 2 (2023): 110–20. http://dx.doi.org/10.48075/ra.v10i2.30319.
Full textAbstract:
Este estudo buscou demonstrar a história de vida de Bert Hellinger. Bert Hellinger nasceu em 1925 e faleceu em 2019, sendo mundialmente reconhecido como o criador das constelações familiares. Atualmente, o estudo das constelações familiares ingressa no âmbito acadêmico, pois já é uma prática integrativa reconhecida pelo Sistema Único de Saúde e também uma forma de resolução de conflitos dentro do Poder Judiciário. Contudo, pelo que o próprio paradigma sistêmico propõe, é impossível abordar um tema isoladamente, sem compreender o contexto em que ele está inserido. Assim, o presente trabalho pontua a história de vida de Bert Hellinger, demonstrando suas constatações sobre o campo sistêmico. Dessa forma, é possível compreender a importância do conhecimento da história de vida de Bert Hellinger para que ele fundamentasse o conhecimento sistêmico, da mesma forma que é possível aplicar tal raciocínio a qualquer pesquisador, pois a história de vida de uma pessoa é indissociável ao seu objeto de pesquisa.
14
Kim, Kyungmo, Seongkeun Park, Jeongwon Min, et al. "Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation." JMIR Medical Informatics 12 (October 30, 2024): e52897-e52897. http://dx.doi.org/10.2196/52897.
Full textAbstract:
Abstract Background The bidirectional encoder representations from transformers (BERT) model has attracted considerable attention in clinical applications, such as patient classification and disease prediction. However, current studies have typically progressed to application development without a thorough assessment of the model’s comprehension of clinical context. Furthermore, limited comparative studies have been conducted on BERT models using medical documents from non–English-speaking countries. Therefore, the applicability of BERT models trained on English clinical notes to non-English contexts is yet to be confirmed. To address these gaps in literature, this study focused on identifying the most effective BERT model for non-English clinical notes. Objective In this study, we evaluated the contextual understanding abilities of various BERT models applied to mixed Korean and English clinical notes. The objective of this study was to identify the BERT model that excels in understanding the context of such documents. Methods Using data from 164,460 patients in a South Korean tertiary hospital, we pretrained BERT-base, BERT for Biomedical Text Mining (BioBERT), Korean BERT (KoBERT), and Multilingual BERT (M-BERT) to improve their contextual comprehension capabilities and subsequently compared their performances in 7 fine-tuning tasks. Results The model performance varied based on the task and token usage. First, BERT-base and BioBERT excelled in tasks using classification ([CLS]) token embeddings, such as document classification. BioBERT achieved the highest F1-score of 89.32. Both BERT-base and BioBERT demonstrated their effectiveness in document pattern recognition, even with limited Korean tokens in the dictionary. Second, M-BERT exhibited a superior performance in reading comprehension tasks, achieving an F1-score of 93.77. Better results were obtained when fewer words were replaced with unknown ([UNK]) tokens. Third, M-BERT excelled in the knowledge inference task in which correct disease names were inferred from 63 candidate disease names in a document with disease names replaced with [MASK] tokens. M-BERT achieved the highest hit@10 score of 95.41. Conclusions This study highlighted the effectiveness of various BERT models in a multilingual clinical domain. The findings can be used as a reference in clinical and language-based applications.
15
Su, Jing, Qingyun Dai, Frank Guerin, and Mian Zhou. "BERT-hLSTMs: BERT and hierarchical LSTMs for visual storytelling." Computer Speech & Language 67 (May 2021): 101169. http://dx.doi.org/10.1016/j.csl.2020.101169.
Full text
16
Kaur, Kamaljit, and Parminder Kaur. "BERT-CNN: Improving BERT for Requirements Classification using CNN." Procedia Computer Science 218 (2023): 2604–11. http://dx.doi.org/10.1016/j.procs.2023.01.234.
Full text
17
Prakash, PKS, Srinivas Chilukuri, Nikhil Ranade, and Shankar Viswanathan. "RareBERT: Transformer Architecture for Rare Disease Patient Identification using Administrative Claims." Proceedings of the AAAI Conference on Artificial Intelligence 35, no. 1 (2021): 453–60. http://dx.doi.org/10.1609/aaai.v35i1.16122.
Full textAbstract:
A rare disease is any disease that affects a very small percentage (1 in 1,500) of population. It is estimated that there are nearly 7,000 rare disease affecting 30 million patients in the U. S. alone. Most of the patients suffering from rare diseases experience multiple misdiagnoses and may never be diagnosed correctly. This is largely driven by the low prevalence of the disease that results in a lack of awareness among healthcare providers. There have been efforts from machine learning researchers to develop predictive models to help diagnose patients using healthcare datasets such as electronic health records and administrative claims. Most recently, transformer models have been applied to predict diseases BEHRT, G-BERT and Med-BERT. However, these have been developed specifically for electronic health records (EHR) and have not been designed to address rare disease challenges such as class imbalance, partial longitudinal data capture, and noisy labels. As a result, they deliver poor performance in predicting rare diseases compared with baselines. Besides, EHR datasets are generally confined to the hospital systems using them and do not capture a wider sample of patients thus limiting the availability of sufficient rare dis-ease patients in the dataset. To address these challenges, we introduced an extension of the BERT model tailored for rare disease diagnosis called RareBERT which has been trained on administrative claims datasets. RareBERT extends Med-BERT by including context embedding and temporal reference embedding. Moreover, we introduced a novel adaptive loss function to handle the class imbal-ance. In this paper, we show our experiments on diagnosing X-Linked Hypophosphatemia (XLH), a genetic rare disease. While RareBERT performs significantly better than the baseline models (79.9% AUPRC versus 30% AUPRC for Med-BERT), owing to the transformer architecture, it also shows its robustness in partial longitudinal data capture caused by poor capture of claims with a drop in performance of only 1.35% AUPRC, compared with 12% for Med-BERT and 33.0% for LSTM and 67.4% for boosting trees based baseline.
18
Liu, Weijie, Peng Zhou, Zhe Zhao, et al. "K-BERT: Enabling Language Representation with Knowledge Graph." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 03 (2020): 2901–8. http://dx.doi.org/10.1609/aaai.v34i03.5681.
Full textAbstract:
Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by being equipped with a KG without pre-training by itself because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.
19
Yu, Geyang. "An analysis of BERT-based model for Berkshire stock performance prediction using Warren Buffet's letters." Applied and Computational Engineering 52, no. 1 (2024): 55–61. http://dx.doi.org/10.54254/2755-2721/52/20241232.
Full textAbstract:
The objective of this study is to discover and validate eective Bidirectional Encoder Representations from Transformers (BERT)-based models for stock market prediction of Berkshire Hathaway. The stock market is full of uncertainty and dynamism and its prediction has always been a critical challenge in the nancial domain. Therefore, accurate predictions of market trends are important for making investment decisions and risk management. The primary approach involves sentiment analysis of reviews on market performance. This work selects Warren Buetts annual letters to investors and the year-by-year stock market performance of the Berkshire Hathway as the dataset. This work leverages three BERT-based models which are BERT-Gated Recurrent Units (BERT-GRU) model, BERT-Long short-term memory (BERT-LSTM) model, and BERT-Multi-Head Attention model to analyse the Buetts annual letters and predict the Berkshire Hathways stock price changes. After conducting experiments, it could be concluded that all three models have a certain degree of predictive capability, with the BERT-Multi-Head Attention model demonstrating the best predictive performance.
20
Dharmendra, Mangal Hemant Makwana. "Performance analysis of different BERT implementation for event burst detection from social media text." Indonesian Journal of Electrical Engineering and Computer Science 38, no. 1 (2025): 439–46. https://doi.org/10.11591/ijeecs.v38.i1.pp439-446.
Full textAbstract:
The language models play very important role in natural language processing (NLP) tasks. To understand natural languages, the learning models are required to be trained on large corpus. This requires a lot of time and computing resources. The detection of information like events, and locations from text is an important NLP task. As events detection is to be done in real-time so that immediate actions can be taken, hence we need efficient decision-making models. The pertained models like bi-directional encoders representation from transformers (BERT) gaining popularity to solve NLP problems. As BERT based models are pre-trained on large language corpus it requires very less time to adapt for domain specific NLP task. Different implementations of BERT have been proposed to enhance efficiency and applicability of the base model. The selection of right implementation is essential for overall performance of NLP based system. This work presents the comparative insights of five widely used BERT implementations named as BERT-base, BERT-large, Distill BERT, Robust BERT approach (RoBERTa-base) and RoBERT-large for event detection from the text extracted from social media streams. The results show that Distill-BERT model outperforms on basis of performance metric like precision, recall, and F1-score while the fastest to train also.
21
Said, Fadillah, and Lindung Parningotan Manik. "Aspect-Based Sentiment Analysis on Indonesian Presidential Election Using Deep Learning." Paradigma - Jurnal Komputer dan Informatika 24, no. 2 (2022): 160–67. http://dx.doi.org/10.31294/paradigma.v24i2.1415.
Full textAbstract:
Pemilihan presiden tahun 2019 merupakan pemilihan presiden yang menjadi perbincangan hangat selama beberapa waktu bahkan orang membicarakan topik ini sejak tahun 2018 di internet. Dalam memprediksi pemenang pemilihan presiden penelitian sebelumnya telah melakukan penelitian terhadap dataset Analisis sentimen berbasis aspek (ABSA) pemilihan presiden tahun 2019 menggunakan algoritma pembelajaran mesin seperti Support Vector Machine (SVM), Naive Bayes (NB), dan K-Nearest Neighbors (KNN) dan menghasilkan akurasi yang cukup baik. Penelitian ini mengusulkan metode deep learning dengan menggunakan model BERT (Bidirectional Encoder Representation form Transformers) dan RoBERTa (A Robustly Optimized BERT Pretraining Approach). Hasil penelitian ini menunjukkan bahwa model BERT indobenchmark dan RoBERTa base-indonesian single label classification pada fitur target dengan preprocessing menghasilkan akurasi yang terbaik yaitu sebesar 98.02%. Model BERT indolem dan indobenchmark single label classification pada fitur target tanpa preprocessing menghasilkan akurasi yang terbaik yaitu sebesar 98.02%. Model BERT indobenchmark single label classification pada fitur aspek dengan preprocessing menghasilkan akurasi yang terbaik yaitu sebesar 74.26%. Model BERT indolem single label classification pada fitur aspek tanpa preprocessing menghasilkan akurasi yang terbaik yaitu sebesar 74.26%. Model BERT indolem single label classification pada fitur sentiment dengan preprocessing menghasilkan akurasi yang terbaik yaitu sebesar 93.07%. Model BERT indolem single label classification pada fitur sentiment tanpa preprocessing menghasilkan akurasi yang terbaik yaitu sebesar 94.06%. Model BERT indobenchmark multi label classification dengan preprocessing menghasilkan akurasi yang terbaik yaitu sebesar 98.66%. Model BERT indobenchmark multi label classification tanpa preprocessing menghasilkan akurasi yang terbaik yaitu sebesar 98.66%.
22
Balderas, Luis, Miguel Lastra, and José M. Benítez. "A Green AI Methodology Based on Persistent Homology for Compressing BERT." Applied Sciences 15, no. 1 (2025): 390. https://doi.org/10.3390/app15010390.
Full textAbstract:
Large Language Models (LLMs) like BERT have gained significant prominence due to their remarkable performance in various natural language processing tasks. However, they come with substantial computational and memory costs. Additionally, they are essentially black-box models, being challenging to explain and interpret. In this article, Persistent BERT Compression and Explainability (PBCE) is proposed, a Green AI methodology to prune BERT models using persistent homology, aiming to measure the importance of each neuron by studying the topological characteristics of their outputs. As a result, PBCE can compress BERT significantly by reducing the number of parameters (47% of the original parameters for BERT Base, 42% for BERT Large). The proposed methodology has been evaluated on the standard GLUE Benchmark, comparing the results with state-of-the-art techniques achieving outstanding results. Consequently, PBCE can simplify the BERT model by providing explainability to its neurons and reducing the model’s size, making it more suitable for deployment on resource-constrained devices.
23
Arefeva, Veronika, and Roman Egger. "When BERT Started Traveling: TourBERT—A Natural Language Processing Model for the Travel Industry." Digital 2, no. 4 (2022): 546–59. http://dx.doi.org/10.3390/digital2040030.
Full textAbstract:
In recent years, Natural Language Processing (NLP) has become increasingly important for extracting new insights from unstructured text data, and pre-trained language models now have the ability to perform state-of-the-art tasks like topic modeling, text classification, or sentiment analysis. Currently, BERT is the most widespread and widely used model, but it has been shown that a potential to optimize BERT can be applied to domain-specific contexts. While a number of BERT models that improve downstream tasks’ performance for other domains already exist, an optimized BERT model for tourism has yet to be revealed. This study thus aimed to develop and evaluate TourBERT, a pre-trained BERT model for the tourism industry. It was trained from scratch and outperforms BERT-Base in all tourism-specific evaluations. Therefore, this study makes an essential contribution to the growing importance of NLP in tourism by providing an open-source BERT model adapted to tourism requirements and particularities.
24
Li, Yanjie, and He Mao. "Evaluation and Construction of College Students’ Growth and Development Index System Based on Data Association Mining and Deep Learning Model." Security and Communication Networks 2021 (December 31, 2021): 1–8. http://dx.doi.org/10.1155/2021/7415129.
Full textAbstract:
The rise of big data in the field of education provides an opportunity to solve college students’ growth and development. The establishment of a personalized student management mode based on big data in universities will promote the change of personalized student management from the empirical mode to the scientific mode, from passive response to active warning, from reliance on point data to holistic data, and thus improve the efficiency and quality of personalized student management. In this paper, using the latest ideas and techniques in deep learning such as self-supervised learning and multitask learning, we propose an open-source educational big data pretrained language model F-BERT based on the BERT model architecture. Based on the BERT architecture, F-BERT can effectively and automatically extract knowledge from educational big data and memorize it in the model without modifying the model structure specific to educational big data tasks so that it can be directly applied to various educational big data domain tasks downstream. The experiment demonstrates that Vanilla F-BERT outperformed the two Vanilla BERT-based models, Vanilla BERT and BERT tasks, by 0.0.6 and 0.03 percent, respectively, in terms of accuracy.
25
Sanchan, Nattapong. "Intent Mining of Thai Phone Call Text Using a Stacking Ensemble Classifier with GPT-3 Embeddings." ECTI Transactions on Computer and Information Technology (ECTI-CIT) 19, no. 1 (2025): 135–45. https://doi.org/10.37936/ecti-cit.2025191.258239.
Full textAbstract:
Intent mining has recently attracted Natural Language Processing (NLP) research communities. Despite the extensive research on English and other widely spoken languages, intent mining in Thai remains unexplored. This paper proposes an extended framework for mining intentions in Thai phone call text. It utilized a stacking ensemble method with GPT-3 embeddings, constructed by systematically determining based and meta-classifiers using Q-statistic and F1 scores. Overall, the based classifiers consisting of Support Vector Classier (SVC), k-nearest Neighbors (KNN), and Random Forest (RF) were derived with a meta-classier, Logistic Regression (LR). We compared the mining results, derived through the proposed Stacking Ensemble Classier (SEC), to 1) the individual base classifiers and 2) the three BERT baselines: BERT Multilingual Uncased, and BERT-th, and BERT Based EN-TH Cased. The results revealed that SEC could outperform SVC, KNN, RF, BERT Multilingual Uncased, and BERT-th, except BERT Based EN-TH Cased. However, a statistical analysis conducted using Friedman and Holm's post hoc tests reported no statistically significant difference between SEC and BERT Based EN-TH Cased, inferring that the two classifiers perform similarly practically.
26
Hoffmann, Birgitt. "Bert G. Fragner." Iranian Studies 55, no. 2 (2022): 599–601. http://dx.doi.org/10.1017/irn.2022.9.
Full text
27
Brown, Karl. "The BERT robot." ACM SIGFORTH Newsletter 3, no. 3 (1991): 15–18. http://dx.doi.org/10.1145/126517.126519.
Full text
28
Feshiach, Herman, and Kosta Tsipis. "Jerome Bert Wiesner." Physics Today 48, no. 4 (1995): 104–6. http://dx.doi.org/10.1063/1.2807995.
Full text
29
Heislbetz, Hans Peter. "�bert-uniforme Gruppen." Archiv der Mathematik 61, no. 4 (1993): 329–39. http://dx.doi.org/10.1007/bf01201448.
Full text
30
Harry Collins, F. B. A. "Remembering Bert Dreyfus." AI & SOCIETY 34, no. 2 (2018): 373–76. http://dx.doi.org/10.1007/s00146-018-0796-x.
Full text
31
Ettinger, M. G. "Abraham Bert Baker." Neurology 38, no. 4 (1988): 513. http://dx.doi.org/10.1212/wnl.38.4.513.
Full text
32
Grimm, Erk. "Bert Papenfuß: Tiské." GDR Bulletin 25, no. 1 (1998): 83–85. http://dx.doi.org/10.4148/gdrb.v25i0.1265.
Full text
33
Mangal, Dharmendra, and Hemant Makwana. "Performance analysis of different BERT implementation for event burst detection from social media text." Indonesian Journal of Electrical Engineering and Computer Science 38, no. 1 (2025): 439. https://doi.org/10.11591/ijeecs.v38.i1.pp439-446.
Full textAbstract:
<p>The language models play very important role in natural language processing (NLP) tasks. To understand natural languages, the learning models are required to be trained on large corpus. This requires a lot of time and computing resources. The detection of information like events, and locations from text is an important NLP task. As events detection is to be done in real-time so that immediate actions can be taken, hence we need efficient decision-making models. The pertained models like bi-directional encoders representation from transformers (BERT) gaining popularity to solve NLP problems. As BERT based models are pre-trained on large language corpus it requires very less time to adapt for domain specific NLP task. Different implementations of BERT have been proposed to enhance efficiency and applicability of the base model. The selection of right implementation is essential for overall performance of NLP based system. This work presents the comparative insights of five widely used BERT implementations named as BERT-base, BERT-large, Distill BERT, Robust BERT approach (RoBERTa-base) and RoBERT-large for event detection from the text extracted from social media streams. The results show that Distill-BERT model outperforms on basis of performance metric like precision, recall, and F1-score while the fastest to train also.</p>
34
Shen, Sheng, Zhen Dong, Jiayu Ye, et al. "Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 05 (2020): 8815–21. http://dx.doi.org/10.1609/aaai.v34i05.6409.
Full textAbstract:
Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use Hessian-based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most 2.3% performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to 13× compression of the model parameters, and up to 4× compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.
35
Jiang, Liangzhen, Jici Jiang, Xiao Wang, et al. "IUP-BERT: Identification of Umami Peptides Based on BERT Features." Foods 11, no. 22 (2022): 3742. http://dx.doi.org/10.3390/foods11223742.
Full textAbstract:
Umami is an important widely-used taste component of food seasoning. Umami peptides are specific structural peptides endowing foods with a favorable umami taste. Laboratory approaches used to identify umami peptides are time-consuming and labor-intensive, which are not feasible for rapid screening. Here, we developed a novel peptide sequence-based umami peptide predictor, namely iUP-BERT, which was based on the deep learning pretrained neural network feature extraction method. After optimization, a single deep representation learning feature encoding method (BERT: bidirectional encoder representations from transformer) in conjugation with the synthetic minority over-sampling technique (SMOTE) and support vector machine (SVM) methods was adopted for model creation to generate predicted probabilistic scores of potential umami peptides. Further extensive empirical experiments on cross-validation and an independent test showed that iUP-BERT outperformed the existing methods with improvements, highlighting its effectiveness and robustness. Finally, an open-access iUP-BERT web server was built. To our knowledge, this is the first efficient sequence-based umami predictor created based on a single deep-learning pretrained neural network feature extraction method. By predicting umami peptides, iUP-BERT can help in further research to improve the palatability of dietary supplements in the future.
36
Li, Fenfang, Zhengzhang Zhao, Li Wang, and Han Deng. "Tibetan Sentence Boundaries Automatic Disambiguation Based on Bidirectional Encoder Representations from Transformers on Byte Pair Encoding Word Cutting Method." Applied Sciences 14, no. 7 (2024): 2989. http://dx.doi.org/10.3390/app14072989.
Full textAbstract:
Sentence Boundary Disambiguation (SBD) is crucial for building datasets for tasks such as machine translation, syntactic analysis, and semantic analysis. Currently, most automatic sentence segmentation in Tibetan adopts the methods of rule-based and statistical learning, as well as the combination of the two, which have high requirements on the corpus and the linguistic foundation of the researchers and are more costly to annotate manually. In this study, we explore Tibetan SBD using deep learning technology. Initially, we analyze Tibetan characteristics and various subword techniques, selecting Byte Pair Encoding (BPE) and Sentencepiece (SP) for text segmentation and training the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model. Secondly, we studied the Tibetan SBD based on different BERT pre-trained language models, which mainly learns the ambiguity of the shad (“།”) in different positions in modern Tibetan texts and determines through the model whether the shad (“།”) in the texts has the function of segmenting sentences. Meanwhile, this study introduces four models, BERT-CNN, BERT-RNN, BERT-RCNN, and BERT-DPCNN, based on the BERT model for performance comparison. Finally, to verify the performance of the pre-trained language models on the SBD task, this study conducts SBD experiments on both the publicly available Tibetan pre-trained language model TiBERT and the multilingual pre-trained language model (Multi-BERT). The experimental results show that the F1 score of the BERT (BPE) model trained in this study reaches 95.32% on 465,669 Tibetan sentences, nearly five percentage points higher than BERT (SP) and Multi-BERT. The SBD method based on pre-trained language models in this study lays the foundation for establishing datasets for the later tasks of Tibetan pre-training, summary extraction, and machine translation.
37
Владимир Александрович, Минаев,, and Симонов, Александр Валерьевич. "COMPARISON OF BERT TRANSFORMER MODELS IN IDENTIFYING DESTRUCTIVE CONTENT IN SOCIAL MEDIA." ИНФОРМАЦИЯ И БЕЗОПАСНОСТЬ, no. 3(-) (October 24, 2022): 341–48. http://dx.doi.org/10.36622/vstu.2022.25.3.003.
Full textAbstract:
Цель статьи состоит в определении наиболее эффективной модели из семейства BERT по выявлению деструктивного контента в социальных медиа. Произведено сравнение пяти наиболее известных моделей BERT по выявлению деструктивного контента. Для этого осуществлено создание текстового корпуса из материалов социальных медиа (СМ), дополненного запрещённым к распространению в Российской Федерации контентом нацистского характера из Федерального списка экстремистских материалов. Представлена структура классификатора текстовых данных, основанного на глубокой искусственной нейронной сети BERT, и описана его работа на каждом этапе. Проведен поиск наиболее эффективного метода предварительной обработки текстов. Оценена эффективность работы различных голов классификаторов, основанных на трансформере BERT. Оценено влияние дообучения модели BERT и доказана эффективность его применения с расчетом перплексии. Представлены сравнительные таблицы работы классификаторов на каждом этапе исследования. Найдена наиболее эффективная архитектура классификатора на основе трансформера BERT, выполняющего задачу выявления деструктивного контента с точностью 96,99%. The purpose of the article is to determine the most effective model from the BERT family for identifying destructive content in social media. A comparison of the five most well-known BERT models for identifying destructive content was made. For this purpose, a text corpus was created from social media materials (CM), supplemented with Nazi content prohibited for distribution in the Russian Federation from the Federal List of Extremist Materials. The structure of the text data classifier based on the deep artificial neural network BERT is presented and its operation at each stage is described. The search for the most effective method of preprocessing texts was carried out. The efficiency of various heads of classifiers based on the BERT transformer is evaluated. The influence of BERT model retraining is estimated and the effectiveness of its application with the calculation of perplexy is proved. Comparative tables of classifiers' work at each stage of the study are presented. The most effective architecture of the classifier based on the BERT transformer has been found, which performs the task of identifying destructive content with an accuracy of 96.99%.
38
Gupta, Rajesh. "Bidirectional encoders to state-of-the-art: a review of BERT and its transformative impact on natural language processing." Информатика. Экономика. Управление - Informatics. Economics. Management 3, no. 1 (2024): 0311–20. http://dx.doi.org/10.47813/2782-5280-2024-3-1-0311-0320.
Full textAbstract:
First developed in 2018 by Google researchers, Bidirectional Encoder Representations from Transformers (BERT) represents a breakthrough in natural language processing (NLP). BERT achieved state-of-the-art results across a range of NLP tasks while using a single transformer-based neural network architecture. This work reviews BERT's technical approach, performance when published, and significant research impact since release. We provide background on BERT's foundations like transformer encoders and transfer learning from universal language models. Core technical innovations include deeply bidirectional conditioning and a masked language modeling objective during BERT's unsupervised pretraining phase. For evaluation, BERT was fine-tuned and tested on eleven NLP tasks ranging from question answering to sentiment analysis via the GLUE benchmark, achieving new state-of-the-art results. Additionally, this work analyzes BERT's immense research influence as an accessible technique surpassing specialized models. BERT catalyzed adoption of pretraining and transfer learning for NLP. Quantitatively, over 10,000 papers have extended BERT and it is integrated widely across industry applications. Future directions based on BERT scale towards billions of parameters and multilingual representations. In summary, this work reviews the method, performance, impact and future outlook for BERT as a foundational NLP technique. We provide background on BERT's foundations like transformer encoders and transfer learning from universal language models. Core technical innovations include deeply bidirectional conditioning and a masked language modeling objective during BERT's unsupervised pretraining phase. For evaluation, BERT was fine-tuned and tested on eleven NLP tasks ranging from question answering to sentiment analysis via the GLUE benchmark, achieving new state-of-the-art results. Additionally, this work analyzes BERT's immense research influence as an accessible technique surpassing specialized models. BERT catalyzed adoption of pretraining and transfer learning for NLP. Quantitatively, over 10,000 papers have extended BERT and it is integrated widely across industry applications. Future directions based on BERT scale towards billions of parameters and multilingual representations. In summary, this work reviews the method, performance, impact and future outlook for BERT as a foundational NLP technique.
39
Qiu, Zhaopeng, Xian Wu, Jingyue Gao, and Wei Fan. "U-BERT: Pre-training User Representations for Improved Recommendation." Proceedings of the AAAI Conference on Artificial Intelligence 35, no. 5 (2021): 4320–27. http://dx.doi.org/10.1609/aaai.v35i5.16557.
Full textAbstract:
Learning user representation is a critical task for recommendation systems as it can encode user preference for personalized services. User representation is generally learned from behavior data, such as clicking interactions and review comments. However, for less popular domains, the behavior data is insufficient to learn precise user representations. To deal with this problem, a natural thought is to leverage content-rich domains to complement user representations. Inspired by the recent success of BERT in NLP, we propose a novel pre-training and fine-tuning based approach U-BERT. Different from typical BERT applications, U-BERT is customized for recommendation and utilizes different frameworks in pre-training and fine-tuning. In pre-training, U-BERT focuses on content-rich domains and introduces a user encoder and a review encoder to model users' behaviors. Two pre-training strategies are proposed to learn the general user representations; In fine-tuning, U-BERT focuses on the target content-insufficient domains. In addition to the user and review encoders inherited from the pre-training stage, U-BERT further introduces an item encoder to model item representations. Besides, a review co-matching layer is proposed to capture more semantic interactions between the reviews of the user and item. Finally, U-BERT combines user representations, item representations and review interaction information to improve recommendation performance. Experiments on six benchmark datasets from different domains demonstrate the state-of-the-art performance of U-BERT.
40
Eshwarappa, Sunil Mugalihalli, and Vinay Shivasubramanyan. "Enhancing sentiment analysis in Kannada texts by feature selection." International Journal of Electrical and Computer Engineering (IJECE) 14, no. 6 (2024): 6572. http://dx.doi.org/10.11591/ijece.v14i6.pp6572-6582.
Full textAbstract:
In recent years, there has been a noticeable surge in research activities focused on sentiment analysis within the Kannada language domain. The existing research highlights a lack of labelled datasets and limited exploration in feature selection for Kannada sentiment analysis, hindering accurate sentiment classification. To address this gap, the study aims to introduce a novel Kannada dataset and develop an effective classifier for improved sentiment analysis in Kannada texts. The study presents a new Kannada dataset from SemEval 2014 Task4 using Google Translate. It then introduces a modified bidirectional encoder representation from transformers BERT for Kannada dataset called as Kannada-BERT (K-BERT). Further, a probability-clustering (PC) approach is presented to extract the topics and its related aspects. Both the K-BERT classifier and PC approach were merged to attain a K-BERT-PC classifier, integrating a modified BERT model and probability clustering approach for achieving better results. Experimental results demonstrate that K-BERT-PC achieves superior performance in polarity and sentiment analysis accuracy, with an impressive accuracy rate of 91%, surpassing existing classifiers. This work contributes by providing a solution to the scarcity of labelled datasets for Kannada sentiment analysis and introduces an effective classifier, K-BERT-PC, for enhanced sentiment analysis outcomes in Kannada texts.
41
Wen, Yu, Yezhang Liang, and Xinhua Zhu. "Sentiment analysis of hotel online reviews using the BERT model and ERNIE model—Data from China." PLOS ONE 18, no. 3 (2023): e0275382. http://dx.doi.org/10.1371/journal.pone.0275382.
Full textAbstract:
The emotion analysis of hotel online reviews is discussed by using the neural network model BERT, which proves that this method can not only help hotel network platforms fully understand customer needs but also help customers find suitable hotels according to their needs and affordability and help hotel recommendations be more intelligent. Therefore, using the pretraining BERT model, a number of emotion analytical experiments were carried out through fine-tuning, and a model with high classification accuracy was obtained by frequently adjusting the parameters during the experiment. The BERT layer was taken as a word vector layer, and the input text sequence was used as the input to the BERT layer for vector transformation. The output vectors of BERT passed through the corresponding neural network and were then classified by the softmax activation function. ERNIE is an enhancement of the BERT layer. Both models can lead to good classification results, but the latter performs better. ERNIE exhibits stronger classification and stability than BERT, which provides a promising research direction for the field of tourism and hotels.
42
Fu, Guanping, and Jianwei Sun. "Chinese text multi-classification based on Sentences Order Prediction improved Bert model." Journal of Physics: Conference Series 2031, no. 1 (2021): 012054. http://dx.doi.org/10.1088/1742-6596/2031/1/012054.
Full textAbstract:
Abstract For the strong noise interference brought by the NSP mechanism (Next Sentences Prediction) in Bert to the model, in order to improve the classification effect of the Bert model when it is used in text classification, an SOP (Sentences Order Prediction) mechanism is used to replace the Bert model of the NSP mechanism-Multi-classification of Chinese news texts. At first, use randomly sorted adjacent sentence pairs for segment embedding. Then use the Transformer structure of the Bert model to encode the Chinese text, and obtain the final CLS vector as the semantic vector of the text. Finally, connect the different semantic vectors to the multi-category Classification. After ablation experiments, the improved SOP-Bert model obtained the highest F1 value of 96.69. The results show that this model is more effective than the original Bert model on text multi-classification problems.
43
Iswarya, M., P. Sai Krishna, K. Naveen, M. Ganesh, and M. Yasin. "Video Transcript Summarization Using Bert." International Journal of Research Publication and Reviews 4, no. 3 (2023): 1837–41. http://dx.doi.org/10.55248/gengpi.2023.4.32991.
Full text
44
Zhang, Min, and Juanle Wang. "Automatic Extraction of Flooding Control Knowledge from Rich Literature Texts Using Deep Learning." Applied Sciences 13, no. 4 (2023): 2115. http://dx.doi.org/10.3390/app13042115.
Full textAbstract:
Flood control is a global problem; increasing number of flooding disasters occur annually induced by global climate change and extreme weather events. Flood studies are important knowledge sources for flood risk reduction and have been recorded in the academic literature. The main objective of this paper was to acquire flood control knowledge from long-tail data of the literature by using deep learning techniques. Screening was conducted to obtain 4742 flood-related academic documents from past two decades. Machine learning was conducted to parse the documents, and 347 sample data points from different years were collected for sentence segmentation (approximately 61,000 sentences) and manual annotation. Traditional machine learning (NB, LR, SVM, and RF) and artificial neural network-based deep learning algorithms (Bert, Bert-CNN, Bert-RNN, and ERNIE) were implemented for model training, and complete sentence-level knowledge extraction was conducted in batches. The results revealed that artificial neural network-based deep learning methods exhibit better performance than traditional machine learning methods in terms of accuracy, but their training time is much longer. Based on comprehensive feature extraction capability and computational efficiency, the performances of deep learning methods were ranked as: ERNIE > Bert-CNN > Bert > Bert-RNN. When using Bert as the benchmark model, several deformation models showed applicable characteristics. Bert, Bert-CNN, and Bert-RNN were good at acquiring global features, local features, and processing variable-length inputs, respectively. ERNIE showed improved masking mechanism and corpus and therefore exhibited better performance. Finally, 124,196 usage method and 8935 quotation method sentences were obtained in batches. The proportions of method sentence in the literature showed increasing trends over the last 20 years. Thus, as literature with more method sentences accumulates, this study lays a foundation for knowledge extraction in the future.
45
Okolo, Omachi, B. Y. Baha, and M. D. Philemon. "Using Causal Graph Model variable selection for BERT models Prediction of Patient Survival in a Clinical Text Discharge Dataset." Journal of Future Artificial Intelligence and Technologies 1, no. 4 (2025): 455–73. https://doi.org/10.62411/faith.3048-3719-61.
Full textAbstract:
Feature selection in most black-box machine learning algorithms, such as BERT, is based on the cor-relations between features and the target variable rather than causal relationships in the dataset. This makes their predictive power and decisions questionable because of their potential bias. This paper presents novel BERT models that learn from causal variables in a clinical discharge dataset. The causal-directed acyclic Graphs (DAG) identify input variables for patients’ survival rate prediction and decisions. The core idea behind our model lies in the ability of the BERT-based model to learn from the causal DAG semi-synthetic dataset, enabling it to model the underlying causal structure accurately in-stead of the generic spurious correlations devoid of causation. The results from Causal DAG Conditional Independence Test (CIT) validation metrics showed that the conceptual assumptions of the causal DAG were supported, the Pearson correlation coefficient ranges between -1 and 1, the p-value was (>0.05), and the confidence interval of 95% and 25% were satisfied. We further mapped the semi-synthetic dataset that evolved from the Causal DAG to three BERT models. Two metrics, pre-diction accuracy, and AUC score, were used to compare the performance of the BERT models. The accuracy of the BERT models showed that the regular BERT has a performance of 96%, while Clinical-BERT performance was 90%, and Clinical-BERT-Discharge-summary was 92%. On the other hand, the AUC score for BERT was 79%, ClinicalBERT was 77%, while ClinicalBERT-discharge summary was 84%. Our experiments on the synthetic dataset for the patient’s survival rate from the causal DAG datasets demonstrate high predictive performance and explainable input variables for human under-standing to justify prediction.
46
Zulkalnain, Mohd Asyraf, A. R. Syafeeza, Wira Hidayat Mohd Saad, and Shahid Rahaman. "Evaluation of Transformer-Based Models for Sentiment Analysis in Bahasa Malaysia." Journal of Telecommunication, Electronic and Computer Engineering (JTEC) 17, no. 1 (2025): 29–33. https://doi.org/10.54554/jtec.2025.17.01.004.
Full textAbstract:
This study investigates the application of advanced Transformer-based models, namely BERT, DistilBERT, BERT-multilingual, ALBERT, and BERT-CNN, for sentiment analysis in Bahasa Malaysia, addressing unique challenges such as mixed-language usage and abbreviated expressions in social media text. Using the Malaya dataset to ensure linguistic diversity and domain coverage, the research incorporates robust preprocessing techniques, including synonym mapping and sentiment-aware tokenization, to enhance feature extraction. Through rigorous evaluation, BERT-CNN exhibits the best accuracy (96.3%), followed by BERT-multilingual (89.84%) and BERT (89.5%). DistilBERT and ALBERT delivered competitive performance (88.96% and 88.76%, respectively) while offering reduced computational requirements, highlighting the trade-offs between performance and efficiency. The study emphasizes optimized strategies for handling challenges in positive sentiment classification and demonstrates the efficacy of transformer architectures in nuanced sentiment detection for low-resource languages. These findings contribute to advancing Natural Language Processing (NLP) for scalable sentiment analysis across domains.
47
Albashayreh, Alaa, Nahid Zeinali, and Stephanie White. "INNOVATING THE DETECTION OF CARE PRIORITIES IN HEART FAILURE USING LARGE LANGUAGE MODELS." Innovation in Aging 8, Supplement_1 (2024): 1339. https://doi.org/10.1093/geroni/igae098.4272.
Full textAbstract:
Abstract Engaging older adults with advanced chronic conditions, such as heart failure, in discussions about their care priorities is crucial for ensuring treatments align with their preferences, especially at the end of life. Despite the abundance of data in electronic health records (EHRs), documentation of care priorities is often inconsistent and underutilized. This study utilizes natural language processing (NLP) to detect and characterize care priorities in the EHRs of older adults with heart failure, aiming to enhance patient-centered care. We retrained Bio-Clinical-BERT, a Bidirectional Encoder Representations from Transformers (BERT) large language model, using EHR data from a Midwestern U.S. hospital to create Care-BERT, a novel model for predicting care priorities in clinical narratives. We developed a gold-standard corpus of 1,068 notes, focusing on comfort measures only and life-sustaining treatments, with the dataset divided into training (80%) and testing (20%) sets. Care-BERT outperformed BERT-base, Bio-Clinical-BERT, and PubMed-BERT, achieving the highest performance in predicting care priorities (internal validation: F1-score = 0.941, AUC = 0.978; external validation with 200 GPT-based synthetic notes: F1-score = 0.876, AUC = 0.966). Applied to 2,218,251 EHR notes for 7,984 older adults with heart failure (mean age = 76.9 years), Care-BERT revealed that 2.8% had comfort measures only and 17.3% had life-sustaining treatments documented. This study highlights the potential of Transformer-based NLP models like Care-BERT to improve the documentation of care priorities in EHRs and enhance patient-centered care. Future research will explore documentation variations across patient groups using NLP-based labels, providing further insights into care preferences.
48
Kannan, Eswariah, and Lakshmi Anusha Kothamasu. "Fine-Tuning BERT Based Approach for Multi-Class Sentiment Analysis on Twitter Emotion Data." Ingénierie des systèmes d information 27, no. 1 (2022): 93–100. http://dx.doi.org/10.18280/isi.270111.
Full textAbstract:
Tweets are difficult to classify due to their simplicity and frequent use of non-standard orthodoxy or slang words. Although several studies have identified highly accurate sentiment data classifications, most have not been tested on Twitter data. Previous research on sentiment interpretation focused on binary or ternary sentiments in monolingual texts. However, emotions emerge in bilingual and multilingual texts. The emotions expressed in today's social media, including microblogs, are different. We use a dataset that combines everyday dialogue, easy and emotional stimulation to carry out the algorithm to create a balanced dataset with five labels: joy, sad, anger, fear, and neutral. This entails the preparation of datasets and conventional machine learning models. We categorized tweets using the Bidirectional Encoder Representations from Transformers (BERT) language model but are pre-trained in plain text instead of tweets using BERT Transfer Learning (TensorFlow Keras). In this paper we use the HuggingFace’s transformers library to fine-tune pretrained BERT model for a classification task which is termed as modified (M-BERT). Our modified (M-BERT) model is an average F1-score of 97.63% in all of our taxonomy, which leaves more space for change, is our modified (M-BERT) model. We show that the dual use of an F1-score as a combination of M-BERT and Machine Learning methods increases classification accuracy by 24.92%. as related to baseline BERT model.
49
Huang, Zhong, Ning An, Juan Liu, and Fuji Ren. "EMSI-BERT: Asymmetrical Entity-Mask Strategy and Symbol-Insert Structure for Drug–Drug Interaction Extraction Based on BERT." Symmetry 15, no. 2 (2023): 398. http://dx.doi.org/10.3390/sym15020398.
Full textAbstract:
Drug-drug interaction (DDI) extraction has seen growing usage of deep models, but their effectiveness has been restrained by limited domain-labeled data, a weak representation of co-occurring entities, and poor adaptation of downstream tasks. This paper proposes a novel EMSI-BERT method for drug–drug interaction extraction based on an asymmetrical Entity-Mask strategy and a Symbol-Insert structure. Firstly, the EMSI-BERT method utilizes the asymmetrical Entity-Mask strategy to address the weak representation of co-occurring entity information using the drug entity dictionary in the pre-training BERT task. Secondly, the EMSI-BERT method incorporates four symbols to distinguish different entity combinations of the same input sequence and utilizes the Symbol-Insert structure to address the week adaptation of downstream tasks in the fine-tuning stage of DDI classification. The experimental results showed that EMSI-BERT for DDI extraction achieved a 0.82 F1-score on DDI-Extraction 2013, and it improved the performances of the multi-classification task of DDI extraction and the two-classification task of DDI detection. Compared with baseline Basic-BERT, the proposed pre-training BERT with the asymmetrical Entity-Mask strategy could obtain better effects in downstream tasks and effectively limit “Other” samples’ effects. The model visualization results illustrated that EMSI-BERT could extract semantic information at different levels and granularities in a continuous space.
50
Freunek, Michael, and André Bodmer. "Transformer-Based Patent Novelty Search by Training Claims to Their Own Description." Applied Economics and Finance 8, no. 5 (2021): 37. http://dx.doi.org/10.11114/aef.v8i5.5182.
Full textAbstract:
In this paper we present a method to concatenate patent claims to their own description. By applying this method, bidirectional encoder representations from transformers (BERT) train suitable descriptions for claims. Such a trained BERT could be able to identify novelty relevant descriptions for patents. In addition, we introduce a new scoring scheme: relevance score or novelty score to interprete the output of BERT. We test the method on patent applications by training BERT on the first claims of patents and corresponding descriptions. The output is processed according to the relevance score and the results compared with the cited X documents in the search reports. The test shows that BERT score some of the cited X documents as highly relevant.