Log in

Relevant bibliographies by topics / ETL Processing / Journal articles

To see the other types of publications on this topic, follow the link: ETL Processing.

Journal articles on the topic 'ETL Processing'

Author: Grafiati

Published: 5 June 2025

Last updated: 16 July 2025

Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles

Select a source type:

Consult the top 50 journal articles for your research on the topic 'ETL Processing.'

Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.

You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.

Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.

1

Seenivasan, Dhamotharan. "Real-Time Data Processing with Streaming ETL." International Journal of Science and Research (IJSR) 12, no. 11 (2023): 2185–92. https://doi.org/10.21275/sr24619000026.

Full text

APA, Harvard, Vancouver, ISO, and other styles

2

Srinivasa Sunil Chippada and Shekhar Agrawal. "Modern ETL/ELT pipeline design for ML workflows." World Journal of Advanced Research and Reviews 26, no. 1 (2025): 351–58. https://doi.org/10.30574/wjarr.2025.26.1.1089.

Full text

Abstract:

machine learning workflows, examining data processing architectures' evolution and current state. The article explores how organizations are transitioning from traditional ETL to contemporary ELT approaches, driven by the increasing complexity of ML applications and exponential growth in data volumes. The article investigates key aspects including metadata-driven frameworks, quality control mechanisms, performance optimization strategies, and pipeline governance. Through analysis of multiple enterprise implementations, the article demonstrates how modern pipeline architectures have transformed

APA, Harvard, Vancouver, ISO, and other styles

3

Simran, Sethi. "Designing ETL Pipelines for Scalable Data Processing." International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences 9, no. 6 (2021): 1–10. https://doi.org/10.5281/zenodo.14945154.

Full text

Abstract:

With the rapid growth of data sources and volumes, organizations require scalable and reliable Extract, Transform, Load (ETL) pipelines to ensure timely and accurate analytics. This paper surveys evolving ETL architectures—from traditional batch-driven processes to modern, service-oriented, and metadata-driven frameworks—highlighting how they address the challenges of handling large data volumes, near-real-time needs, and distributed infrastructures. It discusses how shifting from monolithic ETL scripts to microservices and orchestration-based pipelines (e.g., using Airflow or Kafk

APA, Harvard, Vancouver, ISO, and other styles

4

Chauhan, Satyam. "Zero ETL for Cloud-Native Applications on AWS: A Paradigm Shift in Data Processing and Quality Assurance." International Scientific Journal of Engineering and Management 04, no. 02 (2025): 1–6. https://doi.org/10.55041/isjem02262.

Full text

Abstract:

Abstract—the advent of cloud-native applications has revolutionized the way data is processed, stored, and analyzed. Traditional Extract, Transform, Load (ETL) processes, while effective, often introduce latency, complexity, and scalability challenges. This paper explores the concept of Zero ETL, a paradigm shift in data processing that leverages the inherent capabilities of cloud-native platforms like Amazon Web Services (AWS) to eliminate the need for traditional ETL pipelines. We delve into the architectural principles, benefits, and challenges of Zero ETL, with a focus on its implications

APA, Harvard, Vancouver, ISO, and other styles

5

Nishanth, Reddy Mandala. "ETL Optimization Techniques for Big Data." Journal of Scientific and Engineering Research 6, no. 1 (2019): 291–301. https://doi.org/10.5281/zenodo.14274360.

Full text

Abstract:

Extract, Transform, Load (ETL) processes are crucial in managing and analyzing big data. However, traditional ETL approaches often struggle with the volume, velocity, and variety of big data. This paper explores various optimization techniques for ETL processes in big data environments. We discuss parallel processing, data partitioning, incremental loading, and in-memory processing among other strategies. Our findings indicate that a combination of these techniques can significantly improve ETL performance, reducing processing time by up to 60% in some scenarios. We also present case studies a

APA, Harvard, Vancouver, ISO, and other styles

6

Kartanova, A. Dzh, and T. I. Imanbekov. "OVERVIEW OF OPTIMIZATION METHODS FOR PRODUCTIVITY OF THE ETL PROCESS." Heralds of KSUCTA, №1, 2022, no. 1-2022 (March 14, 2022): 64–70. http://dx.doi.org/10.35803/1694-5298.2022.1.64-70.

Full text

Abstract:

One of the important aspects in management and acceleration of processes, operations in databases and data warehouses is ETL processes, the process of extracting, transforming and loading data. These processes without optimizing, a realization data warehouse project is costly, complex, and time-consuming. This paper provides an overview and research of methods for optimizing the performance of ETL processes; that the most important indicator of ETL system's operation is the time and speed of data processing is shown. The issues of the generalized structure of ETL process flows are considered,

APA, Harvard, Vancouver, ISO, and other styles

7

Kartanova, A. Dzh, and T. I. Imanbekov. "OVERVIEW OF OPTIMIZATION METHODS FOR PRODUCTIVITY OF THE ETL PROCES." Heralds of KSUCTA, №4, 2021, no. 4-2021 (December 27, 2021): 556–63. http://dx.doi.org/10.35803/1694-5298.2021.4.556-563.

Full text

Abstract:

One of the important aspects in management and acceleration of processes, operations in databases and data warehouses is ETL processes, the process of extracting, transforming and loading data. These processes without optimizing, a realization data warehouse project is costly, complex, and time-consuming. This paper provides an overview and research of methods for optimizing the performance of ETL processes; that the most important indicator of ETL system's operation is the time and speed of data processing is shown. The issues of the generalized structure of ETL process flows are considered,

APA, Harvard, Vancouver, ISO, and other styles

8

Martins, Pedro, and Maryam Abbasi. "Elastic Performance For ETL+Q Processing." International Journal of Database Management Systems 8, no. 1 (2016): 13–28. http://dx.doi.org/10.5121/ijdms.2016.8102.

Full text

APA, Harvard, Vancouver, ISO, and other styles

9

Nishanth, Reddy Mandala. "ETL Pipelines for Blockchain Data Processing." International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences 7, no. 6 (2019): 1–11. https://doi.org/10.5281/zenodo.14183801.

Full text

Abstract:

The increasing adoption of **blockchain technol- ogy**has generated a vast amount of transaction al data, creating new challenges for data processing. As blockchains produce large, immutable datasets, **ETL (Extract, Transform, Load)** pipelines are essential for transforming raw blockchain data into usable information for analytics and decision-making. This paper discusses the architecture, challenges, and optimization tech- niques for implementing ETL pipelines in blockchain systems. We analyze various strategies to handle high-volume, decentralized, and cryptographically secured blockchain

APA, Harvard, Vancouver, ISO, and other styles

10

Shashank Rudra. "From ETL to ELT: Modernizing pipelines for consumer identity workflows." World Journal of Advanced Engineering Technology and Sciences 15, no. 3 (2025): 455–63. https://doi.org/10.30574/wjaets.2025.15.3.0933.

Full text

Abstract:

This article examines the paradigm shift from Extract, Transform, Load (ETL) to Extract, Load, Transform (ELT) architectures within consumer identity data processing workflows. As organizations increasingly prioritize unified customer views across digital touchpoints, traditional ETL approaches have revealed limitations in handling the velocity, volume, and complexity of modern data streams. The transition to ELT represents more than a reordering of steps; it reflects a fundamental reimagining of data architecture in response to cloud computing capabilities and evolving identity resolution req

APA, Harvard, Vancouver, ISO, and other styles

11

Boyko, N. I., and A. V. Chernenko. "Modern approaches to data storage: comparison of relational and cloud data warehouses using etl and elt methods." Reporter of the Priazovskyi State Technical University. Section: Technical sciences, no. 48 (June 27, 2024): 7–19. http://dx.doi.org/10.31498/2225-6733.48.2024.310669.

Full text

Abstract:

The paper analyses various aspects of the use of relational and cloud data warehouses as well as methods of integrating ETL and ELT data. A comparative analysis of these approaches, their advantages and disadvantages are provided. A central relational data warehouse is proposed that provides a single version of truth (SVOT), which allows standardising and structuring data, avoiding differences and providing the access to the same information for all users of an organisation. It is analysed the methodological approaches to implementing a data warehouse: top-down, bottom-up, and from middle. It

APA, Harvard, Vancouver, ISO, and other styles

12

Chadha, Kawaljeet Singh. "Machine Learning–Augmented ETL Pipelines for Fraud–Resistant Insurance Claims Processing." International journal of data science and machine learning 05, no. 01 (2025): 410–36. https://doi.org/10.55640/ijdsml-05-01-30.

Full text

Abstract:

The insurance industry is also affected by insurance fraud, which incurs massive financial losses and operational inefficiencies. Current fraud detection methods tend to be based on rule-based systems and static Extract, Transform, Load (ETL) pipelines, which are unable to keep up with the pace of rapidly evolving fraud tactics. However, these conventional approaches exhibit high false-positive rates, limited flexibility, and cannot perform real-time analysis, causing delayed detection and increased operational costs. This article describes the integration of machine learning (ML) techniques i

APA, Harvard, Vancouver, ISO, and other styles

13

Veernapu, Kiran. "Oracle ETL tools and ai integration: New data management approach." International Journal of Multidisciplinary Research and Growth Evaluation 1, no. 5 (2020): 120–24. https://doi.org/10.54660/.ijmrge.2020.1.5-120-124.

Full text

Abstract:

Industries like Healthcare produce enormous amounts of data. Collecting, cleaning, and processing the data to make the data available for deep insights is a greater need in today’s competitive world. This process of data integration and data management is called Extract, Transform, Load (ETL). There are several products and tools in the market to accomplish this task. The focus of this paper is on Oracle data management tools, the Oracle ETL tool set. Combining Artificial Intelligence (AI) with ETL tools is changing how data is managed and processed. Oracle’s ETL tools, like Oracle Data Integr

APA, Harvard, Vancouver, ISO, and other styles

14

Farhan, Marwa Salah, Amira Youssef, and Laila Abdelhamid. "A Model for Enhancing Unstructured Big Data Warehouse Execution Time." Big Data and Cognitive Computing 8, no. 2 (2024): 17. http://dx.doi.org/10.3390/bdcc8020017.

Full text

Abstract:

Traditional data warehouses (DWs) have played a key role in business intelligence and decision support systems. However, the rapid growth of the data generated by the current applications requires new data warehousing systems. In big data, it is important to adapt the existing warehouse systems to overcome new issues and limitations. The main drawbacks of traditional Extract–Transform–Load (ETL) are that a huge amount of data cannot be processed over ETL and that the execution time is very high when the data are unstructured. This paper focuses on a new model consisting of four layers: Extract

APA, Harvard, Vancouver, ISO, and other styles

15

Yerra, Srikanth. "Reducing ETL processing time with SSIS optimizations for large-scale data pipelines." International journal of data science and machine learning 05, no. 01 (2025): 61–69. https://doi.org/10.55640/ijdsml-05-01-12.

Full text

Abstract:

Extract, Transform, Load (ETL) processes form the backbone of data manage- ment and consolidation in today’s data-driven enterprises with prevalent large- scale data pipelines. One of the widely used ETL tools is Microsoft SQL Server Integration Services (SSIS), yet its optimization for performance for large-scale data loads remains a challenge. As the volumes of data grow exponentially, inefficient ETL processes create bottlenecks, increased processing time, and ex- haustion of system resources. This work discusses major SSIS optimizations that minimize ETL processing time, allowing for effec

APA, Harvard, Vancouver, ISO, and other styles

16

Cheruku, Saketh Reddy, Om Goel, and Shalu Jain. "A Comparative Study of ETL Tools: DataStage vs. Talend." Journal of Quantum Science and Technology 1, no. 1 (2024): 80–90. http://dx.doi.org/10.36676/jqst.v1.i1.11.

Full text

Abstract:

ETL tools are essential for handling and manipulating massive amounts of data in data integration and processing. IBM DataStage and Talend are two popular ETL technologies. This article compares their features, performance, usability, and efficacy in various data processing settings. This research provides a complete review to help firms choose the best ETL technology for their requirements and operations.IBM Information Server's DataStage is known for its reliability and scalability. For effective processing of massive datasets, it enables complicated data integration techniques and parallel

APA, Harvard, Vancouver, ISO, and other styles

17

Guo, Xiao Li, and Bo Chen. "Research and Design of Data Processing Based on ETL Framework." Advanced Materials Research 1049-1050 (October 2014): 1966–71. http://dx.doi.org/10.4028/www.scientific.net/amr.1049-1050.1966.

Full text

Abstract:

ETL is a key link in the construction of data warehouse. On the base of analyzing the mainstream ETL tool Datastage, the data extraction, transformation and loading, proposes a ETL framework based on data processing, and the realization method and steps are discussed in detail. The framework uses HIVE as a data processing station, improve the operating efficiency of the file; data task according to the E, T and L three parts and hierarchical partitioning, conversion of data users to better grasp the process; development data using the configuration file of the task, the development personnel f

APA, Harvard, Vancouver, ISO, and other styles

18

Jagan Nalla. "Architecting resilient ETL pipelines: Engineering principles for data-intensive environments." World Journal of Advanced Engineering Technology and Sciences 15, no. 3 (2025): 1337–44. https://doi.org/10.30574/wjaets.2025.15.3.0936.

Full text

Abstract:

Extract, Transform, Load (ETL) pipelines serve as the backbone of modern data infrastructure, yet face increasing challenges as organizations contend with exponential data growth and evolving business requirements. Scalable ETL architecture demands deliberate design considerations across technology selection, transformation logic, quality controls, and operational frameworks. The integration of distributed processing technologies like Apache Spark and Apache Flink, combined with cloud-native services, enables significant performance improvements when properly implemented. Data quality gates, a

APA, Harvard, Vancouver, ISO, and other styles

19

Tsai, Hui Chen, Kuo Chung Lin, and Ching Long Yeh. "Case Study of the Medical Decision-Making System ETL Abnormalities Processing Procedure." Advanced Materials Research 433-440 (January 2012): 894–99. http://dx.doi.org/10.4028/www.scientific.net/amr.433-440.894.

Full text

Abstract:

The primary purpose of this research is to resolve the problem of ETL operation failure in execution of ETL (Extraction, Transformation and Loading) by the medical decision-making system due to data content, system factors and defective program design, thereby affect online daily operation of the application system and even customer complaint. This research first research and develop how to record in database, whether successful or not, the number of ETL file conversion program (including tool and self-wrote PL/SQL program) execution process, data status, and execution time; followed by design

APA, Harvard, Vancouver, ISO, and other styles

20

Harish, Goud Kola. "Optimizing ETL Processes for Big Data Applications." International Journal of Engineering and Management Research 14, no. 5 (2024): 99–112. https://doi.org/10.5281/zenodo.14184235.

Full text

Abstract:

Optimizing large-scale data processing has become crucial in the area of data management due to the constantly growing quantity and complexity of data. Big data analysis involves gathering data in a variety of forms from several sources, cleaning it up, customizing it, and then importing it into a data warehouse. Transformation algorithms are needed to extract data in different forms and convert it to the necessary format. Software programs known as Extraction-Transformation-Loading (ETL) solutions are in charge of extracting data from several sources, cleaning it up, personalizing it, and the

APA, Harvard, Vancouver, ISO, and other styles

21

Harish, Goud Kola. "Optimizing ETL Processes for Big Data Applications." International Journal of Engineering and Management Research 14, no. 5 (2024): 148–61. https://doi.org/10.5281/zenodo.14217972.

Full text

Abstract:

Optimizing large-scale data processing has become crucial in the area of data management due to the constantly growing quantity and complexity of data. Big data analysis involves gathering data in a variety of forms from several sources, cleaning it up, customizing it, and then importing it into a data warehouse. Transformation algorithms are needed to extract data in different forms and convert it to the necessary format. Software programs known as Extraction-Transformation-Loading (ETL) solutions are in charge of extracting data from several sources, cleaning it up, personalizing it, and the

APA, Harvard, Vancouver, ISO, and other styles

22

Camilleri, Carl, Joseph G. Vella, and Vitezslav Nezval. "HTAP With Reactive Streaming ETL." Journal of Cases on Information Technology 23, no. 4 (2021): 1–19. http://dx.doi.org/10.4018/jcit.20211001.oa10.

Full text

Abstract:

In database management systems (DBMSs), query workloads can be classified as online transactional processing (OLTP) or online analytical processing (OLAP). These often run within separate DBMSs. In hybrid transactional and analytical processing (HTAP), both workloads may execute within the same DBMS. This article shows that it is possible to run separate OLTP and OLAP DBMSs, and still support timely business decisions from analytical queries running off fresh transactional data. Several setups to manage OLTP and OLAP workloads are analysed. Then, benchmarks on two industry standard DBMSs empir

APA, Harvard, Vancouver, ISO, and other styles

23

Ravi Kiran Pagidi, Raja Kumar Kolli, Chandrasekhara Mokkapati, Om Goel, Dr. Shakeb Khan, and Prof.(Dr.) Arpit Jain. "Enhancing ETL Performance Using Delta Lake in Data Analytics Solutions." Universal Research Reports 9, no. 4 (2022): 473–95. http://dx.doi.org/10.36676/urr.v9.i4.1381.

Full text

Abstract:

In the rapidly evolving field of data analytics, the performance of Extract, Transform, Load (ETL) processes is crucial for effective data management and insight generation. This study explores the integration of Delta Lake within ETL frameworks to enhance performance and reliability. Delta Lake, an open-source storage layer, facilitates ACID transactions, scalable metadata handling, and unifies batch and streaming data processing, addressing common challenges associated with traditional ETL processes. By leveraging Delta Lake’s capabilities, organizations can optimize data ingestion and trans

APA, Harvard, Vancouver, ISO, and other styles

24

Sangeetha Ashok. "Efficient ETL workflows for big data: Handling massive datasets at scale." International Journal of Science and Research Archive 14, no. 2 (2025): 1567–74. https://doi.org/10.30574/ijsra.2025.14.2.0531.

Full text

Abstract:

Extract, Transform, Load (ETL) processes are the backbone of data integration, enabling organizations to manage and analyze vast amounts of information. However, traditional ETL pipelines often struggle with scalability, performance, and efficiency when dealing with massive datasets in the era of big data. This article explores best practices, architectural considerations, and modern optimizations for designing efficient ETL workflows that can handle big data at scale. We discuss distributed processing, cloud-based ETL, automation, and real-time data ingestion to improve performance and reliab

APA, Harvard, Vancouver, ISO, and other styles

25

Xiao, F., C. Li, Z. Wu, and Y. Wu. "NMSTREAM: A SCALABLE EVENT-DRIVEN ETL FRAMEWORK FOR PROCESSING HETEROGENEOUS STREAMING DATA." ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences IV-4 (September 19, 2018): 243–46. http://dx.doi.org/10.5194/isprs-annals-iv-4-243-2018.

Full text

Abstract:

<p><strong>Abstract.</strong> ETL (Extraction-Transform-Load) tools, traditionally developed to operate offline on historical data for feeding Data-warehouses need to be enhanced to deal with continuously increased streaming data and be executed at network level during data streams acquisition. In this paper, a scalable and web-based ETL system called NMStream was presented. NMStream is based on event-driven architecture and designed for integrating distributed and heterogeneous streaming data by integrating the Apache Flume and Cassandra DB system, and the ETL processes were

APA, Harvard, Vancouver, ISO, and other styles

26

Nishanth, Reddy Mandala. "ETL in Edge Computing Architectures." INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH AND CREATIVE TECHNOLOGY 5, no. 3 (2019): 1–8. https://doi.org/10.5281/zenodo.14183884.

Full text

Abstract:

Edge computing has gained immense traction in recent years due to the proliferation of IoT devices and the exponential growth in data generation. Traditional ETL (Extract, Transform, Load) workflows, typically processed in centralized cloud infrastructures, face significant challenges in such dis- tributed environments. In edge computing, ETL operations must be closer to the data source to reduce latency and bandwidth usage. This paper explores the role of ETL in edge computing architectures, focusing on performance optimization, real-time data processing, and scalability. We provide a compreh

APA, Harvard, Vancouver, ISO, and other styles

27

Researcher. "AUTO-SCALING DISTRIBUTED ETL SYSTEMS WITH SERVERLESS PLATFORMS." International Journal of Research In Computer Applications and Information Technology (IJRCAIT) 7, no. 2 (2024): 2119–30. https://doi.org/10.5281/zenodo.14335649.

Full text

Abstract:

This comprehensive article explores the evolution and implementation of auto-scaling distributed ETL systems using serverless platforms. The article examines how serverless architectures revolutionize traditional ETL processes by providing dynamic resource management, improved scalability, and cost-effective operations. The article investigates performance optimization, memory management, concurrency handling, and security considerations in serverless ETL implementations. The article demonstrates how serverless platforms address traditional ETL challenges through detailed analysis of real-worl

APA, Harvard, Vancouver, ISO, and other styles

28

Lakshmi Ayyappan. "Data Warehouse Automation: Streamlining Multi-Cloud ETL Workflows for Real-Time Analytics." International Journal of Scientific Research in Computer Science, Engineering and Information Technology 11, no. 1 (2025): 1534–43. https://doi.org/10.32628/cseit251112166.

Full text

Abstract:

This article examines the transformative impact of automation technologies on data warehouse management and multi-cloud ETL workflows in enterprise environments. The article explores how organizations leverage advanced automation solutions to address the growing complexity of real-time analytics and data processing requirements. Through comprehensive article analysis of implementation strategies, the article demonstrates how modern data warehouse automation incorporates artificial intelligence, machine learning, and sophisticated orchestration mechanisms to enhance operational efficiency and d

APA, Harvard, Vancouver, ISO, and other styles

29

Santosh Vinnakota, Santosh Vinnakota. "Modernizing ETL Workflows: A Metadata-Driven Framework for Scalable Data Management." Journal of Software Engineering and Simulation 7, no. 9 (2021): 54–59. https://doi.org/10.35629/3795-07095459.

Full text

Abstract:

The exponential growth of data has led to the need for robust, high-performance Extract, Transform, Load (ETL) workflows to manage large-scale data processing efficiently. This paper explores the modernization of ETL systems, emphasizing the adoption of metadata-driven frameworks to enhance scalability, performance, and cost-efficiency. By leveraging advanced ETL tools and transitioning to cloud-based infrastructure, organizations can streamline their data workflows and reduce operational complexities. The paper highlights architectural features, benefits, and challenges associated with implem

APA, Harvard, Vancouver, ISO, and other styles

30

Ratna Vineel Prem Kumar Bodapati. "AI-Driven ETL pipelines for real-time business intelligence: A framework for next-generation data processing." World Journal of Advanced Engineering Technology and Sciences 15, no. 2 (2025): 1066–80. https://doi.org/10.30574/wjaets.2025.15.2.0592.

Full text

Abstract:

This article explores the transformative potential of AI-driven ETL (Extract, Transform, Load) pipelines for real-time business intelligence. Traditional ETL processes face significant challenges in today's data-intensive environment, including scalability limitations, processing latency, and maintenance complexities. The article examines how artificial intelligence and machine learning can revolutionize data processing through predictive transformation patterns, automated schema evolution, and intelligent resource allocation. By implementing modular, event-driven architectures with advanced a

APA, Harvard, Vancouver, ISO, and other styles

31

Admin, Admin. "A Extract, Transform, Load sebagai upaya Pembangunan Data Warehouse." Journal of Informatics and Communication Technology (JICT) 1, no. 1 (2019): 11–20. http://dx.doi.org/10.52661/j_ict.v1i1.11.

Full text

Abstract:

Paper ini dibuat untuk memberikan gambaran secara general dalam proses transformasi Ekstract, Transform, dan Load (ETL) sebagai data masukan untuk multidimensional modeling data mart dan data warehouse. Artikel ini dibuat dengan mengimplementasikan database dari Online Transaction Processing (OLTP) kedalam database Online Analytical processing (OLAP). Pada penelitian ini digunakan database classicmodels yang bersifat open source dari Mysql.Metode yang dilakukan dalam penelitian ini adalah, dengan melakukan proses Extract, Transform danLoad (ETL) pada data classic models yang dilakukan dengan c

APA, Harvard, Vancouver, ISO, and other styles

32

Iskandar, Ade Rahmat, Apri Junaidi, and Asep Herman. "Extract, Transform, Load sebagai upaya Pembangunan Data Warehouse." Journal of Informatics and Communication Technology (JICT) 1, no. 1 (2019): 25–35. http://dx.doi.org/10.52661/j_ict.v1i1.21.

Full text

Abstract:

Paper ini dibuat untuk memberikan gambaran secara general dalam proses transformasi Ekstract, Transform, dan Load (ETL) sebagai data masukan untuk multidimensional modeling data mart dan data warehouse. Artikel ini dibuat dengan mengimplementasikan database dari Online Transaction Processing (OLTP) kedalam database Online Analytical processing (OLAP). Pada penelitian ini digunakan database classicmodels yang bersifat open source dari Mysql.Metode yang dilakukan dalam penelitian ini adalah, dengan melakukan proses Extract, Transform danLoad (ETL) pada data classic models yang dilakukan dengan c

APA, Harvard, Vancouver, ISO, and other styles

33

Srikanth Yerra. "Optimizing Supply Chain Efficiency Using AI-Driven Predictive Analytics in Logistics." International Journal of Scientific Research in Computer Science, Engineering and Information Technology 11, no. 2 (2025): 1212–20. https://doi.org/10.32628/cseit25112475.

Full text

Abstract:

In modern supply chain management, shipping de- lays remain a significant issue, impacting customer satisfaction, operational effectiveness, and overall profitability. Traditional data processing methods don’t provide real-time information due to the latency in extracting, transforming, and loading (ETL) data from disparate sources. To alleviate this challenge, automated ETL processing combined with real-time data analytics offers an effective and scalable approach to minimizing shipping delays. This research explores the ways in which automated ETL workflows streamline shipping operations thr

APA, Harvard, Vancouver, ISO, and other styles

34

Jala Aghazada. "ARRANGEMENT AND MODULATION OF ETL PROCESS IN THE STORAGE." Science Review, no. 1(28) (January 31, 2020): 3–8. http://dx.doi.org/10.31435/rsglobal_sr/31012020/6866.

Full text

Abstract:

    Data warehouse (DW) is the basis of systems for operational data analysis (OLAP-Online Analytical Processing). Data extracted from different sources transforms and load in DW. Proper organization of this process, which is called ETL (Extract, Transform, Load) has important significance in creation of DW and analytical data processing. Forms of organization, methods of realization and modeling of ETL processes are considered in this paper.   

APA, Harvard, Vancouver, ISO, and other styles

35

Gayatri Tavva. "Maximizing ETL efficiency: Patterns for high-volume data." International Journal of Science and Research Archive 15, no. 2 (2025): 1063–70. https://doi.org/10.30574/ijsra.2025.15.2.1477.

Full text

Abstract:

The increasing demands of big data environments have placed a renewed emphasis on the efficiency of Extract, Transform, and Load (ETL) processes. Traditional batch-oriented ETL approaches struggle to cope with the scale, velocity, and variety of modern datasets. This review explores emerging patterns and architectures for maximizing ETL efficiency in high-volume data contexts, focusing on serverless frameworks, real-time processing, distributed computation models, and cost optimization strategies. Experimental evaluations demonstrate that serverless and stream-based ETL frameworks achieve supe

APA, Harvard, Vancouver, ISO, and other styles

36

Nishanth, Reddy Mandala. "Security and Compliance in ETL Pipelines." Journal of Scientific and Engineering Research 8, no. 7 (2021): 305–13. https://doi.org/10.5281/zenodo.14274279.

Full text

Abstract:

ETL (Extract, Transform, Load) pipelines are essential for data integration and analytics in modern enterprises. However, the increasing volume and complexity of data have raised concerns about security and regulatory compliance in ETL processes. This paper explores the key security risks associated with ETL pipelines and highlights best practices for ensuring compliance with data protection regulations. By employing techniques such as data encryption, access control, data masking, and audit logging, organizations can mitigate risks and meet the growing demand for secure, compliant data proces

APA, Harvard, Vancouver, ISO, and other styles

37

Hari, Prasad Bomma. "Revolutionizing ETL with AI Powered Automation." International Journal of Leading Research Publication 5, no. 2 (2024): 1–5. https://doi.org/10.5281/zenodo.14769769.

Full text

Abstract:

In today's era of big data and digital transformation, organizations are actively seeking efficient and scalable methods to manage their data pipelines. Traditional, ETL (Extract, Transform, and Load) processes are both demanding and time consuming, requiring manual intervention at various stages. However, cloud computing and AI advancements has heralded a new era of automated ETL pipelines. These advanced systems employ machine learning and deep learning algorithms to automate the entire data processing pipeline, from extraction to feature engineering, reducing the need for manual involvement

APA, Harvard, Vancouver, ISO, and other styles

38

Santosh, Kumar Vududala. "Automating the Future: Enhancing ETL Workflows with RPA and Intelligent Automation." INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH AND CREATIVE TECHNOLOGY 7, no. 2 (2021): 1–11. https://doi.org/10.5281/zenodo.14883126.

Full text

Abstract:

With the more recent improvements in the BI tool and the exploitation of big data, Extract, Transform, and Load (ETL) processes have become critically important. ETL workflows, in particular, are also associated with problems like slowness, poor performance when managing large amounts of data, and the risk of human intervention. Analysing how RPA and IA can revolutionise ETL, this paper aims to discuss the opportunities for change within the field. In light of this fact, RPA and IA enhance the ways organisations work with data by automating routine efforts, increasing data quality, and enablin

APA, Harvard, Vancouver, ISO, and other styles

39

Researcher. "MAXIMIZING FINANCIAL INTELLIGENCE - THE ROLE OF OPTIMIZED ETL IN FINTECH DATA WAREHOUSING." International Journal of Computer Engineering and Technology (IJCET) 15, no. 4 (2024): 464–71. https://doi.org/10.5281/zenodo.13302451.

Full text

Abstract:

Data management is crucial in sustaining competitiveness and challenges regarding regulations in fintech. This management involves the Extract, Transform, Load (ETL) method that entails the extraction of data, the transformation of that data, and the loading of data warehouses. This paper evaluates practices for ETL operations in the financial context of data warehousing, with a focus on the novel technologies and methods. Tackles include data quality, real-time processing, and security; solutions range from machine learning to cloud-based ETL to cross-functional collaboration. This paper also

APA, Harvard, Vancouver, ISO, and other styles

40

Balachandar Paulraj. "SCALABLE ETL PIPELINES FOR TELECOM BILLING SYSTEMS: A COMPARATIVE STUDY." Darpan International Research Analysis 12, no. 3 (2024): 555–73. http://dx.doi.org/10.36676/dira.v12.i3.107.

Full text

Abstract:

This paper aims at comparing the following scalable ETL processes that are used in telecom billing systems. Telecom environment requires the use of ETL pipelines to process huge amounts of data for billing and other data related functions. This analysis covers various types of ETL solutions such as batch, streaming and cloud based ETL techniques. There are several parameters which have been considered while analyzing the issue including scalability, performance, cost and precision. The results indicate that streaming ETL pipelines come as more efficient in real-time data processing, in contras

APA, Harvard, Vancouver, ISO, and other styles

41

Mohammed, Basheer. "A Systematic Taxonomy of ETL Activities for Modern Data Pipelines." International Journal of Advanced Multidisciplinary Research and Studies 5, no. 2 (2025): 106–11. https://doi.org/10.62225/2583049x.2025.5.2.3816.

Full text

Abstract:

ETL (Extract, Transform, Load) is a critical process in data management, enabling organizations to extract data from multiple sources, transform it into a usable format, and load it into target systems for analysis and reporting. However, as data pipelines become more complex with advancements in cloud computing, big data, and real-time processing, a structured framework for understanding ETL activities is essential. This article presents a systematic taxonomy of ETL activities, categorizing them into extraction, transformation, loading, and supporting processes. By providing a well-defined cl

APA, Harvard, Vancouver, ISO, and other styles

42

Wang, XP, and JY Li. "Design of Data Quality Control System Based on ETL." Journal of Physics: Conference Series 2476, no. 1 (2023): 012083. http://dx.doi.org/10.1088/1742-6596/2476/1/012083.

Full text

Abstract:

Abstract Aiming at the problems of incremental data extraction and task scheduling in ETL processing, an optimization strategy based on data quality control is proposed. We discuss the common mechanisms of data incremental extraction and quality control in the ET subsystem of data warehouse, and propose an incremental data extraction method based on key attribute comparison. The key functions of ETL data extraction is utilized, and the database incremental extraction method is adopted. During the process of data processing, the problem of data loss is solved by means of auxiliary tables, and t

APA, Harvard, Vancouver, ISO, and other styles

43

Godleti, Srihari Babu. "Leveraging SonarQube and Snowflake for Advanced ETL Solutions." European Journal of Computer Science and Information Technology 13, no. 49 (2025): 153–62. https://doi.org/10.37745/ejcsit.2013/vol13n49153162.

Full text

Abstract:

This article examines the integration of SonarQube for code quality and Snowflake's cloud platform to address critical challenges in ETL (Extract, Transform, Load) processes. Organizations processing large datasets frequently encounter pipeline failures due to code inefficiencies and resource constraints. SonarQube's static analysis capabilities identify optimization opportunities and memory management issues before deployment, while Snowflake's decoupled architecture enables independent scaling of compute and storage resources. When combined, these technologies create a synergistic effect tha

APA, Harvard, Vancouver, ISO, and other styles

44

Dhaouadi, Asma, Khadija Bousselmi, Mohamed Mohsen Gammoudi, Sébastien Monnet, and Slimane Hammoudi. "Data Warehousing Process Modeling from Classical Approaches to New Trends: Main Features and Comparisons." Data 7, no. 8 (2022): 113. http://dx.doi.org/10.3390/data7080113.

Full text

Abstract:

The extract, transform, and load (ETL) process is at the core of data warehousing architectures. As such, the success of data warehouse (DW) projects is essentially based on the proper modeling of the ETL process. As there is no standard model for the representation and design of this process, several researchers have made efforts to propose modeling methods based on different formalisms, such as unified modeling language (UML), ontology, model-driven architecture (MDA), model-driven development (MDD), and graphical flow, which includes business process model notation (BPMN), colored Petri net

APA, Harvard, Vancouver, ISO, and other styles

45

Researcher. "FULLY AUTOMATED DATA WAREHOUSE FRAMEWORK USING ETL PROCESS FOR DECISION SUPPORT SYSTEM." International Journal of Information Technology (IJIT) 5, no. 2 (2024): 1–12. https://doi.org/10.5281/zenodo.13306158.

Full text

Abstract:

Consider data comparison tools, ETL testing frameworks, and data visualisation tools when choosing tools for your data warehouse architecture, ETL framework, and testing requirements. These tools are all important to consider. An assortment of data comparison tools, including Informatica Data Validation Option, Talend Data Quality, and SQL Server Data Tools, can assist in the identification of inconsistencies or mistakes that exist between the source data sets and the target data sets. To automate ETL test cases, scenarios, and workflows, ETL testing frameworks such as Pytest-ETL, ETL Validato

APA, Harvard, Vancouver, ISO, and other styles

46

Liu, Xiufeng, Nadeem Iftikhar, Huan Huo, and Per Sieverts Nielsen. "Optimizing ETL by a Two-Level Data Staging Method." International Journal of Data Warehousing and Mining 12, no. 3 (2016): 32–50. http://dx.doi.org/10.4018/ijdwm.2016070103.

Full text

Abstract:

In data warehousing, the data from source systems are populated into a central data warehouse (DW) through extraction, transformation and loading (ETL). The standard ETL approach usually uses sequential jobs to process the data with dependencies, such as dimension and fact data. It is a non-trivial task to process the so-called early-/late-arriving data, which arrive out of order. This paper proposes a two-level data staging area method to optimize ETL. The proposed method is an all-in-one solution that supports processing different types of data from operational systems, including early-/late

APA, Harvard, Vancouver, ISO, and other styles

47

Choi, Jun, Young Ki Park, Hee Dong Lee, Seok Il Hong, Woosung Lee, and Jae Woong Jung. "ZrSnO4: A Solution-Processed Robust Electron Transport Layer of Efficient Planar-Heterojunction Perovskite Solar Cells." Nanomaterials 11, no. 11 (2021): 3090. http://dx.doi.org/10.3390/nano11113090.

Full text

Abstract:

A robust electron transport layer (ETL) is an essential component in planar-heterojunction perovskite solar cells (PSCs). Herein, a sol-gel-driven ZrSnO4 thin film is synthesized and its optoelectronic properties are systematically investigated. The optimized processing conditions for sol-gel synthesis produce a ZrSnO4 thin film that exhibits high optical transmittance in the UV-Vis-NIR range, a suitable conduction band maximum, and good electrical conductivity, revealing its potential for application in the ETL of planar-heterojunction PSCs. Consequently, the ZrSnO4 ETL-based devices deliver

APA, Harvard, Vancouver, ISO, and other styles

48

Varun, Garg. "Operational Effectiveness Using Cloud-Based ETL Pipelines on Large-Scale Data Platforms." INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH AND CREATIVE TECHNOLOGY 7, no. 1 (2021): 1–5. https://doi.org/10.5281/zenodo.14945046.

Full text

Abstract:

Processing big amounts of data across several sources in real-time depends critically on cloud-based ETL (Extract, Transform, Load) pipelines. Maintaining operational efficiency, meantime, when handling multi-source data intake creates major difficulties. These involve control of scalability, handling of data variance, low latency assurance, and error recovery automation. This work points out the primary difficulties keeping operating efficiency in cloud-based ETL pipelines and suggests solutions like using real-time processing systems and automation. We show how automatic scaling, resource op

APA, Harvard, Vancouver, ISO, and other styles

49

Ujjawal, Nayak. "Building a Scalable ETL Pipeline with Apache Spark, Airflow, and Snowflake." INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH AND CREATIVE TECHNOLOGY 11, no. 2 (2025): 1–3. https://doi.org/10.5281/zenodo.15125062.

Full text

Abstract:

Extract, Transform, and Load (ETL) pipelines are critical in modern data engineering, enabling efficient data integration and analytics. This paper presents a scalable ETL pipeline leveraging Apache Spark for distributed data processing, Apache Airflow for workflow orchestration, and Snowflake as a cloud-based data warehouse. The proposed architecture ensures fault tolerance, cost efficiency, and high scalability, making it suitable for handling large-scale enterprise data workloads.

APA, Harvard, Vancouver, ISO, and other styles

50

Abharian, Yura. "Conceptual Approaches to Optimizing ETL Processes in Distributed Systems." American Journal of Engineering and Technology 07, no. 04 (2025): 113–18. https://doi.org/10.37547/tajet/volume07issue04-15.

Full text

Abstract:

This article explores conceptual approaches to optimizing ETL processes in distributed systems using a hybrid algorithmic solution based on the integration of Grey Wolf Optimizer (GWO) and Tabu Search (TS) methods. The study analyzes the characteristics of ETL under cloud-based architectures and identifies key challenges, such as high computational complexity, data redundancy, and the difficulty of clustering when handling large volumes of information. The results confirm the hypothesis that the synergy between GWO and TS algorithms leads to more efficient ETL processes, which is especially re

APA, Harvard, Vancouver, ISO, and other styles

We offer discounts on all premium plans for authors whose works are included in thematic literature selections. Contact us to get a unique promo code!