Journal articles: 'Data processing pipeline'

1

Curcoll, R. Firpo, M. Delfino, C. Neissner, I. Reichardt, J. Rico, P. Tallada, and N. Tonello. "The MAGIC data processing pipeline." Journal of Physics: Conference Series 331, no. 3 (December 23, 2011): 032040. http://dx.doi.org/10.1088/1742-6596/331/3/032040.

Full text

APA, Harvard, Vancouver, ISO, and other styles

2

Weilbacher, Peter M., Ralf Palsa, Ole Streicher, Roland Bacon, Tanya Urrutia, Lutz Wisotzki, Simon Conseil, et al. "The data processing pipeline for the MUSE instrument." Astronomy & Astrophysics 641 (September 2020): A28. http://dx.doi.org/10.1051/0004-6361/202037855.

Full text

Abstract:

The processing of raw data from modern astronomical instruments is often carried out nowadays using dedicated software, known as pipelines, largely run in automated operation. In this paper we describe the data reduction pipeline of the Multi Unit Spectroscopic Explorer (MUSE) integral field spectrograph operated at the ESO Paranal Observatory. This spectrograph is a complex machine: it records data of 1152 separate spatial elements on detectors in its 24 integral field units. Efficiently handling such data requires sophisticated software with a high degree of automation and parallelization. We describe the algorithms of all processing steps that operate on calibrations and science data in detail, and explain how the raw science data is transformed into calibrated datacubes. We finally check the quality of selected procedures and output data products, and demonstrate that the pipeline provides datacubes ready for scientific analysis.

APA, Harvard, Vancouver, ISO, and other styles

3

Shen, Hong, and Nobuyoshi Numata. "Instruction Scheduling on a Pipelined Processor for Mechanical Measurements." Key Engineering Materials 381-382 (June 2008): 647–48. http://dx.doi.org/10.4028/www.scientific.net/kem.381-382.647.

Full text

Abstract:

Pipeline processing provides us an effective way to enhance processing speed with low hardware costs. However, pipeline hazards are obstacles to the smooth pipelined execution of instructions. This paper analyzes the pipeline hazards occur in a pipeline processor designed for data processing in mechanical measurements. Instruction scheduling and register renaming are performed to eliminate hazards. The simulation experiments are performed, and the effectiveness is confirmed.

APA, Harvard, Vancouver, ISO, and other styles

4

Leroy, Adam K., Annie Hughes, Daizhong Liu, Jérôme Pety, Erik Rosolowsky, Toshiki Saito, Eva Schinnerer, et al. "PHANGS–ALMA Data Processing and Pipeline." Astrophysical Journal Supplement Series 255, no. 1 (July 1, 2021): 19. http://dx.doi.org/10.3847/1538-4365/abec80.

Full text

APA, Harvard, Vancouver, ISO, and other styles

5

Andrews, Peter, Charles Baltay, Anne Bauer, Nancy Ellman, Jonathan Jerke, Rochelle Lauer, David Rabinowitz, and Julia Silge. "The QUEST Data Processing Software Pipeline." Publications of the Astronomical Society of the Pacific 120, no. 868 (June 2008): 703–14. http://dx.doi.org/10.1086/588828.

Full text

APA, Harvard, Vancouver, ISO, and other styles

6

Zuo, S., J. Li, Y. Li, D. Santanu, A. Stebbins, K. W. Masui, R. Shaw, J. Zhang, F. Wu, and X. Chen. "Data processing pipeline for Tianlai experiment." Astronomy and Computing 34 (January 2021): 100439. http://dx.doi.org/10.1016/j.ascom.2020.100439.

Full text

APA, Harvard, Vancouver, ISO, and other styles

7

Shipman, R. F., S. F. Beaulieu, D. Teyssier, P. Morris, M. Rengel, C. McCoey, K. Edwards, et al. "Data processing pipeline for Herschel HIFI." Astronomy & Astrophysics 608 (December 2017): A49. http://dx.doi.org/10.1051/0004-6361/201731385.

Full text

Abstract:

Context. The HIFI instrument on the Herschel Space Observatory performed over 9100 astronomical observations, almost 900 of which were calibration observations in the course of the nearly four-year Herschel mission. The data from each observation had to be converted from raw telemetry into calibrated products and were included in the Herschel Science Archive. Aims. The HIFI pipeline was designed to provide robust conversion from raw telemetry into calibrated data throughout all phases of the HIFI missions. Pre-launch laboratory testing was supported as were routine mission operations. Methods. A modular software design allowed components to be easily added, removed, amended and/or extended as the understanding of the HIFI data developed during and after mission operations. Results. The HIFI pipeline processed data from all HIFI observing modes within the Herschel automated processing environment as well as within an interactive environment. The same software can be used by the general astronomical community to reprocess any standard HIFI observation. The pipeline also recorded the consistency of processing results and provided automated quality reports. Many pipeline modules were in use since the HIFI pre-launch instrument level testing. Conclusions. Processing in steps facilitated data analysis to discover and address instrument artefacts and uncertainties. The availability of the same pipeline components from pre-launch throughout the mission made for well-understood, tested, and stable processing. A smooth transition from one phase to the next significantly enhanced processing reliability and robustness.

APA, Harvard, Vancouver, ISO, and other styles

8

Brumer, Irène, Dominik F. Bauer, Lothar R. Schad, and Frank G. Zöllner. "Synthetic Arterial Spin Labeling MRI of the Kidneys for Evaluation of Data Processing Pipeline." Diagnostics 12, no. 8 (July 31, 2022): 1854. http://dx.doi.org/10.3390/diagnostics12081854.

Full text

Abstract:

Accurate quantification of perfusion is crucial for diagnosis and monitoring of kidney function. Arterial spin labeling (ASL), a completely non-invasive magnetic resonance imaging technique, is a promising method for this application. However, differences in acquisition (e.g., ASL parameters, readout) and processing (e.g., registration, segmentation) between studies impede the comparison of results. To alleviate challenges arising solely from differences in processing pipelines, synthetic data are of great value. In this work, synthetic renal ASL data were generated using body models from the XCAT phantom and perfusion was added using the general kinetic model. Our in-house developed processing pipeline was then evaluated in terms of registration, quantification, and segmentation using the synthetic data. Registration performance was evaluated qualitatively with line profiles and quantitatively with mean structural similarity index measures (MSSIMs). Perfusion values obtained from the pipeline were compared to the values assumed when generating the synthetic data. Segmentation masks obtained by semi-automated procedure of the processing pipeline were compared to the original XCAT organ masks using the Dice index. Overall, the pipeline evaluation yielded good results. After registration, line profiles were smoother and, on average, MSSIMs increased by 25%. Mean perfusion values for cortex and medulla were close to the assumed perfusion of 250 mL/100 g/min and 50 mL/100 g/min, respectively. Dice indices ranged 0.80–0.93, 0.78–0.89, and 0.64–0.84 for whole kidney, cortex, and medulla, respectively. The generation of synthetic ASL data allows flexible choice of parameters and the generated data are well suited for evaluation of processing pipelines.

APA, Harvard, Vancouver, ISO, and other styles

9

Chen, Rongxin, Zongyue Wang, and Yuling Hong. "Pipelined XPath Query Based on Cost Optimization." Scientific Programming 2021 (May 27, 2021): 1–16. http://dx.doi.org/10.1155/2021/5559941.

Full text

Abstract:

XPath query is the key part of XML data processing, and its performance is usually critical for XML applications. In the process of XPath query, there is inherent seriality between query steps, which makes it difficult to parallelize the query effectively as a whole. On the other hand, although XPath query has the characteristics of data stream processing and is suitable for pipeline processing, the data flow of each query step usually varies a lot, which results in limited performance under multithreading conditions. In this paper, we propose a pipelined XPath query method (PXQ) based on cost optimization. This method uses pipelined query primitives to process query steps based on relation index. During pipeline construction, a cost estimation model based on XML statistics is proposed to estimate the cost of the query primitive and provide guidance for the creation of a pipeline phase through the partition of query primitive sequence. The pipeline construction technique makes full use of available worker threads and optimizes the load balance between pipeline stages. The experimental results show that our method can adapt to the multithreaded environment and stream processing scenarios of XPath query, and its performance is better than the existing typical query methods based on data parallelism.

APA, Harvard, Vancouver, ISO, and other styles

10

Alblehai, Fahad. "A Caching-Based Pipelining Model for Improving the Input/Output Performance of Distributed Data Storage Systems." Journal of Nanoelectronics and Optoelectronics 17, no. 6 (June 1, 2022): 946–57. http://dx.doi.org/10.1166/jno.2022.3269.

Full text

Abstract:

Distributed data storage requires swift input/output (I/O) processing features to prevent pipelines from balancing requests and responses. Unpredictable data streams and fetching intervals congest the data retrieval from distributed systems. To address this issue, in this article, a Coordinated Pipeline Caching Model (CPCM) is proposed. The proposed model distinguishes request and response pipelines for different intervals of time by reallocating them. The reallocation is performed using storage and service demand analysis; in the analysis, edge-assisted federated learning is utilized. The shared pipelining process is fetched from the connected edge devices to prevent input and output congestion. In pipeline allocation and storage management, the current data state and I/O responses are augmented by distributed edges. This prevents pipeline delays and aids storage optimization through replication mitigation. Therefore, the proposed model reduces the congestion rate (57.60%), replication ratio (59.90%), and waiting time (54.95%) and improves the response ratio (5.16%) and processing rate (74.25%) for different requests.

APA, Harvard, Vancouver, ISO, and other styles

11

Melet, O., D. Youssefi, C. L'Helguen, J. Michel, E. Sarrazin, F. Languille, and L. Lebègue. "CO3D MISSION DIGITAL SURFACE MODEL PRODUCTION PIPELINE." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B2-2020 (August 12, 2020): 143–48. http://dx.doi.org/10.5194/isprs-archives-xliii-b2-2020-143-2020.

Full text

Abstract:

Abstract. Earth Observation (EO) remote sensing missions are producing an increasing volume of data due to higher spatial and spectral resolutions, and higher frequency of acquisitions. Thus, in order to prepare the future of image processing pipelines, CNES has carried out Research & Development studies related to the use of Big Data and Cloud technologies for image processing chains made. Since mid-2019, CNES in partnership with Airbus Defense & Space, has started a new High Resolution Optical EO mission dedicated to very high resolution 3D observation called CO3D (“Constellation Optique 3D”). To achieve those objectives, a new image processing pipeline prototype is being developed taking in consideration the lessons learned from the previous studies. The paper will introduce this new image processing pipeline, the processing paradigms used to take advantage of big data technologies and the results of production benchmarks at a large scale. The on-going works to optimize the processing pipeline and Cloud cluster will be also discussed.

APA, Harvard, Vancouver, ISO, and other styles

12

Fu, Shenming, Ian Dell’Antonio, Ranga-Ram Chary, Douglas Clowe, M. C. Cooper, Megan Donahue, August Evrard, et al. "LoVoCCS. I. Survey Introduction, Data Processing Pipeline, and Early Science Results." Astrophysical Journal 933, no. 1 (July 1, 2022): 84. http://dx.doi.org/10.3847/1538-4357/ac68e8.

Full text

Abstract:

Abstract We present the Local Volume Complete Cluster Survey (LoVoCCS; we pronounce it as “low-vox” or “law-vox,” with stress on the second syllable), an NSF’s National Optical-Infrared Astronomy Research Laboratory survey program that uses the Dark Energy Camera to map the dark matter distribution and galaxy population in 107 nearby (0.03 < z < 0.12) X-ray luminous ([0.1–2.4 keV] L X500 > 1044 erg s−1) galaxy clusters that are not obscured by the Milky Way. The survey will reach Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) Year 1–2 depth (for galaxies r = 24.5, i = 24.0, signal-to-noise ratio (S/N) > 20; u = 24.7, g = 25.3, z = 23.8, S/N > 10) and conclude in ∼2023 (coincident with the beginning of LSST science operations), and will serve as a zeroth-year template for LSST transient studies. We process the data using the LSST Science Pipelines that include state-of-the-art algorithms and analyze the results using our own pipelines, and therefore the catalogs and analysis tools will be compatible with the LSST. We demonstrate the use and performance of our pipeline using three X-ray luminous and observation-time complete LoVoCCS clusters: A3911, A3921, and A85. A3911 and A3921 have not been well studied previously by weak lensing, and we obtain similar lensing analysis results for A85 to previous studies. (We mainly use A3911 to show our pipeline and give more examples in the Appendix.)

APA, Harvard, Vancouver, ISO, and other styles

13

Xu, Yifei, Fan Yang-Turner, Denis Volk, and Derrick Crook. "NanoSPC: a scalable, portable, cloud compatible viral nanopore metagenomic data processing pipeline." Nucleic Acids Research 48, W1 (May 22, 2020): W366—W371. http://dx.doi.org/10.1093/nar/gkaa413.

Full text

Abstract:

Abstract Metagenomic sequencing combined with Oxford Nanopore Technology has the potential to become a point-of-care test for infectious disease in public health and clinical settings, providing rapid diagnosis of infection, guiding individual patient management and treatment strategies, and informing infection prevention and control practices. However, publicly available, streamlined, and reproducible pipelines for analyzing Nanopore metagenomic sequencing data are still lacking. Here we introduce NanoSPC, a scalable, portable and cloud compatible pipeline for analyzing Nanopore sequencing data. NanoSPC can identify potentially pathogenic viruses and bacteria simultaneously to provide comprehensive characterization of individual samples. The pipeline can also detect single nucleotide variants and assemble high quality complete consensus genome sequences, permitting high-resolution inference of transmission. We implement NanoSPC using Nextflow manager within Docker images to allow reproducibility and portability of the analysis. Moreover, we deploy NanoSPC to our scalable pathogen pipeline platform, enabling elastic computing for high throughput Nanopore data on HPC cluster as well as multiple cloud platforms, such as Google Cloud, Amazon Elastic Computing Cloud, Microsoft Azure and OpenStack. Users could either access our web interface (https://nanospc.mmmoxford.uk) to run cloud-based analysis, monitor process, and visualize results, as well as download Docker images and run command line to analyse data locally.

APA, Harvard, Vancouver, ISO, and other styles

14

Zhang, Fan, Feng Wang, Wei Wang, Wei Dai, Hui Deng, Kai Fan Ji, and Yi Hua Yan. "High Performance Data Processing Pipeline of Chinese Solar Radio Heliograph." Applied Mechanics and Materials 347-350 (August 2013): 1012–17. http://dx.doi.org/10.4028/www.scientific.net/amm.347-350.1012.

Full text

Abstract:

The Chinese Solar Radio Heliograph (CSRH) is a new generation radio heliograph would produce more than 4 terabytes data every day. As a aperture synthesis telescope, CSRH is facing the challenge of processing and storing such a vast data. Pipeline system is the key issue of data automatical processing for CSRH. In this study, to push the development of CSRH, we present a framework of high performance data processing pipeline system for saving and processing real-time observation data. The related techniques of pipeline software are presented in detail including raw data acquisition, UVFITS file, data calibration, parallel computing and data publication. The pipeline has been deployed and has played an important role for the development of CSRH.

APA, Harvard, Vancouver, ISO, and other styles

15

Ismail, Mohd Fadly Hisham, Zazilah May, Vijanth Sagayan Asirvadam, and Nazrul Anuar Nayan. "Machine-Learning-Based Classification for Pipeline Corrosion with Monte Carlo Probabilistic Analysis." Energies 16, no. 8 (April 21, 2023): 3589. http://dx.doi.org/10.3390/en16083589.

Full text

Abstract:

Pipeline corrosion is one of the leading causes of failures in the transmission of gas and hazardous liquids in the oil and gas industry. In-line inspection is a non-destructive inspection for detecting corrosion defects in pipelines. Defects are measured in terms of their width, length and depth. Consecutive in-line inspection data are used to determine the pipeline’s corrosion growth rate and its remnant life, which set the operational and maintenance activities of the pipeline. The traditional approach of manually processing in-line inspection data has various weaknesses, including being time consuming due to huge data volume and complexity, prone to error, subject to biased judgement by experts and challenging for matching of in-line inspection datasets. This paper aimed to contribute to the adoption of machine learning approaches in classifying pipeline defects as per Pipeline Operator Forum requirements and matching in-line inspection data for determining the corrosion growth rate and remnant life of pipelines. Machine learning techniques, namely, decision tree, random forest, support vector machines and logistic regression, were applied in the classification of pipeline defects using Phyton programming. The performance of each technique in terms of the accuracy of results was compared. The results showed that the decision tree classifier model was the most accurate (99.9%) compared with the other classifiers.

APA, Harvard, Vancouver, ISO, and other styles

16

Guy, J., S. Bailey, A. Kremin, Shadab Alam, D. M. Alexander, C. Allende Prieto, S. BenZvi, et al. "The Spectroscopic Data Processing Pipeline for the Dark Energy Spectroscopic Instrument." Astronomical Journal 165, no. 4 (March 3, 2023): 144. http://dx.doi.org/10.3847/1538-3881/acb212.

Full text

Abstract:

Abstract We describe the spectroscopic data processing pipeline of the Dark Energy Spectroscopic Instrument (DESI), which is conducting a redshift survey of about 40 million galaxies and quasars using a purpose-built instrument on the 4 m Mayall Telescope at Kitt Peak National Observatory. The main goal of DESI is to measure with unprecedented precision the expansion history of the universe with the baryon acoustic oscillation technique and the growth rate of structure with redshift space distortions. Ten spectrographs with three cameras each disperse the light from 5000 fibers onto 30 CCDs, covering the near-UV to near-infrared (3600–9800 Å) with a spectral resolution ranging from 2000 to 5000. The DESI data pipeline generates wavelength- and flux-calibrated spectra of all the targets, along with spectroscopic classifications and redshift measurements. Fully processed data from each night are typically available to the DESI collaboration the following morning. We give details about the pipeline’s algorithms, and provide performance results on the stability of the optics, the quality of the sky background subtraction, and the precision and accuracy of the instrumental calibration. This pipeline has been used to process the DESI Survey Validation data set, and has exceeded the project’s requirements for redshift performance, with high efficiency and a purity greater than 99% for all target classes.

APA, Harvard, Vancouver, ISO, and other styles

17

Gillmann, Christina, Pablo Arbelaez, Jose Hernandez, Hans Hagen, and Thomas Wischgoll. "An Uncertainty-Aware Visual System for Image Pre-Processing." Journal of Imaging 4, no. 9 (September 10, 2018): 109. http://dx.doi.org/10.3390/jimaging4090109.

Full text

Abstract:

Due to image reconstruction process of all image capturing methods, image data is inherently affected by uncertainty. This is caused by the underlying image reconstruction model, that is not capable to map all physical properties in its entirety. In order to be aware of these effects, image uncertainty needs to be quantified and propagated along the entire image processing pipeline. In classical image processing methodologies, pre-processing algorithms do not consider this information. Therefore, this paper presents an uncertainty-aware image pre-processing paradigm, that is aware of the input image’s uncertainty and propagates it trough the entire pipeline. To accomplish this, we utilize rules for transformation and propagation of uncertainty to incorporate this additional information with a variety of operations. Resulting from this, we are able to adapt prominent image pre-processing algorithms such that they consider the input images uncertainty. Furthermore, we allow the composition of arbitrary image pre-processing pipelines and visually encode the accumulated uncertainty throughout this pipeline. The effectiveness of the demonstrated approach is shown by creating image pre-processing pipelines for a variety of real world datasets.

APA, Harvard, Vancouver, ISO, and other styles

18

Hokamp, K., F. M. Roche, M. Acab, M. E. Rousseau, B. Kuo, D. Goode, D. Aeschliman, et al. "ArrayPipe: a flexible processing pipeline for microarray data." Nucleic Acids Research 32, Web Server (July 1, 2004): W457—W459. http://dx.doi.org/10.1093/nar/gkh446.

Full text

APA, Harvard, Vancouver, ISO, and other styles

19

Perrin, Sandrine, Cyril Firmo, Sophie Lemoine, Stéphane Le Crom, and Laurent Jourdren. "Aozan: an automated post-sequencing data-processing pipeline." Bioinformatics 33, no. 14 (March 24, 2017): 2212–13. http://dx.doi.org/10.1093/bioinformatics/btx154.

Full text

APA, Harvard, Vancouver, ISO, and other styles

20

Albrizio, Rosa, Albino Mazzone, Nicola Veneziani, and Giovanni Aloisio. "Parallel/pipeline multiprocessor architectures for SAR data processing." European Transactions on Telecommunications 2, no. 6 (November 1991): 635–42. http://dx.doi.org/10.1002/ett.4460020606.

Full text

APA, Harvard, Vancouver, ISO, and other styles

21

Pikoula, Maria, Constantinos Kallis, Sephora Madjiheurem, Jennifer K. Quint, Mona Bafadhel, and Spiros Denaxas. "Evaluation of data processing pipelines on real-world electronic health records data for the purpose of measuring patient similarity." PLOS ONE 18, no. 6 (June 15, 2023): e0287264. http://dx.doi.org/10.1371/journal.pone.0287264.

Full text

Abstract:

Background The ever-growing size, breadth, and availability of patient data allows for a wide variety of clinical features to serve as inputs for phenotype discovery using cluster analysis. Data of mixed types in particular are not straightforward to combine into a single feature vector, and techniques used to address this can be biased towards certain data types in ways that are not immediately obvious or intended. In this context, the process of constructing clinically meaningful patient representations from complex datasets has not been systematically evaluated. Aims Our aim was to a) outline and b) implement an analytical framework to evaluate distinct methods of constructing patient representations from routine electronic health record data for the purpose of measuring patient similarity. We applied the analysis on a patient cohort diagnosed with chronic obstructive pulmonary disease. Methods Using data from the CALIBER data resource, we extracted clinically relevant features for a cohort of patients diagnosed with chronic obstructive pulmonary disease. We used four different data processing pipelines to construct lower dimensional patient representations from which we calculated patient similarity scores. We described the resulting representations, ranked the influence of each individual feature on patient similarity and evaluated the effect of different pipelines on clustering outcomes. Experts evaluated the resulting representations by rating the clinical relevance of similar patient suggestions with regard to a reference patient. Results Each of the four pipelines resulted in similarity scores primarily driven by a unique set of features. It was demonstrated that data transformations according to each pipeline prior to clustering can result in a variation of clustering results of over 40%. The most appropriate pipeline was selected on the basis of feature ranking and clinical expertise. There was moderate agreement between clinicians as measured by Cohen’s kappa coefficient. Conclusions Data transformation has downstream and unforeseen consequences in cluster analysis. Rather than viewing this process as a black box, we have shown ways to quantitatively and qualitatively evaluate and select the appropriate preprocessing pipeline.

APA, Harvard, Vancouver, ISO, and other styles

22

Nakane, Takanori, Yasumasa Joti, Kensuke Tono, Makina Yabashi, Eriko Nango, So Iwata, Ryuichiro Ishitani, and Osamu Nureki. "Data processing pipeline for serial femtosecond crystallography at SACLA." Journal of Applied Crystallography 49, no. 3 (April 18, 2016): 1035–41. http://dx.doi.org/10.1107/s1600576716005720.

Full text

Abstract:

A data processing pipeline for serial femtosecond crystallography at SACLA was developed, based onCheetah[Bartyet al.(2014).J. Appl. Cryst.47, 1118–1131] andCrystFEL[Whiteet al.(2016).J. Appl. Cryst.49, 680–689]. The original programs were adapted for data acquisition through the SACLA API, thread and inter-node parallelization, and efficient image handling. The pipeline consists of two stages: The first, online stage can analyse all images in real time, with a latency of less than a few seconds, to provide feedback on hit rate and detector saturation. The second, offline stage converts hit images into HDF5 files and runsCrystFELfor indexing and integration. The size of the filtered compressed output is comparable to that of a synchrotron data set. The pipeline enables real-time feedback and rapid structure solution during beamtime.

APA, Harvard, Vancouver, ISO, and other styles

23

Foss, Marie K., Håvard T. Ihle, Jowita Borowska, Kieran A. Cleary, Hans Kristian Eriksen, Stuart E. Harper, Junhan Kim, et al. "COMAP Early Science. III. CO Data Processing." Astrophysical Journal 933, no. 2 (July 1, 2022): 184. http://dx.doi.org/10.3847/1538-4357/ac63ca.

Full text

Abstract:

Abstract We describe the first-season CO Mapping Array Project (COMAP) analysis pipeline that converts raw detector readouts to calibrated sky maps. This pipeline implements four main steps: gain calibration, filtering, data selection, and mapmaking. Absolute gain calibration relies on a combination of instrumental and astrophysical sources, while relative gain calibration exploits real-time total-power variations. High-efficiency filtering is achieved through spectroscopic common-mode rejection within and across receivers, resulting in nearly uncorrelated white noise within single-frequency channels. Consequently, near-optimal but biased maps are produced by binning the filtered time stream into pixelized maps; the corresponding signal bias transfer function is estimated through simulations. Data selection is performed automatically through a series of goodness-of-fit statistics, including χ 2 and multiscale correlation tests. Applying this pipeline to the first-season COMAP data, we produce a data set with very low levels of correlated noise. We find that one of our two scanning strategies (the Lissajous type) is sensitive to residual instrumental systematics. As a result, we no longer use this type of scan and exclude data taken this way from our Season 1 power spectrum estimates. We perform a careful analysis of our data processing and observing efficiencies and take account of planned improvements to estimate our future performance. Power spectrum results derived from the first-season COMAP maps are presented and discussed in companion papers.

APA, Harvard, Vancouver, ISO, and other styles

24

Pang, Gaozhao, Niannian Wang, Hongyuan Fang, Hai Liu, and Fan Huang. "Study of Damage Quantification of Concrete Drainage Pipes Based on Point Cloud Segmentation and Reconstruction." Buildings 12, no. 2 (February 15, 2022): 213. http://dx.doi.org/10.3390/buildings12020213.

Full text

Abstract:

The urban drainage system is an important part of the urban water cycle. However, with the aging of drainage pipelines and other external reasons, damages such as cracks, corrosion, and deformation of underground pipelines can cause serious consequences such as urban waterlogging and road collapse. At present, the detection of underground drainage pipelines mostly focuses on the qualitative identification of pipeline damage, and it is impossible to quantitatively analyze pipeline damage. Therefore, a method to quantify the damage volume of concrete pipes that combines surface segmentation and reconstruction is proposed. An RGB-D sensor is used to collect the damage information of the drainage pipeline, and the collected depth frame is registered to generate the pipeline’s surface point cloud. Voxel sampling and Gaussian filtering are used to improve data processing efficiency and reduce noise, respectively, and the RANSAC algorithm is used to remove the pipeline’s surface information. The ball-pivoting algorithm is used to reconstruct the surface of the segmented damage data and pipe’s surface information, and finally to obtain the damage volume. In order to evaluate, we conducted our research on real-world materials. The measurement results show that the method proposed in this paper measures an average relative error of 7.17% for the external damage volume of concrete pipes and an average relative error of 5.22% for the internal damage measurements of concrete pipes.

APA, Harvard, Vancouver, ISO, and other styles

25

Sankarasubramanian, Praveen, and Dr Ganesh E.N. "IoT based Optimal Liquid Metal Pipeline Damage Detection Using Hybrid Soft Computing Techniques." NeuroQuantology 20, no. 5 (May 18, 2022): 773–84. http://dx.doi.org/10.14704/nq.2022.20.5.nq22234.

Full text

Abstract:

The pipeline plays an important role in transporting liquid metals over long distances. However, due to the harsh conditions on the construction site, the pipelines are always exposed to structural damage due to corrosion, damage, etc. These pipelines are often vulnerable to natural and third-party events such as explosions, earthquakes, explosions, drilling and vehicle traffic. Methods for monitoring liquid metal pipes include the use of continuous pipe assessments and pipe integration management systems. One of the biggest challenges faced by liquid metal pipeline system in the past has been the real-time monitoring of seamless pipes at certain locations. Previous studies of liquid metal pipeline monitoring have rarely focused on real-time wireless data transmission and data monitoring in the Internet of Things (IoT) operating system. In this paper, we propose an optimal liquid metal pipeline damage detection using IoT sensor platform and hybrid soft computing techniques. The proposed work consists of two fold systems such as data collection and data processing unit. In data collection unit, we utilize IoT sensors deployment with the multimedia sensors to gather liquid metal pipeline images which improve the detection accuracy. In data processing unit, we introduce a hybrid cat hunting based neural network (hybrid CHNN) to detect and localize the pipeline damages/cracks to avoid unwanted leakage and accidents. Finally, we evaluate the performance of our proposed hybrid CHNN detector with the different test samples and the simulation results are compared with the existing stateof- art pipeline damage detectors in terms of accuracy, precision, Recall and F-measure.

APA, Harvard, Vancouver, ISO, and other styles

26

Li, Li, Fenghua Li, Guozhen Shi, and Kui Geng. "An Efficient Stream Data Processing Model for Multiuser Cryptographic Service." Journal of Electrical and Computer Engineering 2018 (July 31, 2018): 1–10. http://dx.doi.org/10.1155/2018/3917827.

Full text

Abstract:

In view of the demand for high-concurrency massive data encryption and decryption application services in the security field, this paper proposes a dual-channel pipeline parallel data processing model (DPP) according to the characteristics of cryptographic operations and realized cryptographic operations of cross-data streams with different service requirements in a multiuser environment. By encapsulating cryptographic operation requirements in job packages, the input data flow is divided by the dual-channel mechanism and job packages parallel scheduling, which ensures the synchronization between the processing of the dependent job packages and parallel packages and hides the processing of the independent job package in the processing of the dependent job package. Prototyping experiments prove that this model can realize the correct and rapid processing of multiservice cross-data streams. Increasing the pipeline depth and improving the processing performance in each stage of the pipeline are the key to improving the system performance.

APA, Harvard, Vancouver, ISO, and other styles

27

Lai, Kwei-Herng, Daochen Zha, Guanchu Wang, Junjie Xu, Yue Zhao, Devesh Kumar, Yile Chen, et al. "TODS: An Automated Time Series Outlier Detection System." Proceedings of the AAAI Conference on Artificial Intelligence 35, no. 18 (May 18, 2021): 16060–62. http://dx.doi.org/10.1609/aaai.v35i18.18012.

Full text

Abstract:

We present TODS, an automated Time Series Outlier Detection System for research and industrial applications. TODS is a highly modular system that supports easy pipeline construction. The basic building block of TODS is primitive, which is an implementation of a function with hyperparameters. TODS currently supports 70 primitives, including data processing, time series processing, feature analysis, detection algorithms, and a reinforcement module. Users can freely construct a pipeline using these primitives and perform end- to-end outlier detection with the constructed pipeline. TODS provides a Graphical User Interface (GUI), where users can flexibly design a pipeline with drag-and-drop. Moreover, a data-driven searcher is provided to automatically discover the most suitable pipelines given a dataset. TODS is released under Apache 2.0 license at https://github.com/datamllab/tods. A video is available on YouTube (https://youtu.be/JOtYxTclZgQ)

APA, Harvard, Vancouver, ISO, and other styles

28

Wingett, Steven W., Philip Ewels, Mayra Furlan-Magaril, Takashi Nagano, Stefan Schoenfelder, Peter Fraser, and Simon Andrews. "HiCUP: pipeline for mapping and processing Hi-C data." F1000Research 4 (November 20, 2015): 1310. http://dx.doi.org/10.12688/f1000research.7334.1.

Full text

Abstract:

HiCUP is a pipeline for processing sequence data generated by Hi-C and Capture Hi-C (CHi-C) experiments, which are techniques used to investigate three-dimensional genomic organisation. The pipeline maps data to a specified reference genome and removes artefacts that would otherwise hinder subsequent analysis. HiCUP also produces an easy-to-interpret yet detailed quality control (QC) report that assists in refining experimental protocols for future studies. The software is freely available and has already been used for processing Hi-C and CHi-C data in several recently published peer-reviewed studies.

APA, Harvard, Vancouver, ISO, and other styles

29

Fulton, T., D. A. Naylor, E. T. Polehampton, I. Valtchanov, R. Hopwood, N. Lu, J. P. Baluteau, et al. "The data processing pipeline for theHerschelSPIRE Fourier Transform Spectrometer." Monthly Notices of the Royal Astronomical Society 458, no. 2 (February 25, 2016): 1977–89. http://dx.doi.org/10.1093/mnras/stw343.

Full text

APA, Harvard, Vancouver, ISO, and other styles

30

Wang, F., Y. Mei, H. Deng, W. Wang, C. Y. Liu, D. H. Liu, S. L. Wei, et al. "Distributed Data-Processing Pipeline for Mingantu Ultrawide Spectral Radioheliograph." Publications of the Astronomical Society of the Pacific 127, no. 950 (April 2015): 383–96. http://dx.doi.org/10.1086/680852.

Full text

APA, Harvard, Vancouver, ISO, and other styles

31

Zimmer, S., L. Arrabito, T. Glanzman, T. Johnson, C. Lavalley, and A. Tsaregorodtsev. "Extending theFermi-LAT Data Processing Pipeline to the Grid." Journal of Physics: Conference Series 396, no. 3 (December 13, 2012): 032121. http://dx.doi.org/10.1088/1742-6596/396/3/032121.

Full text

APA, Harvard, Vancouver, ISO, and other styles

32

Giesselmann, Pay, Sara Hetzel, Franz-Josef Müller, Alexander Meissner, and Helene Kretzmer. "Nanopype: a modular and scalable nanopore data processing pipeline." Bioinformatics 35, no. 22 (June 13, 2019): 4770–72. http://dx.doi.org/10.1093/bioinformatics/btz461.

Full text

Abstract:

Abstract Summary Long-read third-generation nanopore sequencing enables researchers to now address a range of questions that are difficult to tackle with short read approaches. The rapidly expanding user base and continuously increasing throughput have sparked the development of a growing number of specialized analysis tools. However, streamlined processing of nanopore datasets using reproducible and transparent workflows is still lacking. Here we present Nanopype, a nanopore data processing pipeline that integrates a diverse set of established bioinformatics software while maintaining consistent and standardized output formats. Seamless integration into compute cluster environments makes the framework suitable for high-throughput applications. As a result, Nanopype facilitates comparability of nanopore data analysis workflows and thereby should enhance the reproducibility of biological insights. Availability and implementation https://github.com/giesselmann/nanopype, https://nanopype.readthedocs.io. Supplementary information Supplementary data are available at Bioinformatics online.

APA, Harvard, Vancouver, ISO, and other styles

33

Giesselmann, Pay, Sara Hetzel, Franz-Josef Müller, Alexander Meissner, and Helene Kretzmer. "Nanopype: a modular and scalable nanopore data processing pipeline." Bioinformatics 35, no. 21 (August 6, 2019): 4536. http://dx.doi.org/10.1093/bioinformatics/btz547.

Full text

APA, Harvard, Vancouver, ISO, and other styles

34

Edwards, K., R. F. Shipman, D. Kester, A. Lorenzani, and M. Melchior. "The data processing pipeline for the Herschel - HIFI instrument." Astronomy and Computing 27 (April 2019): 156–70. http://dx.doi.org/10.1016/j.ascom.2019.04.003.

Full text

APA, Harvard, Vancouver, ISO, and other styles

35

Wang, X., H. He, L. Li, R. Chen, X. W. Deng, and S. Li. "NMPP: a user-customized NimbleGen microarray data processing pipeline." Bioinformatics 22, no. 23 (October 12, 2006): 2955–57. http://dx.doi.org/10.1093/bioinformatics/btl525.

Full text

APA, Harvard, Vancouver, ISO, and other styles

36

Tibi, Rigobert, Andre Encarnacao, Sanford Ballard, Christopher J. Young, Ronald Brogan, and Amy Sundermier. "The Iterative Processing Framework: A New Paradigm for Automatic Event Building." Bulletin of the Seismological Society of America 109, no. 6 (September 17, 2019): 2501–9. http://dx.doi.org/10.1785/0120190093.

Full text

Abstract:

Abstract In a traditional data‐processing pipeline, waveforms are acquired, a detector makes the signal detections (i.e., arrival times, slownesses, and azimuths) and passes them to an associator. The associator then links the detections to the fitting‐event hypotheses to generate an event bulletin. Most of the time, this traditional pipeline requires substantial human‐analyst involvement to improve the quality of the resulting event bulletin. For the year 2017, for example, International Data Center (IDC) analysts rejected about 40% of the events in the automatic bulletin and manually built 30% of the legitimate events. We propose an iterative processing framework (IPF) that includes a new data‐processing module that incorporates automatic analyst behaviors (auto analyst [AA]) into the event‐building pipeline. In the proposed framework, through an iterative process, the AA takes over many of the tasks traditionally performed by human analysts. These tasks can be grouped into two major processes: (1) evaluating small events with a low number of location‐defining arrival phases to improve their formation; and (2) scanning for and exploiting unassociated arrivals to form potential events missed by previous association runs. To test the proposed framework, we processed a two‐week period (15–28 May 2010) of the signal‐detections dataset from the IDC. Comparison with an expert analyst‐reviewed bulletin for the same time period suggests that IPF performs better than the traditional pipelines (IDC and baseline pipelines). Most of the additional events built by the AA are low‐magnitude events that were missed by these traditional pipelines. The AA also adds additional signal detections to existing events, which saves analyst time, even if the event locations are not significantly affected.

APA, Harvard, Vancouver, ISO, and other styles

37

Deutsch, Eric W., Luis Mendoza, David Shteynberg, Joseph Slagel, Zhi Sun, and Robert L. Moritz. "Trans-Proteomic Pipeline, a standardized data processing pipeline for large-scale reproducible proteomics informatics." PROTEOMICS - Clinical Applications 9, no. 7-8 (April 2, 2015): 745–54. http://dx.doi.org/10.1002/prca.201400164.

Full text

APA, Harvard, Vancouver, ISO, and other styles

38

Li, Rui, Maolin Cai, Yan Shi, Qingshan Feng, and Pengchao Chen. "Technologies and application of pipeline centerline and bending strain of In-line inspection based on inertial navigation." Transactions of the Institute of Measurement and Control 40, no. 5 (March 29, 2017): 1554–67. http://dx.doi.org/10.1177/0142331216685392.

Full text

Abstract:

Inertial mapping unit (IMU) in-line inspection (ILI) has become routine practice for long-distance buried transport pipelines of oil and gas. It is capable of measuring the pipeline centerline position coordinates and locating the pipeline anomalies, features and fittings to help the oil company manage it. The IMU inspection data also can be used to compute the pipeline bending strain and assess the potential deviation from the original position where endures the extra stress. This paper introduces the main principle, measurement and data processing for IMU ILI. As a key point of calculation for centerline and bending strain, the identification and optimization of the signal are also discussed. At the end of this paper, the developments of IMU ILI are presented. The IMU ILI becomes an important and effective method for pipeline integrity management and safe operation of buried oil and gas pipelines.

APA, Harvard, Vancouver, ISO, and other styles

39

Liu, Bao Jun, Ling Zhang, and Chen Guan. "Hydraulic Experimental Study on Nonmetallic Plastic Pipeline." Advanced Materials Research 594-597 (November 2012): 1961–64. http://dx.doi.org/10.4028/www.scientific.net/amr.594-597.1961.

Full text

Abstract:

Corrosion of steel pipeline can form numerous leaks, reduce service life of pipeline, and make large losses. Use of nonmetallic pipeline can solve the problem. Glass fiber reinforced plastic pipeline is nonmetallic pipeline. It can be installed conveniently, would not corrode in electrochemistry environment, and its service life is very long. So, it is widely used in many fields such as petroleum, city water supply, waste water processing, etc. In the paper, hydraulic calculations of pipeline are researched, and frictional resistance losses of water in glass fiber reinforced plastic pipelines with different diameters are measured. Based on experimental data, errors between experimental values and calculation values of formulas for hydraulic calculation are analyzed. The formulae to calculate the frictional resistance in glass fiber reinforced plastic pipeline are recommended.

APA, Harvard, Vancouver, ISO, and other styles

40

Sartoretti, P., D. Katz, M. Cropper, P. Panuzzo, G. M. Seabroke, Y. Viala, K. Benson, et al. "Gaia Data Release 2." Astronomy & Astrophysics 616 (August 2018): A6. http://dx.doi.org/10.1051/0004-6361/201832836.

Full text

Abstract:

Context. The Gaia Data Release 2 (DR2) contains the first release of radial velocities complementing the kinematic data of a sample of about 7 million relatively bright, late-type stars. Aims. This paper provides a detailed description of the Gaia spectroscopic data processing pipeline, and of the approach adopted to derive the radial velocities presented in DR2. Methods. The pipeline must perform four main tasks: (i) clean and reduce the spectra observed with the Radial Velocity Spectrometer (RVS); (ii) calibrate the RVS instrument, including wavelength, straylight, line-spread function, bias non-uniformity, and photometric zeropoint; (iii) extract the radial velocities; and (iv) verify the accuracy and precision of the results. The radial velocity of a star is obtained through a fit of the RVS spectrum relative to an appropriate synthetic template spectrum. An additional task of the spectroscopic pipeline was to provide first-order estimates of the stellar atmospheric parameters required to select such template spectra. We describe the pipeline features and present the detailed calibration algorithms and software solutions we used to produce the radial velocities published in DR2. Results. The spectroscopic processing pipeline produced median radial velocities for Gaia stars with narrow-band near-IR magnitude GRVS ≤ 12 (i.e. brighter than V ~ 13). Stars identified as double-lined spectroscopic binaries were removed from the pipeline, while variable stars, single-lined, and non-detected double-lined spectroscopic binaries were treated as single stars. The scatter in radial velocity among different observations of a same star, also published in Gaia DR2, provides information about radial velocity variability. For the hottest (Teff ≥ 7000 K) and coolest (Teff ≤ 3500 K) stars, the accuracy and precision of the stellar parameter estimates are not sufficient to allow selection of appropriate templates. The radial velocities obtained for these stars were removed from DR2. The pipeline also provides a first-order estimate of the performance obtained. The overall accuracy of radial velocity measurements is around ~200–300 m s−1, and the overall precision is ~1 km s−1; it reaches ~200 m s−1 for the brightest stars.

APA, Harvard, Vancouver, ISO, and other styles

41

Kalasarinis, Ioannis, and Anestis Koutsoudis. "Assisting Pottery Restoration Procedures with Digital Technologies." International Journal of Computational Methods in Heritage Science 3, no. 1 (January 2019): 20–32. http://dx.doi.org/10.4018/ijcmhs.2019010102.

Full text

Abstract:

The fragmentary nature of pottery is considered a common place. Conservators are requested to apply a proper restoration solution by taking under consideration a wide range of morphological features and physicochemical properties that derive from the artefact itself. In this work, the authors discuss on a low-cost pottery-oriented restoration pipeline that is based on the exploitation of technologies such as 3D digitisation, data analysis, processing and printing. The pipeline uses low-cost commercial and open source software tools and on the authors' previously published 3D pose normalisation algorithm that was initially designed for 3D vessel shape matching. The authors objectively evaluate the pipeline by applying it on two ancient Greek vessels of the Hellenistic period. The authors describe in detail the involved procedures such as the photogrammetric 3D digitisation, the 3D data analysis and processing, the 3D printing procedures and the synthetic shreds post processing. They quantify the pipeline's applicability and efficiency in terms of cost, knowledge overhead and other aspects related to restoration tasks.

APA, Harvard, Vancouver, ISO, and other styles

42

Toropov, E. S., S. M. Dorofeev, T. G. Ponomareva, and S. Yu Toropov. "Repair-and-renewal operations of pipelines from the data on their maintenance." Oil and Gas Studies, no. 5 (November 12, 2020): 94–103. http://dx.doi.org/10.31660/0445-0108-2020-5-94-103.

Full text

Abstract:

Maintaining of the pipeline system in an operational condition can't be achieved without solving the problem of their protection from internal corrosion as the main factor that leads to numerous accidents. In conditions of limited funding, the creation of scientifically based methods that regulate repair work on difficult areas [1] or those that are not repairable using "classical" methods is a very urgent task. In this way, the use of repair methods without stopping product pumping, in terms of justifying the placement of technological equipment, even more increase the importance of the problem being solved. Research methods are experimental and theoretical character and based on the analysis and processing of statistical data received during the experimental studies of field objects. The result of this work was the creation of a methodology that allows determining the order of repair work on pipelines with different degrees of corrosion damage and its speed on different sections of the route [2]. And as a result, reasonable placement of technological equipment along the pipeline route for in-line pipeline repair, without stopping the pumping of the transported product.

APA, Harvard, Vancouver, ISO, and other styles

43

Jensen, Scott, Beth Plale, Mehmet S. Aktas, Yuan Luo, Peng Chen, and Helen Conover. "Provenance Capture and Use in a Satellite Data Processing Pipeline." IEEE Transactions on Geoscience and Remote Sensing 51, no. 11 (November 2013): 5090–97. http://dx.doi.org/10.1109/tgrs.2013.2266929.

Full text

APA, Harvard, Vancouver, ISO, and other styles

44

von Landesberger, Tatiana, Dieter W. Fellner, and Roy A. Ruddle. "Visualization System Requirements for Data Processing Pipeline Design and Optimization." IEEE Transactions on Visualization and Computer Graphics 23, no. 8 (August 1, 2017): 2028–41. http://dx.doi.org/10.1109/tvcg.2016.2603178.

Full text

APA, Harvard, Vancouver, ISO, and other styles

45

Lyon, R. J., B. W. Stappers, L. Levin, M. B. Mickaliger, and A. Scaife. "A processing pipeline for high volume pulsar candidate data streams." Astronomy and Computing 28 (July 2019): 100291. http://dx.doi.org/10.1016/j.ascom.2019.100291.

Full text

APA, Harvard, Vancouver, ISO, and other styles

46

Goncalves, Angela, Andrew Tikhonov, Alvis Brazma, and Misha Kapushesky. "A pipeline for RNA-seq data processing and quality assessment." Bioinformatics 27, no. 6 (January 13, 2011): 867–69. http://dx.doi.org/10.1093/bioinformatics/btr012.

Full text

APA, Harvard, Vancouver, ISO, and other styles

47

Christley, Scott, Mikhail Levin, John Fonner, Nancy Monson, William H. Rounds, Florian Rubelt, Walter Scarborough, Richard H. Scheuermann, Inimary Toby, and Lindsay G. Cowell. "VDJPipe: a pre-processing pipeline for immune repertoire sequencing data." Journal of Immunology 196, no. 1_Supplement (May 1, 2016): 209.26. http://dx.doi.org/10.4049/jimmunol.196.supp.209.26.

Full text

Abstract:

Abstract Pre-processing of high-throughput sequencing data for immune repertoire profiling is essential to insure high quality input for downstream analysis. VDJPipe is a flexible, high-performance tool that can perform multiple pre-processing tasks with just a single pass of the data. These tasks include base composition statistics, read quality statistics, numerous quality filters, homopolymer filtering, length and nucleotide filtering, barcode demultiplexing, 5′ and 3′ PCR primer matching, and filtering of duplicate reads. VDJPipe utilizes a “pipeline” approach whereby multiple processing steps are described in a sequential workflow, with the output of each step passed as input to the next step. The workflow is flexible enough to handle the complex barcoding schemes used in many immunosequencing experiments. VDJPipe is an integral part of the VDJServer (http://www.vdjserver.org) web resource for performing immune repertoire analysis and can be accessed via the VDJServer Software page.

APA, Harvard, Vancouver, ISO, and other styles

48

Breckels, Lisa M., Claire M. Mulvey, Kathryn S. Lilley, and Laurent Gatto. "A Bioconductor workflow for processing and analysing spatial proteomics data." F1000Research 5 (December 28, 2016): 2926. http://dx.doi.org/10.12688/f1000research.10411.1.

Full text

Abstract:

Spatial proteomics is the systematic study of protein sub-cellular localisation. In this workflow, we describe the analysis of a typical quantitative mass spectrometry-based spatial proteomics experiment using the MSnbase and pRoloc Bioconductor package suite. To walk the user through the computational pipeline, we use a recently published experiment predicting protein sub-cellular localisation in pluripotent embryonic mouse stem cells. We describe the software infrastructure at hand, importing and processing data, quality control, sub-cellular marker definition, visualisation and interactive exploration. We then demonstrate the application and interpretation of statistical learning methods, including novelty detection using semi-supervised learning, classification, clustering and transfer learning and conclude the pipeline with data export. The workflow is aimed at beginners who are familiar with proteomics in general and spatial proteomics in particular.

APA, Harvard, Vancouver, ISO, and other styles

49

Breckels, Lisa M., Claire M. Mulvey, Kathryn S. Lilley, and Laurent Gatto. "A Bioconductor workflow for processing and analysing spatial proteomics data." F1000Research 5 (July 3, 2018): 2926. http://dx.doi.org/10.12688/f1000research.10411.2.

Full text

Abstract:

Spatial proteomics is the systematic study of protein sub-cellular localisation. In this workflow, we describe the analysis of a typical quantitative mass spectrometry-based spatial proteomics experiment using the MSnbase and pRoloc Bioconductor package suite. To walk the user through the computational pipeline, we use a recently published experiment predicting protein sub-cellular localisation in pluripotent embryonic mouse stem cells. We describe the software infrastructure at hand, importing and processing data, quality control, sub-cellular marker definition, visualisation and interactive exploration. We then demonstrate the application and interpretation of statistical learning methods, including novelty detection using semi-supervised learning, classification, clustering and transfer learning and conclude the pipeline with data export. The workflow is aimed at beginners who are familiar with proteomics in general and spatial proteomics in particular.

APA, Harvard, Vancouver, ISO, and other styles

50

Zhang, Weikang, Haihang You, Chao Wang, Hong Zhang, and Yixian Tang. "Parallel Optimization for Large Scale Interferometric Synthetic Aperture Radar Data Processing." Remote Sensing 15, no. 7 (March 30, 2023): 1850. http://dx.doi.org/10.3390/rs15071850.

Full text

Abstract:

Interferometric synthetic aperture radar (InSAR) has developed rapidly over the past years and is considered as an important method for surface deformation monitoring, benefiting from growing data quantities and improving data quality. However, the handing of SAR big data poses significant challenges for related algorithms and pipeline, particularly in large-scale SAR data processing. In addition, InSAR algorithms are highly complex, and their task dependencies are intricate. There is a lack of efficient optimization models and task scheduling for InSAR pipeline. In this paper, we design parallel time-series InSAR processing models based on multi-thread technology for high efficiency in processing InSAR big data. These models concentrate on parallelizing critical algorithms that have high complexity, with a focus on deconstructing two computationally intensive algorithms through loop unrolling. Our parallel models have shown a significant improvement of 10–20 times in performance. We have also developed a parallel optimization tool, Simultaneous Task Automatic Runtime (STAR), which utilizes a data flow optimization strategy with thread pool technology to address the problem of low CPU utilization resulting from multiple modules and task dependencies in the InSAR processing pipeline. STAR provides a data-driven pipeline and enables concurrent execution of multiple tasks, with greater flexibility to keep the CPU busy and further improve CPU utilization through predetermined task flow. Additionally, a supercomputing-based system has been constructed for processing massive InSAR scientific big data and providing technical support for nationwide surface deformation measurement, in accordance with the framework of time series InSAR data processing. Using this system, we processed InSAR data with the volumes of 500 TB and 700 TB in 5 and 7 days, respectively. Finally we generated two maps of land surface deformation all over China.

APA, Harvard, Vancouver, ISO, and other styles

Journal articles on the topic 'Data processing pipeline'

Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles