To see the other types of publications on this topic, follow the link: Compute as a Commodity (CaaC).
Journal articles on the topic 'Compute as a Commodity (CaaC)'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Compute as a Commodity (CaaC).'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Dr.A.Shaji, George. "Democratizing Compute Power: The Rise of Computation as a Commodity and its Impacts." Partners Universal Innovative Research Publication (PUIRP) 02, no. 03 (2024): 57–74. https://doi.org/10.5281/zenodo.11654354.
Full textAbstract:
This paper investigates the emerging concept of Compute as a Commodity (CaaC), which promises to revolutionize business innovation by providing easy access to vast compute resources, unlocked by cloud computing. CaaC aims to treat compute like electricity or water - conveniently available for consumption on demand. The pay-as-you-go cloud model enables click-button provisioning of processing capacity, without major capital investments. Our research defines CaaC, its objectives of ubiquitous, low-cost compute, and its self-service consumption vision. We analyze the CaaC technical model, which comprises a code/data repository, automated resource discovery, and a dynamic deployment engine. Innovations like spot pricing, provider federation, and deployment automation are highlighted. Numerous CaaC benefits are studied, including heightened business agility from scalable compute, lowered costs from utilizing surplus capacity, and boosted creativity from removing innovation barriers. Despite its advantages, CaaC poses infrastructural intricacies around seamless management across environments. Our work then elucidates CaaC's transformative capacity across verticals like healthcare, banking, media, and retail. For instance, healthcare workloads around genomic sequencing, drug discovery datasets, clinical trial analytics, personalized medicine, and more can leverage CaaC's elastic resources. Financial sectors can tap scalable computing to enable real-time fraud analysis, trade insights, and security evaluations. Media production houses can parallelize rendering and animation via CaaC instead of investing in high-performance computing farms. Further CaaC innovations are expected to be elaborated, like edge computing for reduced latency analytics, quantum computing for tackling complex optimizations, and serverless architectures for simplified access. In conclusion, CaaC represents an important shift in democratizing compute power, unlocking a new wave of innovation by making high-performance computing affordable and accessible. As CaaC matures, widespread adoption can transform businesses, industries, and society by accelerating digital transformation and fueling new data-driven competition. This paper serves as a primer on CaaC capabilities and provides both technological and strategic recommendations for its adoption. Further research can evaluate the societal impacts of democratized computing and guide policy decisions around data regulation, algorithmic accountability, and technology leadership in the age of CaaC.
2
Pankaj, Kumar Rai, Pandey Digvijay, and Kumar Pandey Binay. "The Future of Enterprise and Innovation is Compute as a Commodity, or CaaC." Partners Universal International Research Journal (PUIRJ) 03, no. 02 (2024): 89–94. https://doi.org/10.5281/zenodo.11973804.
Full textAbstract:
A great amount of processing power may now be accessible with only a few clicks of the mouse owing to the advent of cloud computing, which has made this option viable. Cloud computing has made it possible to access this data. "Computer as a commodity" rather than "Computer as a service" is the appropriate manner in which businesses should start addressing its use for the very first time. The method in which companies approach computers for the purposes of conducting research and carrying out commercial operations will undergo a significant transformation as a consequence of this. This is because of the fact that this is the case.
3
Zasada, S. J., D. W. Wright, and P. V. Coveney. "Large-scale binding affinity calculations on commodity compute clouds." Interface Focus 10, no. 6 (2020): 20190133. http://dx.doi.org/10.1098/rsfs.2019.0133.
Full textAbstract:
In recent years, it has become possible to calculate binding affinities of compounds bound to proteins via rapid, accurate, precise and reproducible free energy calculations. This is imperative in drug discovery as well as personalized medicine. This approach is based on molecular dynamics (MD) simulations and draws on sequence and structural information of the protein and compound concerned. Free energies are determined by ensemble averages of many MD replicas, each of which requires hundreds of cores and/or GPU accelerators, which are now available on commodity cloud computing platforms; there are also requirements for initial model building and subsequent data analysis stages. To automate the process, we have developed a workflow known as the binding affinity calculator. In this paper, we focus on the software infrastructure and interfaces that we have developed to automate the overall workflow and execute it on commodity cloud platforms, in order to reliably predict their binding affinities on time scales relevant to the domains of application, and illustrate its application to two free energy methods.
4
Bouhali, Othmane, and Ali Sheharyar. "Distributed rendering of computer-generated images on commodity compute clusters." Qatar Foundation Annual Research Forum Proceedings, no. 2012 (October 2012): CSP16. http://dx.doi.org/10.5339/qfarf.2012.csp16.
Full text
5
Park, Chonhyon, Jia Pan, and Dinesh Manocha. "Real-Time Optimization-Based Planning in Dynamic Environments Using GPUs." Proceedings of the International Symposium on Combinatorial Search 3, no. 1 (2021): 168–70. http://dx.doi.org/10.1609/socs.v3i1.18263.
Full textAbstract:
We present a novel algorithm to compute collision-free trajectories in dynamic environments. Our approach is general and makes no assumption about the obstacles or their motion. We use a replanning framework that interleaves optimization-based planning with execution. Furthermore, we describe a parallel formulation that exploits high number of cores on commodity graphics processors (GPUs) to compute a high-quality path in a given time interval. Overall, we show that search in configuration spaces can be significantly accelerated by using GPU parallelism.
6
Gu, Yunhong, and Robert L. Grossman. "Sector and Sphere: the design and implementation of a high-performance data cloud." Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 367, no. 1897 (2009): 2429–45. http://dx.doi.org/10.1098/rsta.2009.0053.
Full textAbstract:
Cloud computing has demonstrated that processing very large datasets over commodity clusters can be done simply, given the right programming model and infrastructure. In this paper, we describe the design and implementation of the Sector storage cloud and the Sphere compute cloud. By contrast with the existing storage and compute clouds, Sector can manage data not only within a data centre, but also across geographically distributed data centres. Similarly, the Sphere compute cloud supports user-defined functions (UDFs) over data both within and across data centres. As a special case, MapReduce-style programming can be implemented in Sphere by using a Map UDF followed by a Reduce UDF. We describe some experimental studies comparing Sector/Sphere and Hadoop using the Terasort benchmark. In these studies, Sector is approximately twice as fast as Hadoop. Sector/Sphere is open source.
7
Abbes, Heithem, Franck Butelle, and Christophe Cérin. "Parallelization of Littlewood-Richardson Coefficients Computation and its Integration into the BonjourGrid Meta-Desktop Grid Middleware." International Journal of Grid and High Performance Computing 3, no. 4 (2011): 71–86. http://dx.doi.org/10.4018/jghpc.2011100106.
Full textAbstract:
This paper shows how to parallelize a compute intensive application in mathematics (Group Theory) for an institutional Desktop Grid platform coordinated by a meta-grid middleware named BonjourGrid. The paper is twofold: it shows how to parallelize a sequential program for a multicore CPU which participates in the computation; and it demonstrates the effort for launching multiple instances of the solutions for the mathematical problem with the BonjourGrid middleware. BonjourGrid is a fully decentralized Desktop Grid middleware. The main results of the paper are: a) an efficient multi-threaded version of a sequential program to compute Littlewood-Richardson coefficients, namely the Multi-LR program and b) a proof of concept, centered around the user needs, for the BonjourGrid middleware dedicated to coordinate multiple instances of programsfor Desktop Grids and with the help of Multi-LR. In this paper, the scientific work consists in starting from a model for the solution of a compute intensive problem in mathematics, to incorporate the concrete model into a middleware and running it on commodity PCs platform managed by an innovative meta Desktop Grid middleware.
8
Lamberton, Barbara A. "Baier Building Products, Inc.: Performance Incentives and Variance Analysis in Sales Distribution." Issues in Accounting Education 23, no. 2 (2008): 281–90. http://dx.doi.org/10.2308/iace.2008.23.2.281.
Full textAbstract:
Faced with price volatility and changes in key personnel responsibilities, a small privately held distributor of commodity building products with limited resources needs to re-evaluate its performance incentives. The company's owners and controller need your help in assessing the extent to which the current compensation scheme has encouraged opportunistic behavior, resulting in large commissions without significant movement toward the company's strategic objectives. By completing this case successfully, you will learn how to develop a balanced scorecard suitable for a small sales distribution business and learn how to compute and interpret marketing variances.
9
Crooks, Natacha. "Efficient Data Sharing across Trust Domains." ACM SIGMOD Record 52, no. 2 (2023): 36–37. http://dx.doi.org/10.1145/3615952.3615962.
Full textAbstract:
Cross-Trust-Domain Processing. Data is now a commodity. We know how to compute and store it efficiently and reliably at scale. We have, however, paid less attention to the notion of trust. Yet, data owners today are no longer the entities storing or processing their data (medical records are stored on the cloud, data is shared across banks, etc.). In fact, distributed systems today consist of many different parties, whether it is cloud providers, jurisdictions, organisations or humans. Modern data processing and storage always straddles trust domains.
10
ASAFU-ADJAYE, JOHN, and RENUKA MAHADEVAN. "THE WELFARE EFFECTS OF THE AUSTRALIAN GOODS AND SERVICES TAX." Singapore Economic Review 47, no. 01 (2002): 49–63. http://dx.doi.org/10.1142/s0217590802000407.
Full textAbstract:
This paper analyses the effect of the Australian goods and services tax. First, we compute the social marginal cost per dollar revenue raised for nine broad commodity groups to determine whether a uniform flat rate is efficient. Second, we evaluate the welfare effects of the tax on the consumption of different income groups. The results indicate that a uniform tax may not be efficient and that the goods and services tax has adversely affected the distribution of purchasing power and thus, there is little scope for using the indirect tax system as a means to redistribute consumption towards the poor.
11
YANG, RUIGANG, MARC POLLEFEYS, HUA YANG, and GREG WELCH. "A UNIFIED APPROACH TO REAL-TIME, MULTI-RESOLUTION, MULTI-BASELINE 2D VIEW SYNTHESIS AND 3D DEPTH ESTIMATION USING COMMODITY GRAPHICS HARDWARE." International Journal of Image and Graphics 04, no. 04 (2004): 627–51. http://dx.doi.org/10.1142/s0219467804001579.
Full textAbstract:
We present a new method for using commodity graphics hardware to achieve real-time, on-line, 2D view synthesis or 3D depth estimation from two or more calibrated cameras. Our method combines a 3D plane-sweeping approach with 2D multi-resolution color consistency tests. We project camera imagery onto each plane, compute measures of color consistency throughout the plane at multiple resolutions, and then choose the color or depth (corresponding plane) that is most consistent. The key to achieving real-time performance is our use of the advanced features included with recent commodity computer graphics hardware to implement the computations simultaneously (in parallel) across all reference image pixels on a plane. Our method is relatively simple to implement, and flexible in term of the number and placement of cameras. With two cameras and an NVIDIA GeForce4 graphics card we can achieve 50–70 M disparity evaluations per second, including image download and read-back overhead. This performance matches the fastest available commercial software-only implementation of correlation-based stereo algorithms, while freeing up the CPU for other uses.
12
Chen, Siyuan, Zhuofeng Wang, Zelong Guan, Yudong Liu, and Phillip B. Gibbons. "Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors." Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 22 (2025): 23614–22. https://doi.org/10.1609/aaai.v39i22.34531.
Full textAbstract:
Fine-tuning large language models (LLMs) requires significant memory, often exceeding the capacity of a single GPU. A common solution to this memory challenge is offloading compute and data from the GPU to the CPU. However, this approach is hampered by the limited bandwidth of commodity hardware, which constrains communication between the CPU and GPU, and by slower matrix multiplications on the CPU. In this paper, we present an offloading framework, LSP-Offload, that enables near-native speed LLM fine-tuning on commodity hardware through learned sparse projectors. Our data-driven approach involves learning efficient sparse compressors that minimize communication with minimal precision loss. Additionally, we introduce a novel layer-wise communication schedule to maximize parallelism between communication and computation. As a result, our framework can fine-tune a 1.3 billion parameter model on a 4GB laptop GPU and a 6.7 billion parameter model on an NVIDIA RTX 4090 GPU with 24GB memory. Compared to state-of-the-art offloading frameworks, our approach reduces end-to-end fine-tuning time by 33.1%-62.5% when converging to the same accuracy.
13
Gülerce, Mustafa, and Gazanfer Ünal. "Using wavelet analysis to uncover the co-movement behavior of multiple energy commodity prices." International Journal of Wavelets, Multiresolution and Information Processing 14, no. 06 (2016): 1650047. http://dx.doi.org/10.1142/s0219691316500478.
Full textAbstract:
This study aims to investigate the dynamic correlations (co-movement) in between energy commodities such as WTI Crude Oil (WOIL), Brent Crude Oil (BOIL), Heating Oil and Electricity prices. To achieve this goal, we employed partial wavelet coherence (PWC) and multiple wavelet coherence (MWC). Wavelet analysis constitutes the core of these methodologies and MWC is essential to determine the dynamic correlation (co-movement) of time intervals and scales between the time series. We have developed a software program to compute PWC and MWC for quadruple data set. Coherent time intervals of the time series are determined. Vector ARMA models are shown to give a good fit due to having low mean squared errors compared to the univariate case. This allowed us to have better forecast performance.
14
Dazzi, Patrizio. "Toward Sci-φ: A Lightweight Cloud PaaS for Developing Embarrassingly Parallel Applications Based on Jini". Scientific World Journal 2014 (2014): 1–8. http://dx.doi.org/10.1155/2014/526953.
Full textAbstract:
Embarrassingly parallel problems are characterised by a very small amount of information to be exchanged among the parts they are split in, during their parallel execution. As a consequence they do not require sophisticated, low-latency, high-bandwidth interconnection networks but can be efficiently computed in parallel by exploiting commodity hardware. Basically, this means cheap clusters, networks of workstations and desktops, and Computational Clouds. This computational model can be exploited to compute a quite large range of problems. This paper describes Sci-φ, an almost complete redesign of a previous tool of ours aimed at developing task parallel applications based on Java and Jini that were shown to be an effective and efficient solution in environments like clusters and networks of workstations and desktops.
15
Assa, Hirbod. "Financial engineering in pricing agricultural derivatives based on demand and volatility." Agricultural Finance Review 76, no. 1 (2016): 42–53. http://dx.doi.org/10.1108/afr-11-2015-0053.
Full textAbstract:
Purpose – The purpose of this paper is twofold. First, the author proposes a financial engineering framework to model commodity prices based on market demand processes and demand functions. This framework explains the relation between demand, volatility and the leverage effect of commodities. It is also shown how the proposed framework can be used to price derivatives on commodity prices. Second, the author estimates the model parameters for agricultural commodities and discuss the implications of the results on derivative prices. In particular, the author see how leverage effect (or inverse leverage effect) is related to market demand. Design/methodology/approach – This paper uses a power demand function along with the Cox, Ingersoll and Ross mean-reverting process to find the price process of commodities. Then by using the Ito theorem the constant elastic volatility (CEV) model is derived for the market prices. The partial differential equation that the dynamics of derivative prices satisfy is found and, by the Feynman-Kac theorem, the market derivative prices are provided within a Monte-Carlo simulation framework. Finally, by using a maximum likelihood estimator, the parameters of the CEV model for the agricultural commodity prices are found. Findings – The results of this paper show that derivative prices on commodities are heavily affected by the elasticity of volatility and, consequently, by market demand elasticity. The empirical results show that different groups of agricultural commodities have different values of demand and volatility elasticity. Practical implications – The results of this paper can be used by practitioners to price derivatives on commodity prices and by insurance companies to better price insurance contracts. As in many countries agricultural insurances are subsidised by the government, the results of this paper are useful for setting more efficient policies. Originality/value – Approaches that use the methodology of financial engineering to model agricultural prices and compute the derivative prices are rather new within the literature and still need to be developed for further applications.
16
Boehm, J., K. Liu, and C. Alis. "SIDELOADING – INGESTION OF LARGE POINT CLOUDS INTO THE APACHE SPARK BIG DATA ENGINE." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B2 (June 7, 2016): 343–48. http://dx.doi.org/10.5194/isprs-archives-xli-b2-343-2016.
Full textAbstract:
In the geospatial domain we have now reached the point where data volumes we handle have clearly grown beyond the capacity of most desktop computers. This is particularly true in the area of point cloud processing. It is therefore naturally lucrative to explore established big data frameworks for big geospatial data. The very first hurdle is the import of geospatial data into big data frameworks, commonly referred to as data ingestion. Geospatial data is typically encoded in specialised binary file formats, which are not naturally supported by the existing big data frameworks. Instead such file formats are supported by software libraries that are restricted to single CPU execution. We present an approach that allows the use of existing point cloud file format libraries on the Apache Spark big data framework. We demonstrate the ingestion of large volumes of point cloud data into a compute cluster. The approach uses a map function to distribute the data ingestion across the nodes of a cluster. We test the capabilities of the proposed method to load billions of points into a commodity hardware compute cluster and we discuss the implications on scalability and performance. The performance is benchmarked against an existing native Apache Spark data import implementation.
17
Boehm, J., K. Liu, and C. Alis. "SIDELOADING – INGESTION OF LARGE POINT CLOUDS INTO THE APACHE SPARK BIG DATA ENGINE." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B2 (June 7, 2016): 343–48. http://dx.doi.org/10.5194/isprsarchives-xli-b2-343-2016.
Full textAbstract:
In the geospatial domain we have now reached the point where data volumes we handle have clearly grown beyond the capacity of most desktop computers. This is particularly true in the area of point cloud processing. It is therefore naturally lucrative to explore established big data frameworks for big geospatial data. The very first hurdle is the import of geospatial data into big data frameworks, commonly referred to as data ingestion. Geospatial data is typically encoded in specialised binary file formats, which are not naturally supported by the existing big data frameworks. Instead such file formats are supported by software libraries that are restricted to single CPU execution. We present an approach that allows the use of existing point cloud file format libraries on the Apache Spark big data framework. We demonstrate the ingestion of large volumes of point cloud data into a compute cluster. The approach uses a map function to distribute the data ingestion across the nodes of a cluster. We test the capabilities of the proposed method to load billions of points into a commodity hardware compute cluster and we discuss the implications on scalability and performance. The performance is benchmarked against an existing native Apache Spark data import implementation.
18
Xi, Wenwen, Dermot Hayes, and Sergio Horacio Lence. "Variance risk premia for agricultural commodities." Agricultural Finance Review 79, no. 3 (2019): 286–303. http://dx.doi.org/10.1108/afr-07-2018-0056.
Full textAbstract:
Purpose The purpose of this paper is to study the variance risk premium in corn and soybean markets, where the variance risk premium is defined as the difference between the historical realized variance and the corresponding risk-neutral expected variance. Design/methodology/approach The authors compute variance risk premiums using historical derivatives data. The authors use regression analysis and time series econometrics methods, including EGARCH and the Kalman filter, to analyze variance risk premiums. Findings There are moderate commonalities in variance within the agricultural sector, but fairly weak commonalities between the agricultural and the equity sectors. Corn and soybean variance risk premia in dollar terms are time-varying and correlated with the risk-neutral expected variance. In contrast, agricultural commodity variance risk premia in log return terms are more likely to be constant and less correlated with the log risk-neutral expected variance. Variance and price (return) risk premia in agricultural markets are weakly correlated, and the correlation depends on the sign of the returns in the underlying commodity. Practical implications Commodity variance (i.e. volatility) risk cannot be hedged using futures markets. The results have practical implications for US crop insurance programs because the implied volatilities from the relevant options markets are used to estimate the price volatility factors used to generate premia for revenue insurance products such as “Revenue Protection” and “Revenue Protection with Harvest Price Exclusion.” The variance risk premia found implies that revenue insurance premia are overpriced. Originality/value The empirical results suggest that the implied volatilities in corn and soybean futures market overestimate true expected volatility by approximately 15 percent. This has implications for derivative products, such as revenue insurance, that use these implied volatilities to calculate fair premia.
19
Li, Zhi, Yuxuan Yao, and Yuan Yuan. "The commodity recommendation algorithm for automatic security sale system based on the internet of things." Journal of Computational Methods in Sciences and Engineering 24, no. 6 (2024): 4101–16. https://doi.org/10.1177/14727978241299492.
Full textAbstract:
The objective of this paper is to investigate the promptness of recommendation data in automated product recommendation systems, enhancing their ability to facilitate product promotions and compute user similarities. This study also aims to streamline and refine the product purchasing process. To ease the user’s decision-making, an innovative design for the user terminal’s official account is proposed, creating an effective communication link between users and automated sales products. Second, based on the traditional user-based collaborative filtering recommendation algorithm, user characteristics, time factors, and promotion functions are integrated into the product recommendation algorithm model. To ensure the timeliness of recommendation information and avoid resource waste caused by expired products, time factors and promotion functions are incorporated into the recommendation algorithm model. Finally, a comprehensive test is conducted on the performance of the product recommendation system designed in this experiment. The results show that the average absolute error of the recommended model is reduced by 4.3%, and the improvement effect is significant. The average accuracy rate is increased by about 3.0% compared with the traditional recommendation algorithm. This paper aims to provide important technical support for the improvement of the product recommendation algorithm in the automatic sale system. Besides, this paper provides users with good recommendation services and convenient product purchase processes.
20
Knibbe, H., W. A. Mulder, C. W. Oosterlee, and C. Vuik. "Closing the performance gap between an iterative frequency-domain solver and an explicit time-domain scheme for 3D migration on parallel architectures." GEOPHYSICS 79, no. 2 (2014): S47—S61. http://dx.doi.org/10.1190/geo2013-0214.1.
Full textAbstract:
Three-dimensional reverse-time migration with the constant-density acoustic wave equation requires an efficient numerical scheme for the computation of wavefields. An explicit finite-difference scheme in the time domain is a common choice. However, it requires a significant amount of disk space for the imaging condition. The frequency-domain approach simplifies the correlation of the source and receiver wavefields, but requires the solution of a large sparse linear system of equations. For the latter, we use an iterative Krylov solver based on a shifted Laplace multigrid preconditioner with matrix-dependent prolongation. The question is whether migration in the frequency domain can compete with a time-domain implementation when both are performed on a parallel architecture. Both methods are naturally parallel over shots, but the frequency-domain method is also parallel over frequencies. If we have a sufficiently large number of compute nodes, we can compute the result for each frequency in parallel and the required time is dominated by the number of iterations for the highest frequency. As a parallel architecture, we consider a commodity hardware cluster that consists of multicore central processing units (CPUs), each of them connected to two graphics processing units (GPUs). Here, GPUs are used as accelerators and not as an independent compute node. The parallel implementation of the 3D migration in frequency domain is compared to a time-domain implementation. We optimize the throughput of the latter with dynamic load balancing, asynchronous I/O, and compression of snapshots. Because the frequency-domain solver uses matrix-dependent prolongation, the coarse-grid operators require more storage than available on GPUs for problems of realistic size. Due to data transfer, there is no significant speedup using GPU-accelerators. Therefore, we consider an implementation on CPUs only. Nevertheless, with the parallelization over shots and frequencies, this approach could compete with the time-domain implementation on multiple GPUs.
21
Desai, Falguni Pankaj. "A Study of Diversification in Exports of China and India." Journal of Global Economy 11, no. 4 (2015): 253–72. http://dx.doi.org/10.1956/jge.v11i4.410.
Full textAbstract:
The paper presents a detailed comparative study of the evolution in the direction and composition of exports of China and India between 1993 and 2010. The changes in the direction of exports is analyzed through country/commodity intersection by grouping countries together on the basis of income of which there are eight groups, and changes in the composition of exports is examined by employing the Revealed comparative advantage index and decomposing the growth in exports in terms of Intensive (existing products) and Extensive margins (new products) following the methodology of Amiti and Freund (2007). To analyze the country/commodity intersection and to compute the RCA index, the exports are classified into 5 main groups/categories: Product A group: Primary Products, Product B group: Natural Resource Intensive Products, Product C group: Unskilled Labour Intensive Products, Product D group: Technology Intensive Products, and Product E group: Human Capital Intensive Products and Sectors not classified according to factor intensity. The main findings of the study are: i) High income OECD plus non- OECD countries had dominant but declining shares in all the five product categories of exports of China and India, ii) China’s comparative advantage has shifted from Unskilled labour intensive group to Technology intensive group, but India continues to have comparative advantage in exporting Primary and Unskilled labour intensive products and, iii) the growth in exports of China and India was mainly accounted for by high growth of Intensive margins rather than in Extensive margins.
22
Duckworth, Daniel, Peter Hedman, Christian Reiser, et al. "SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration." ACM Transactions on Graphics 43, no. 4 (2024): 1–13. http://dx.doi.org/10.1145/3658193.
Full textAbstract:
Recent techniques for real-time view synthesis have rapidly advanced in fidelity and speed, and modern methods are capable of rendering near-photorealistic scenes at interactive frame rates. At the same time, a tension has arisen between explicit scene representations amenable to rasterization and neural fields built on ray marching, with state-of-the-art instances of the latter surpassing the former in quality while being prohibitively expensive for real-time applications. We introduce SMERF, a view synthesis approach that achieves state-of-the-art accuracy among real-time methods on large scenes with footprints up to 300 m 2 at a volumetric resolution of 3.5 mm 3 . Our method is built upon two primary contributions: a hierarchical model partitioning scheme, which increases model capacity while constraining compute and memory consumption, and a distillation training strategy that simultaneously yields high fidelity and internal consistency. Our method enables full six degrees of freedom navigation in a web browser and renders in real-time on commodity smartphones and laptops. Extensive experiments show that our method exceeds the state-of-the-art in real-time novel view synthesis by 0.78 dB on standard benchmarks and 1.78 dB on large scenes, renders frames three orders of magnitude faster than state-of-the-art radiance field models, and achieves real-time performance across a wide variety of commodity devices, including smartphones. We encourage readers to explore these models interactively at our project website: https://smerf-3d.github.io.
23
Tsamoura, Efthymia, David Carral, Enrico Malizia, and Jacopo Urbani. "Materializing knowledge bases via trigger graphs." Proceedings of the VLDB Endowment 14, no. 6 (2021): 943–56. http://dx.doi.org/10.14778/3447689.3447699.
Full textAbstract:
Thechaseis a well-established family of algorithms used to materialize Knowledge Bases (KBs) for tasks like query answering under dependencies or data cleaning. A general problem of chase algorithms is that they might perform redundant computations. To counter this problem, we introduce the notion ofTrigger Graphs(TGs), which guide the execution of the rules avoiding redundant computations. We present the results of an extensive theoretical and empirical study that seeks to answer when and how TGs can be computed and what are the benefits of TGs when applied over real-world KBs. Our results include introducing algorithms that compute (minimal) TGs. We implemented our approach in a new engine, called GLog, and our experiments show that it can be significantly more efficient than the chase enabling us to materialize Knowledge Graphs with 17B facts in less than 40 min using a single machine with commodity hardware.
24
Manohar, Nathan, Abhishek Jain, and Amit Sahai. "Self-Processing Private Sensor Data via Garbled Encryption." Proceedings on Privacy Enhancing Technologies 2020, no. 4 (2020): 434–60. http://dx.doi.org/10.2478/popets-2020-0081.
Full textAbstract:
AbstractWe introduce garbled encryption, a relaxation of secret-key multi-input functional encryption (MiFE) where a function key can be used to jointly compute upon only a particular subset of all possible tuples of ciphertexts. We construct garbled encryption for general functionalities based on one-way functions.We show that garbled encryption can be used to build a self-processing private sensor data system where after a one-time trusted setup phase, sensors deployed in the field can periodically broadcast encrypted readings of private data that can be computed upon by anyone holding function keys to learn processed output, without any interaction. Such a system can be used to periodically check, e.g., whether a cluster of servers are in an “alarm” state.We implement our garbled encryption scheme and find that it performs quite well, with function evaluations in the microseconds. The performance of our scheme was tested on a standard commodity laptop.
25
D’Ambrosio, Donato, Giuseppe Filippone, Rocco Rongo, William Spataro, and Giuseppe A. Trunfio. "Cellular Automata and GPGPU." International Journal of Grid and High Performance Computing 4, no. 3 (2012): 30–47. http://dx.doi.org/10.4018/jghpc.2012070102.
Full textAbstract:
This paper presents an efficient implementation of the SCIARA Cellular Automata computational model for simulating lava flows using the Compute Unified Device Architecture (CUDA) interface developed by NVIDIA and carried out on Graphical Processing Units (GPU). GPUs are specifically designated for efficiently processing graphic data sets. However, they are also recently being exploited for achieving excellent computational results for applications non-directly connected with Computer Graphics. The authors show an implementation of SCIARA and present results referred to a Tesla GPU computing processor, a NVIDIA device specifically designed for High Performance Computing, and a Geforce GT 330M commodity graphic card. Their carried out experiments show that significant performance improvements are achieved, over a factor of 100, depending on the problem size and type of performed memory optimization. Experiments have confirmed the effectiveness and validity of adopting graphics hardware as an alternative to expensive hardware solutions, such as cluster or multi-core machines, for the implementation of Cellular Automata models.
26
DU, LIU-GE, KANG LI, FAN-MIN KONG, and YUAN HU. "PARALLEL 3D FINITE-DIFFERENCE TIME-DOMAIN METHOD ON MULTI-GPU SYSTEMS." International Journal of Modern Physics C 22, no. 02 (2011): 107–21. http://dx.doi.org/10.1142/s012918311101618x.
Full textAbstract:
Finite-difference time-domain (FDTD) is a popular but computational intensive method to solve Maxwell's equations for electrical and optical devices simulation. This paper presents implementations of three-dimensional FDTD with convolutional perfect match layer (CPML) absorbing boundary conditions on graphics processing unit (GPU). Electromagnetic fields in Yee cells are calculated in parallel millions of threads arranged as a grid of blocks with compute unified device architecture (CUDA) programming model and considerable speedup factors are obtained versus sequential CPU code. We extend the parallel algorithm to multiple GPUs in order to solve electrically large structures. Asynchronous memory copy scheme is used in data exchange procedure to improve the computation efficiency. We successfully use this technique to simulate pointwise source radiation and validate the result by comparison to high precision computation, which shows favorable agreements. With four commodity GTX295 graphics cards on a single personal computer, more than 4000 million Yee cells can be updated in one second, which is hundreds of times faster than traditional CPU computation.
27
Medin, Safa C., Gengyan Li, Ruofei Du, et al. "FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces." Proceedings of the ACM on Computer Graphics and Interactive Techniques 7, no. 1 (2024): 1–17. http://dx.doi.org/10.1145/3651304.
Full textAbstract:
3D rendering of dynamic face captures is a challenging problem, and it demands improvements on several fronts---photorealism, efficiency, compatibility, and configurability. We present a novel representation that enables high-quality volumetric rendering of an actor's dynamic facial performances with minimal compute and memory footprint. It runs natively on commodity graphics soft- and hardware, and allows for a graceful trade-off between quality and efficiency. Our method utilizes recent advances in neural rendering, particularly learning discrete radiance manifolds to sparsely sample the scene to model volumetric effects. We achieve efficient modeling by learning a single set of manifolds for the entire dynamic sequence, while implicitly modeling appearance changes as temporal canonical texture. We export a single layered mesh and view-independent RGBA texture video that is compatible with legacy graphics renderers without additional ML integration. We demonstrate our method by rendering dynamic face captures of real actors in a game engine, at comparable photorealism to state-of-the-art neural rendering techniques at previously unseen frame rates.
28
Kumar, B., and O. Dikshit. "PARALLEL IMPLEMENTATION OF MORPHOLOGICAL PROFILE BASED SPECTRAL-SPATIAL CLASSIFICATION SCHEME FOR HYPERSPECTRAL IMAGERY." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B7 (June 21, 2016): 263–67. http://dx.doi.org/10.5194/isprs-archives-xli-b7-263-2016.
Full textAbstract:
Extended morphological profile (EMP) is a good technique for extracting spectral-spatial information from the images but large size of hyperspectral images is an important concern for creating EMPs. However, with the availability of modern multi-core processors and commodity parallel processing systems like graphics processing units (GPUs) at desktop level, parallel computing provides a viable option to significantly accelerate execution of such computations. In this paper, parallel implementation of an EMP based spectralspatial classification method for hyperspectral imagery is presented. The parallel implementation is done both on multi-core CPU and GPU. The impact of parallelization on speed up and classification accuracy is analyzed. For GPU, the implementation is done in compute unified device architecture (CUDA) C. The experiments are carried out on two well-known hyperspectral images. It is observed from the experimental results that GPU implementation provides a speed up of about 7 times, while parallel implementation on multi-core CPU resulted in speed up of about 3 times. It is also observed that parallel implementation has no adverse impact on the classification accuracy.
29
Schütz, Markus, Bernhard Kerbl, and Michael Wimmer. "Software Rasterization of 2 Billion Points in Real Time." Proceedings of the ACM on Computer Graphics and Interactive Techniques 5, no. 3 (2022): 1–17. http://dx.doi.org/10.1145/3543863.
Full textAbstract:
The accelerated collection of detailed real-world 3D data in the form of ever-larger point clouds is sparking a demand for novel visualization techniques that are capable of rendering billions of point primitives in real-time. We propose a software rasterization pipeline for point clouds that is capable of rendering up to two billion points in real-time (60 FPS) on commodity hardware. Improvements over the state of the art are achieved by batching points, enabling a number of batch-level optimizations before rasterizing them within the same rendering pass. These optimizations include frustum culling, level-of-detail (LOD) rendering, and choosing the appropriate coordinate precision for a given batch of points directly within a compute workgroup. Adaptive coordinate precision, in conjunction with visibility buffers, reduces the required data for the majority of points to just four bytes, making our approach several times faster than the bandwidth-limited state of the art. Furthermore, support for LOD rendering makes our software rasterization approach suitable for rendering arbitrarily large point clouds, and to meet the elevated performance demands of virtual reality applications.
30
Haddadin, Munther J. "Shadow water: quantification and significance for water strained countries." Water Policy 9, no. 5 (2007): 439–56. http://dx.doi.org/10.2166/wp.2007.017.
Full textAbstract:
Shadow water, a term introduced to the water literature in this paper, is shown to be a crucial component of the supply side of the population–water resources equation in water-strained countries and helps keep this equation in a state of equilibrium. A virtual environment is imagined in order to compute the water demand for the country under consideration, enabling the subject country to produce all the commodities it needs. The water demand is thus calculated in a virtual plane and is transformed to the real plane in the calculation process. The demand for each of the three purposes considered (municipal, industrial and agricultural) is determined. The blue water equivalent of green water, responsible for the support of rain-fed agriculture and range land, is calculated and added to the other agricultural water resources of blue and grey water. The demand generated by the uses as determined in the virtual model is compared with the available supply. The gap between the supply of and the demand for production water (agricultural and industrial) is bridged by shadow water through commodity imports.
31
Kumar, B., and O. Dikshit. "PARALLEL IMPLEMENTATION OF MORPHOLOGICAL PROFILE BASED SPECTRAL-SPATIAL CLASSIFICATION SCHEME FOR HYPERSPECTRAL IMAGERY." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B7 (June 21, 2016): 263–67. http://dx.doi.org/10.5194/isprsarchives-xli-b7-263-2016.
Full textAbstract:
Extended morphological profile (EMP) is a good technique for extracting spectral-spatial information from the images but large size of hyperspectral images is an important concern for creating EMPs. However, with the availability of modern multi-core processors and commodity parallel processing systems like graphics processing units (GPUs) at desktop level, parallel computing provides a viable option to significantly accelerate execution of such computations. In this paper, parallel implementation of an EMP based spectralspatial classification method for hyperspectral imagery is presented. The parallel implementation is done both on multi-core CPU and GPU. The impact of parallelization on speed up and classification accuracy is analyzed. For GPU, the implementation is done in compute unified device architecture (CUDA) C. The experiments are carried out on two well-known hyperspectral images. It is observed from the experimental results that GPU implementation provides a speed up of about 7 times, while parallel implementation on multi-core CPU resulted in speed up of about 3 times. It is also observed that parallel implementation has no adverse impact on the classification accuracy.
32
Carsten, Kutzner, Páll Szilárd, Fechner Martin, Esztermann Ansgar, L. de Groot Bert, and Grubmüller Helmut. "Accurate and Rigorous Prediction of the Changes in Protein Free Energies in a Large-Scale Mutation Scan." Journal of Computational Chemistry 36, no. 26 (2016): 1990–2008. https://doi.org/10.1002/anie.201510054.
Full textAbstract:
The molecular dynamics simulation package GROMACS runs efficiently on a wide variety of hardware from commodity workstations to high performance computing clusters. Hardware features are well-exploited with a combination of single instruction multiple data, multithreading, and message passing interface (MPI)-based single program multiple data/multiple program multiple data parallelism while graphics processing units (GPUs) can be used as accelerators to compute interactions off-loaded from the CPU. Here, we evaluate which hardware produces trajectories with GROMACS 4.6 or 5.0 in the most economical way. We have assembled and benchmarked compute nodes with various CPU/GPU combinations to identify optimal compositions in terms of raw trajectory production rate, performance-to-price ratio, energy efficiency, and several other criteria. Although hardware prices are naturally subject to trends and fluctuations, general tendencies are clearly visible. Adding any type of GPU significantly boosts a node's simulation performance. For inexpensive consumer-class GPUs this improvement equally reflects in the performance-to-price ratio. Although memory issues in consumer-class GPUs could pass unnoticed as these cards do not support error checking and correction memory, unreliable GPUs can be sorted out with memory checking tools. Apart from the obvious determinants for cost-efficiency like hardware expenses and raw performance, the energy consumption of a node is a major cost factor. Over the typical hardware lifetime until replacement of a few years, the costs for electrical power and cooling can become larger than the costs of the hardware itself. Taking that into account, nodes with a well-balanced ratio of CPU and consumer-class GPU resources produce the maximum amount of GROMACS trajectory over their lifetime.
33
Forghani-elahabad, Majid, and Omar Mutab Alsalami. "Using a Node–Child Matrix to Address the Quickest Path Problem in Multistate Flow Networks under Transmission Cost Constraints." Mathematics 11, no. 24 (2023): 4889. http://dx.doi.org/10.3390/math11244889.
Full textAbstract:
The quickest path problem in multistate flow networks, which is also known as the quickest path reliability problem (QPRP), aims at calculating the probability of successfully sending a minimum of d flow units/data/commodity from a source node to a destination node via one minimal path (MP) within a specified time frame of T units. Several exact and approximative algorithms have been proposed in the literature to address this problem. Most of the exact algorithms in the literature need prior knowledge of all of the network’s minimal paths (MPs), which is considered a weak point. In addition to the time, the budget is always limited in real-world systems, making it an essential consideration in the analysis of systems’ performance. Hence, this study considers the QPRP under cost constraints and provides an efficient approach based on a node–child matrix to address the problem without knowing the MPs. We show the correctness of the algorithm, compute the complexity results, illustrate it through a benchmark example, and describe our extensive experimental results on one thousand randomly generated test problems and well-established benchmarks to showcase its practical superiority over the available algorithms in the literature.
34
Lim, Chaemin, Suhyun Lee, Jinwoo Choi, et al. "Design and Analysis of a Processing-in-DIMM Join Algorithm: A Case Study with UPMEM DIMMs." Proceedings of the ACM on Management of Data 1, no. 2 (2023): 1–27. http://dx.doi.org/10.1145/3589258.
Full textAbstract:
Modern dual in-line memory modules (DIMMs) support processing-in-memory (PIM) by implementing in-DIMM processors (IDPs) located near memory banks. PIM can greatly accelerate in-memory join, whose performance is frequently bounded by main-memory accesses, by offloading the operations of join from host central processing units (CPUs) to the IDPs. As real PIM hardware has not been available until very recently, the prior PIM-assisted join algorithms have relied on PIM hardware simulators which assume fast shared memory between the IDPs and fast inter-IDP communication; however, on commodity PIM-enabled DIMMs, the IDPs do not share memory and demand the CPUs to mediate inter-IDP communication. Such discrepancies in the architectural characteristics make the prior studies incompatible with the DIMMs. Thus, to exploit the high potential of PIM on commodity PIM-enabled DIMMs, we need a new join algorithm designed and optimized for the DIMMs and their architectural characteristics. In this paper, we design and analyze Processing-In-DIMM Join (PID-Join), a fast in-memory join algorithm which exploits UPMEM DIMMs, currently the only publicly-available PIM-enabled DIMMs. The DIMMs impose several key challenges on efficient acceleration of join including the shared-nothing nature and limited compute capabilities of the IDPs, the lack of hardware support for fast inter-IDP communication, and the slow IDP-wise data transfers between the IDPs and the main memory. PID-Join overcomes the challenges by prototyping and evaluating hash, sort-merge, and nested-loop algorithms optimized for the IDPs, enabling fast inter-IDP communication using host CPU cache streaming and vector instructions, and facilitating fast rank-wise data transfers between the IDPs and the main memory. Our evaluation using a real system equipped with eight UPMEM DIMMs and 1,024 IDPs shows that PID-Join greatly improves the performance of in-memory join over various CPU-based in-memory join algorithms.
35
Dee, Robin, Armin Fügenschuh, and George Kaimakamis. "The Unit Re-Balancing Problem." Mathematics 9, no. 24 (2021): 3205. http://dx.doi.org/10.3390/math9243205.
Full textAbstract:
We describe the problem of re-balancing a number of units distributed over a geographic area. Each unit consists of a number of components. A value between 0 and 1 describes the current rating of each component. By a piecewise linear function, this value is converted into a nominal status assessment. The lowest of the statuses determines the efficiency of a unit, and the highest status its cost. An unbalanced unit has a gap between these two. To re-balance the units, components can be transferred. The goal is to maximize the efficiency of all units. On a secondary level, the cost for the re-balancing should be minimal. We present a mixed-integer nonlinear programming formulation for this problem, which describes the potential movement of components as a multi-commodity flow. The piecewise linear functions needed to obtain the status values are reformulated using inequalities and binary variables. This results in a mixed-integer linear program, and numerical standard solvers are able to compute proven optimal solutions for instances with up to 100 units. We present numerical solutions for a set of open test instances and a bi-criteria objective function, and discuss the trade-off between cost and efficiency.
36
Bose, Aritra, Vassilis Kalantzis, Eugenia-Maria Kontopoulou, Mai Elkady, Peristera Paschou, and Petros Drineas. "TeraPCA: a fast and scalable software package to study genetic variation in tera-scale genotypes." Bioinformatics 35, no. 19 (2019): 3679–83. http://dx.doi.org/10.1093/bioinformatics/btz157.
Full textAbstract:
Abstract Motivation Principal Component Analysis is a key tool in the study of population structure in human genetics. As modern datasets become increasingly larger in size, traditional approaches based on loading the entire dataset in the system memory (Random Access Memory) become impractical and out-of-core implementations are the only viable alternative. Results We present TeraPCA, a C++ implementation of the Randomized Subspace Iteration method to perform Principal Component Analysis of large-scale datasets. TeraPCA can be applied both in-core and out-of-core and is able to successfully operate even on commodity hardware with a system memory of just a few gigabytes. Moreover, TeraPCA has minimal dependencies on external libraries and only requires a working installation of the BLAS and LAPACK libraries. When applied to a dataset containing a million individuals genotyped on a million markers, TeraPCA requires <5 h (in multi-threaded mode) to accurately compute the 10 leading principal components. An extensive experimental analysis shows that TeraPCA is both fast and accurate and is competitive with current state-of-the-art software for the same task. Availability and implementation Source code and documentation are both available at https://github.com/aritra90/TeraPCA. Supplementary information Supplementary data are available at Bioinformatics online.
37
Chen, Shan, Yuying Ma, and Qiutong Wu. "Pricing and Replenishment Prediction Model for Vegetable Commodities Based on Mixed Integer Linear Programming Optimization XGBoost." Highlights in Science, Engineering and Technology 93 (May 8, 2024): 388–96. http://dx.doi.org/10.54097/rps0q540.
Full textAbstract:
In fresh food superstores, the replenishment volume and pricing strategy of vegetables are affected by various factors such as seasonal seasons, supply demand, purchasing costs, and profits of vegetable categories. In this paper, we use the data related to the sales flow details and wholesale prices of each commodity from July 1, 2020 to June 30, 2023 as a dataset, and build a model based on the gradient boosting tree algorithm, XGBoost algorithm [1], to predict the total daily replenishment volume and pricing strategy of each vegetable category in the coming week (July 1-7, 2023) [2]. It is also optimized using a mixed-variable 0-1 linear programming (MILP) [3] model with decision variables including a binary variable of whether to order the vegetable category or not and a continuous variable of the number of vegetable categories ordered to compute the optimal replenishment volume and pricing strategy for July 1, 2023 for each item. Through the model in this paper, under the premise of meeting the market demand for each category of vegetable goods, while keeping the control within a specific range, the superstore can get the optimal replenishment plan and pricing strategy, reasonably allocate the resources of the warehouse space, improve the market competitiveness of the superstore, and maximize the revenue.
38
Falzon, Francesca, and Evangelia Anna Markatou. "Re-visiting Authorized Private Set Intersection: A New Privacy-Preserving Variant and Two Protocols." Proceedings on Privacy Enhancing Technologies 2025, no. 1 (2025): 792–807. http://dx.doi.org/10.56553/popets-2025-0041.
Full textAbstract:
We revisit the problem of Authorized Private Set Intersection (APSI), which allows mutually untrusting parties to authorize their items using a trusted third-party judge before privately computing the intersection. We also initiate the study of Partial-APSI, a novel privacy-preserving generalization of APSI in which the client only reveals a subset of their items to a third-party semi-honest judge for authorization. Partial-APSI allows for partial verification of the set, preserving the privacy of the party whose items are being verified. Both APSI and Partial-APSI have a number of applications, including genome matching, ad conversion, and compliance with privacy policies such as the GDPR. We present two protocols based on bilinear pairings with linear communication. The first realizes the APSI functionality, is secure against a malicious client, and requires only one round of communication during the online phase. Our second protocol realizes the Partial-APSI functionality and is secure against a client that may maliciously inject elements into its input set, but who follows the protocol semi-honestly otherwise. We formally prove correctness and security of these protocols and provide an experimental evaluation to demonstrate their practicality. Our protocols can be efficiently run on commodity hardware. We also show that our protocols are massively parallelizable by running our experiments on a compute grid across 50 cores.
39
Van Aerde, M., J. Shortreed, A. M. Stewart, and M. Matthews. "Assessing the risks associated with the transport of dangerous goods by truck and rail using the RISKMOD model." Canadian Journal of Civil Engineering 16, no. 3 (1989): 326–34. http://dx.doi.org/10.1139/l89-058.
Full textAbstract:
To provide an objective tool for assessing the risks involved in the transport of dangerous goods, a model called RISKMOD-I was developed for Transport Canada by the Institute for Risk Research. The model implements the computational aspects of a comprehensive risk assessment methodology which addresses both the quantitative and qualitative aspects of dangerous goods releases following a transport incident. This paper describes the model's implementation, data requirements, and outputs.The risk assessment model consists of an analysis of link-specific accident rates and a fault tree analysis to determine the probability of a dangerous good release of a particular type. In parallel, a series of commodity-specified damage models compute the impact areas associated with representative sizes and types of releases. The impact areas for each damage threshold are then multiplied by the population density and property exposure along each link to determine the total risk associated with a dangerous goods shipment that uses the link. This estimate of total risk can optimally be further converted into a risk cost value in monetary terms and summarized for the entire route. Both the risk and risk cost estimates are intended to be used in conjunction with other subjective analyses and performance measures to provide a comprehensive decision-making aid for risk managers. Key words: dangerous goods transport, truck and rail accidents, risk analysis.
40
V, Aravinda Rajan, and Marimuthu T. "COMPARATIVE ANALYSIS OF HIGH-PERFORMANCE COMPUTING SOLUTIONS IN BIG DATA ENVIRONMENT." ICTACT Journal on Data Science and Machine Learning 4, no. 3 (2023): 451–56. https://doi.org/10.21917/ijdsml.2023.0105.
Full textAbstract:
High Performance Computing (HPC) technologies and solutions provide an increasingly important means for organizations and institutions to process larger volumes of data and to generate insights. In particular, big data environments, which are characterized by large and complex datasets, with an unbounded potential for data growth and a need to process both structured and unstructured data quickly, require advanced HPC solutions and technologies. HPC solutions are used to perform complex data transformations, analytics, and simulations, including solving complex numerical problems and modeling complex phenomena. The growing capabilities of HPC technologies provide an array of potential solutions for big data challenges. These include cloud computing, distributed computing, virtualization, high-performance storage, and faster networking solutions, to name a few. Cloud solutions are used for rapid provisioning and scalability of compute and storage resources, while virtualization technologies enable the runtime isolation of application components to scale applications to massive datasets. High-speed networking technologies enable better collaboration, data exchange, and data transfer within big data platforms. Distributed computing solutions, such as Apache Hadoop and Apache Spark, provide solutions for performing Map Reduce operations across clusters of commodity hardware. High-performance storage solutions, such as Alluxio, provide an efficient way to handle massive data sets, by providing a unified storage tier across multiple platforms, including in-memory, distributed file system, and object storage.
41
Fung, Larry S. K., Mohammad O. Sindi, and Ali H. Dogru. "Multiparadigm Parallel Acceleration for Reservoir Simulation." SPE Journal 19, no. 04 (2014): 716–25. http://dx.doi.org/10.2118/163591-pa.
Full textAbstract:
Summary With the advent of the multicore central-processing unit (CPU), today's commodity PC clusters are effectively a collection of interconnected parallel computers, each with multiple multicore CPUs and large shared random access memory (RAM), connected together by means of high-speed networks. Each computer, referred to as a compute node, is a powerful parallel computer on its own. Each compute node can be equipped further with acceleration devices such as the general-purpose graphical processing unit (GPGPU) to further speed up computational-intensive portions of the simulator. Reservoir-simulation methods that can exploit this heterogeneous hardware system can be used to solve very-large-scale reservoir-simulation models and run significantly faster than conventional simulators. Because typical PC clusters are essentially distributed share-memory computers, this suggests that the use of the mixed-paradigm parallelism (distributed-shared memory), such as message-passing interface and open multiprocessing (MPI-OMP), should work well for computational efficiency and memory use. In this work, we compare and contrast the single-paradigm programming models, MPI or OMP, with the mixed paradigm, MPI-OMP, programming model for a class of solver method that is suited for the different modes of parallelism. The results showed that the distributed memory (MPI-only) model has superior multicompute-node scalability, whereas the shared memory (OMP-only) model has superior parallel performance on a single compute node. The mixed MPI-OMP model and OMP-only model are more memory-efficient for the multicore architecture than the MPI-only model because they require less or no halo-cell storage for the subdomains. To exploit the fine-grain shared memory parallelism available on the GPGPU architecture, algorithms should be suited to the single-instruction multiple-data (SIMD) parallelism, and any recursive operations are serialized. In addition, solver methods and data store need to be reworked to coalesce memory access and to avoid shared memory-bank conflicts. Wherever possible, the cost of data transfer through the peripheral component interconnect express (PCIe) bus between the CPU and GPGPU needs to be hidden by means of asynchronous communication. We applied multiparadigm parallelism to accelerate compositional reservoir simulation on a GPGPU-equipped PC cluster. On a dual-CPU-dual-GPGPU compute node, the parallelized solver running on the dual-GPGPU Fermi M2090Q achieved up to 19 times speedup over the serial CPU (1-core) results and up to 3.7 times speedup over the parallel dual-CPU X5675 results in a mixed MPI + OMP paradigm for a 1.728-million-cell compositional model. Parallel performance shows a strong dependency on the subdomain sizes. Parallel CPU solve has a higher performance for smaller domain partitions, whereas GPGPU solve requires large partitions for each chip for good parallel performance. This is related to improved cache efficiency on the CPU for small subdomains and the loading requirement for massive parallelism on the GPGPU. Therefore, for a given model, the multinode parallel performance decreases for the GPGPU relative to the CPU as the model is further subdivided into smaller subdomains to be solved on more compute nodes. To illustrate this, a modified SPE5 (Killough and Kossack 1987) model with various grid dimensions was run to generate comparative results. Parallel performances for three field compositional models of various sizes and dimensions are included to further elucidate and contrast CPU-GPGPU single-node and multiple-node performances. A PC cluster with the Tesla M2070Q GPGPU and the 6-core Xeon X5675 Westmere was used to produce the majority of the reported results. Another PC cluster with the Tesla M2090Q GPGPU was available for some cases, and the results are reported for the modified SPE5 (Killough and Kossack 1987) problems for comparison.
42
Et. al., Sirisha N,. "Integrated Security and Privacy Framework for Big Data in Hadoop MapReduce Framework." Turkish Journal of Computer and Mathematics Education (TURCOMAT) 12, no. 11 (2021): 646–62. http://dx.doi.org/10.17762/turcomat.v12i11.5941.
Full textAbstract:
Public cloud infrastructure is widely used by enterprises to store and process big data. Cloud and its distributed computing phenomena not only provides scalable, available and affordable solution for storage and compute services but also raises security concerns. Many security solutions that came into existence encrypt data and allow accessing plaintext for data analytics in the confines of secure hardware. However, the fact remains that the large volumes of data is processed in distributed environment involving hundreds of commodity machines. There exist numerous communications between machines in MapReduce computing model. In the process, compromised MapReduce machines or functions are vulnerable to query based inference attacks on big data that lead to leakage of sensitive information. The main focus of this paper is to overcome the problem aforementioned. Towards this end, a methodology is proposed with an underlying algorithm for defeating query based inference attacks on big data in Hadoop. The proposed algorithm is known as Multi-Model Defence Against Query Based Inference Attacks (MMD-QBIA). A realistic attack model is considered for validating the effectiveness of the proposed methodology. Then an integrated framework for security and privacy to big data is evaluated. Cloudera Distribution Hadoop (CDH) is the environment used for empirical study. The experimental results revealed that the proposed solution prevents different kinds of query based inference attacks on big data besides security to big data in Hadoop MapReduce framework.
43
Zuo, Zhiqiang, Kai Wang, Aftab Hussain, et al. "Systemizing Interprocedural Static Analysis of Large-scale Systems Code with Graspan." ACM Transactions on Computer Systems 38, no. 1-2 (2021): 1–39. http://dx.doi.org/10.1145/3466820.
Full textAbstract:
There is more than a decade-long history of using static analysis to find bugs in systems such as Linux. Most of the existing static analyses developed for these systems are simple checkers that find bugs based on pattern matching. Despite the presence of many sophisticated interprocedural analyses, few of them have been employed to improve checkers for systems code due to their complex implementations and poor scalability. In this article, we revisit the scalability problem of interprocedural static analysis from a “Big Data” perspective. That is, we turn sophisticated code analysis into Big Data analytics and leverage novel data processing techniques to solve this traditional programming language problem. We propose Graspan , a disk-based parallel graph system that uses an edge-pair centric computation model to compute dynamic transitive closures on very large program graphs. We develop two backends for Graspan, namely, Graspan-C running on CPUs and Graspan-G on GPUs, and present their designs in the article. Graspan-C can analyze large-scale systems code on any commodity PC, while, if GPUs are available, Graspan-G can be readily used to achieve orders of magnitude speedup by harnessing a GPU’s massive parallelism. We have implemented fully context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases written in multiple languages such as Linux and Apache Hadoop demonstrates that their Graspan implementations are language-independent, scale to millions of lines of code, and are much simpler than their original implementations. Moreover, we show that these analyses can be used to uncover many real-world bugs in large-scale systems code.
44
Pitt-Francis, Joe, Alan Garny, and David Gavaghan. "Enabling computer models of the heart for high-performance computers and the grid." Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 364, no. 1843 (2006): 1501–16. http://dx.doi.org/10.1098/rsta.2006.1783.
Full textAbstract:
Although it is now feasible to compute multi-cellular models of the heart on a personal desktop or laptop computer, it is not feasible to undertake the detailed sweeps of high-dimensional parameter spaces required if we are to undertake in silico experimentation of the complex processes that constitute heart disease. For this research, modelling requirements move rapidly beyond the limit of commodity computers' resource both in terms of their memory footprint and the speed of calculation, so that multi-processor architectures must be considered. In addition, as such models have become more mature and have been validated against experimental data, there is increasing pressure for experimentalists to be able to make use of these models themselves as a key tool for hypothesis formulation and in planning future experimental studies to test those hypotheses. This paper discusses our initial experiences in a large-scale project (the Integrative Biology (IB) e-Science project) aimed at meeting these dual aims. We begin by putting the research in context by describing in outline the overall aims of the IB project, in particular focusing on the challenge of enabling novice users to make full use of high-performance resources without the need to gain detailed technical expertise in computing. We then discuss our experience of adapting one particular heart modelling package, Cellular Open Resource, and show how the solving engine of this code was dissected from the rest of the package, ported to C++ and parallelized using the Message-Passing Interface. We show that good parallel efficiency and realistic memory reduction can be achieved on simple geometries. We conclude by discussing lessons learnt in this process.
45
Griesbach, Svenja M., Martin Hoefer, Max Klimm, and Tim Koglin. "Information Design for Congestion Games with Unknown Demand." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 9 (2024): 9722–30. http://dx.doi.org/10.1609/aaai.v38i9.28830.
Full textAbstract:
We study a novel approach to information design in the standard traffic model of network congestion games. It captures the natural condition that the demand is unknown to the users of the network. A principal (e.g., a mobility service) commits to a signaling strategy, observes the realized demand and sends a (public) signal to agents (i.e., users of the network). Based on the induced belief about the demand, the users then form an equilibrium. We consider the algorithmic goal of the principal: Compute a signaling scheme that minimizes the expected total cost of the induced equilibrium. We concentrate on single-commodity networks and affine cost functions, for which we obtain the following results. First, we devise a fully polynomial-time approximation scheme (FPTAS) for the case that the demand can only take two values. It relies on several structural properties of the cost of the induced equilibrium as a function of the updated belief about the distribution of demands. We show that this function is piecewise linear for any number of demands, and monotonic for two demands. Second, we give a complete characterization of the graph structures for which it is optimal to fully reveal the information about the realized demand. This signaling scheme turns out to be optimal for all cost functions and probability distributions over demands if and only if the graph is series-parallel. Third, we propose an algorithm that computes the optimal signaling scheme for any number of demands whose time complexity is polynomial in the number of supports that occur in a Wardrop equilibrium for some demand. Finally, we conduct a computational study that tests this algorithm on real-world instances.
46
Romeiko, Xiaobo Xue, Wangjian Zhang, Xuesong Zhang, and Jun-Ki Choi. "Spatially Explicit Life Cycle Global Warming and Eutrophication Potentials of Confined Dairy Production in the Contiguous US." Environments 11, no. 11 (2024): 230. http://dx.doi.org/10.3390/environments11110230.
Full textAbstract:
Assessing the spatially explicit life cycle environmental impacts of livestock production systems is critical for understanding the spatial heterogeneity of environmental releases and devising spatially targeted remediation strategies. This study presents the first spatially explicit assessment on life cycle global warming and eutrophication potentials of confined dairy production at a county scale in the contiguous US. The Environmental Policy Integrated Climate model was used to estimate greenhouse gases (GHGs), NH3, and aqueous nutrient releases of feed production. The Greenhouse gases, Regulated Emissions, and Energy use in Transportation model and Commodity Flow Survey were used to assess GHGs and NH3 from feed transportation. Emission-factor-based approaches were primarily used to calculate GHGs from enteric fermentation, and GHGs, NH3, and aqueous nutrient releases from manure management. Characterization factors reported by the Intergovernmental Panel on Climate Change and Tool for Reduction and Assessment of Chemicals and other Environmental Impacts model were used to compute global warming and eutrophication potentials, respectively. The analyses revealed that life cycle global warming and eutrophication potentials of confined dairy production presented significant spatial heterogeneity among the US counties. For example, the life cycle global warming potential ranged from 462 kg CO2-eq/head to 14,189 kg CO2-eq/head. Surprisingly, sourcing feed locally cannot effectively reduce life cycle global warming and eutrophication potentials of confined dairy production. The feed supply scenarios with the lowest life cycle environmental impacts depend on the life cycle environmental impacts of feed production, geographic locations of confined dairy production, and specific impact categories. In addition, installing buffer strips in feed-producing hotspots can effectively reduce life cycle nutrient releases of confined dairy production. If 200 counties with the highest life cycle EP of corn adopt buffer strips, the reduction in life cycle EP of confined dairy production could reach 24.4%.
47
Alis, C., J. Boehm, and K. Liu. "PARALLEL PROCESSING OF BIG POINT CLOUDS USING Z-ORDER-BASED PARTITIONING." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B2 (June 7, 2016): 71–77. http://dx.doi.org/10.5194/isprs-archives-xli-b2-71-2016.
Full textAbstract:
As laser scanning technology improves and costs are coming down, the amount of point cloud data being generated can be prohibitively difficult and expensive to process on a single machine. This data explosion is not only limited to point cloud data. Voluminous amounts of high-dimensionality and quickly accumulating data, collectively known as Big Data, such as those generated by social media, Internet of Things devices and commercial transactions, are becoming more prevalent as well. New computing paradigms and frameworks are being developed to efficiently handle the processing of Big Data, many of which utilize a compute cluster composed of several commodity grade machines to process chunks of data in parallel. <br><br> A central concept in many of these frameworks is data locality. By its nature, Big Data is large enough that the entire dataset would not fit on the memory and hard drives of a single node hence replicating the entire dataset to each worker node is impractical. The data must then be partitioned across worker nodes in a manner that minimises data transfer across the network. This is a challenge for point cloud data because there exist different ways to partition data and they may require data transfer. <br><br> We propose a partitioning based on <i>Z</i>-order which is a form of locality-sensitive hashing. The <i>Z</i>-order or Morton code is computed by dividing each dimension to form a grid then interleaving the binary representation of each dimension. For example, the <i>Z</i>-order code for the grid square with coordinates (<i>x</i> = 1 = 01<sub>2</sub>, <i>y</i> = 3 = 11<sub>2</sub>) is 1011<sub>2</sub> = 11. The number of points in each partition is controlled by the number of bits per dimension: the more bits, the fewer the points. The number of bits per dimension also controls the level of detail with more bits yielding finer partitioning. We present this partitioning method by implementing it on Apache Spark and investigating how different parameters affect the accuracy and running time of the <i>k</i> nearest neighbour algorithm for a hemispherical and a triangular wave point cloud.
48
Gozlan, Itamar, Chen Avin, Gil Einziger, and Gabriel Scalosub. "Go-to-Controller is Better: Efficient and Optimal LPM Caching with Splicing." Proceedings of the ACM on Measurement and Analysis of Computing Systems 7, no. 1 (2023): 1–33. http://dx.doi.org/10.1145/3579441.
Full textAbstract:
Modern data center networks are required to support huge and complex forwarding policies as they handle the traffic of the various tenants. However, these policies cannot be stored in their entirety within the limited memory available at commodity switches. The common approach in such scenarios is to have SDN controllers manage the memory available at the switch as a fast cache, updating and changing the forwarding rules in the cache according to the workloads dynamics and the global policy at hand. Many such policies, such as Longest-prefix-match (LPM) policies, introduce dependencies between the forwarding rules. Ensuring that the cache content is always consistent with the global policy often requires the switch to store (potentially many) superfluous rules, which may lead to suboptimal performance in terms of delay and throughput. To overcome these deficiencies, previous work suggested the concept of splicing, where modified Go-to-Controller rules can be inserted into the cache to improve performance while maintaining consistency. These works focused mostly on heuristics, and it was conjectured that the problem is computationally intractable. As our main result, we show that the problem of determining the optimal set of rules, with splicing, can actually be solved efficiently by presenting a polynomial-time algorithm that produces an optimal solution, i.e., for a given cache size we find an optimal set of rules, some of which are go-to-controller, which maximize the total weight of the cache while maintaining consistency. However, such optimality comes at a cost, encompassed by the fact that our algorithm has a significantly larger running time than SoTA solutions which do not employ splicing. Therefore, we further present a heuristic exhibiting close-to-optimal performance, with significantly improved running time, matching that of the best algorithm, which does not employ splicing. In addition, we present the results of an evaluation study that compares the performance of our solutions with that of SoTA approaches, showing that splicing can reduce the cache miss ratio by as much as 30%, without increasing the cache size. Lastly, we propose a simple and fast-to-compute metric (that is consistency-oblivious) in order to evaluate the potential benefits of splicing compared to classical LPM-caching, for a given policy and traffic distribution. We show that our metric is highly correlated with such benefits, thus serving as an indication of whether splicing should be incorporated within the system architecture.
49
Alis, C., J. Boehm, and K. Liu. "PARALLEL PROCESSING OF BIG POINT CLOUDS USING Z-ORDER-BASED PARTITIONING." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B2 (June 7, 2016): 71–77. http://dx.doi.org/10.5194/isprsarchives-xli-b2-71-2016.
Full textAbstract:
As laser scanning technology improves and costs are coming down, the amount of point cloud data being generated can be prohibitively difficult and expensive to process on a single machine. This data explosion is not only limited to point cloud data. Voluminous amounts of high-dimensionality and quickly accumulating data, collectively known as Big Data, such as those generated by social media, Internet of Things devices and commercial transactions, are becoming more prevalent as well. New computing paradigms and frameworks are being developed to efficiently handle the processing of Big Data, many of which utilize a compute cluster composed of several commodity grade machines to process chunks of data in parallel. &lt;br&gt;&lt;br&gt; A central concept in many of these frameworks is data locality. By its nature, Big Data is large enough that the entire dataset would not fit on the memory and hard drives of a single node hence replicating the entire dataset to each worker node is impractical. The data must then be partitioned across worker nodes in a manner that minimises data transfer across the network. This is a challenge for point cloud data because there exist different ways to partition data and they may require data transfer. &lt;br&gt;&lt;br&gt; We propose a partitioning based on &lt;i&gt;Z&lt;/i&gt;-order which is a form of locality-sensitive hashing. The &lt;i&gt;Z&lt;/i&gt;-order or Morton code is computed by dividing each dimension to form a grid then interleaving the binary representation of each dimension. For example, the &lt;i&gt;Z&lt;/i&gt;-order code for the grid square with coordinates (&lt;i&gt;x&lt;/i&gt; = 1 = 01&lt;sub&gt;2&lt;/sub&gt;, &lt;i&gt;y&lt;/i&gt; = 3 = 11&lt;sub&gt;2&lt;/sub&gt;) is 1011&lt;sub&gt;2&lt;/sub&gt; = 11. The number of points in each partition is controlled by the number of bits per dimension: the more bits, the fewer the points. The number of bits per dimension also controls the level of detail with more bits yielding finer partitioning. We present this partitioning method by implementing it on Apache Spark and investigating how different parameters affect the accuracy and running time of the &lt;i&gt;k&lt;/i&gt; nearest neighbour algorithm for a hemispherical and a triangular wave point cloud.
50
Dodo, Mansir, Abdulmalik Badamasi, Kabir Ibrahim, et al. "Appraising the potentials of reusing plastic bottles as building blocks for housing construction at Paipe village Abuja Nigeria." Building Engineering 3, no. 1 (2024): 1459. http://dx.doi.org/10.59400/be1459.
Full textAbstract:
Plastic bottles package a multitude of commodities consumed worldwide. Upon consumption of the commodity, the disposed plastic bottles accumulate as waste, having impacts on both the aquatic and terrestrial environment. In a bid to convert such waste to wealth, plastic bottles are creatively reused for different applications, such as pedestrian bridge boats and street furniture, amongst others. Another application of reusing plastic bottles is their serving as building blocks for housing construction. Reports and research in Nigeria confirm the proliferation of plastic bottles littering the environment, which if reused in housing construction has the potential to contribute to achieving both UN Sustainable Development Goals (SDG) 11 (making human settlements sustainable) and 12 (ensuring sustainable consumption and production). Although Nigeria is traced to being the first country in Africa to reuse plastic bottles in housing construction, not much research output exists from practitioners’ experience on the potentials of reusing plastic bottles as a sustainable construction material as practiced in countries like Vietnam, India, and the Philippines, among others. As such, this study investigates the potential factors driving the practice of reusing plastic bottles in Nigeria with a view to ascertaining the satisfaction derived from the practice for sustainable housing construction. Primary data was collected using a structured questionnaire from 41 respondents identified as having experience in using plastic bottles in construction (5 staffs of Awonto Konsult as well as 36 staffs of Brains and Hammers Construction). Data was analysed descriptively using both IBM SPSS Statistics 23 as well as MS Excel to compute the Mean Score as well as the Relative Satisfaction Index (RSI). Only 30 questionnaires were successfully retrieved and fully answered. Amongst the 10 potential factors studied driving reusing plastic bottles, results show that almost all respondents tend to be ‘satisfied’ with both ‘strength and stability’ (having a Mean Value of 4.70 and RSI of 0.94) as well as ‘durability’ (having a Mean Value of 4.50; RSI of 0.90) of buildings built with plastic bottles. These two factors recorded the highest ‘satisfaction’ ratings, leaning towards ‘very satisfied’. Regarding the factor ‘fire resistance’ of buildings built with plastic bottles (having a Mean Value of 3.40; RSI of 0.68), results reveal that 50 percent of the respondents are ‘unsure’ if it is a satisfactory factor driving reusing plastic bottles or not. The study found that the satisfaction ratings of technical and environmental factors have higher appeal to respondents compared to health and safety and also financial factors. It is recommended that Awonto Konsult and also Brains and Hammers Construction invest more in information related to the fire resistance of plastic bottles used in construction because fire outbreaks pose great threats to buildings. Equally, wider empirical research on plastic bottle wastes, if undertaken, could support the development of policies for waste management, particularly in developing countries. This research has the potential to convert waste into wealth in a bid to minimising environmental impacts of disposed plastic bottles as well as contribute to sustainable materials, particularly for rural housing. Since this study was based on a survey, experimental studies of potentials driving the reuse of plastic bottles in housing construction will reveal results that could enable more sustainable housing construction in Nigeria.