Dissertations / Theses on the topic 'And Cluster Computing'

Los esfuerzos de esta tesis se centran en onstruir una máquina virtual sobre un sistema Cluster que proporcione la doble funcionalidad de ejecutar eficientemente tanto trabajos tradicionales (o locales) de estaciones de trabajo
así como aplicaciones distribuidas.
Para solucionar el problema, deben tenerse en cuenta dos importantes consideraciones:
* Como compartir y planificar los recursos de las diferentes estaciones de trabajo (especialmente la CPU) entre las aplicaciones locales y distribuidas.

* Como gestionar y controlar la totalidad del sistema para
conseguir ejecuciones eficientes de ambos tipos de aplicaciones.

Coscheduling es el principio básico usado para compartir
y planificar la CPU. Cosche-duling se basa en la reducción
del tiempo de espera de comunicación de aplicaciones distribuidas,
planificando simultáneamente todas (o un subconjunto de)
las tareas que la componen. Por lo tanto, mediante el uso
de técnicas de coscheduling, únicamente se puede incrementar
el rendimiento de aplicaciones distribuidas con comunicación
remota entre las tareas que la componen.

Las técnicas de Coscheduling se clasifican en dos grandes
grupos: control-explícito y control-implícito. Esta clasificación
se basa en la forma de coplanificar las tareas distribuidas.
En control-explícito, la coplanificación es realizada por
procesos y (o) procesadores especializados. En cambio, en
control-implícito, las técnicas de coscheduling se realizan
tomando decisiones de planificación localmente, dependiendo
de los eventos que ocurren en cada estación de trabajo.

En este proyecto se presentan dos mecanismos de coscheduling,
los cuales siguen las dos diferentes filosofías explicadas
anteriormente, control-implícito y control-explí-cito. También
proporcionan características adicionales incluyendo un buen
rendimiento en la ejecución de aplicaciones distribuidas,
ejecución simultánea de varias aplicaciones distribuidas,
bajo overhead y también bajo impacto en el rendimiento de
la carga local.

También se presenta un modelo de coscheduling, el cual proporciona
una base teórica para el desarrollo de nuevas técnicas de
control-implícito. La técnica de control-implícito propuesta
se basa en este modelo.

El buen comportamiento de las técnicas de coscheduling presentadas
en este trabajo se analiza en primer lugar por medio de
simulación. También se ha realizado un gran esfuerzo en
la implementación de estas técnicas de coscheduling en un
Cluster real. El estudio de los resultados obtenidos proporciona
una orientación importante para la investigación futura
en el campo de coscheduling.

En la experimentación en el Cluster real, se han utilizado
varios benchmarks distribuidos con diversos patrones de
comunicación de paso de mensajes: regulares e irregulares,
anillos lógicos, todos-a-todos, etc. También se han utilizado
benchmarks que medían diferentes primitivas de comunicación,
tales como barreras, enlaces uni y bidireccionales, etc.
El uso de esta amplia gama de aplicaciones distribuidas
ha servido para demostrar la aplicabilidad de las técnicas
de coscheduling en computación distribuida basados en Clusters.
Efforts of this Thesis are centered on constructing a Virtual
Machine over a Cluster system that provides the double functionality
of executing traditional workstation jobs as well as distributed
applications efficiently.

To solve the problem, two major considerations must be addressed:

* How share and schedule the workstation resources (especially
the CPU) between the local and distributed applications.

* How to manage and control the overall system for the efficient
execution of both application kinds.

Coscheduling is the base principle used for the sharing and
scheduling of the CPU. Coscheduling is based on reducing
the communication waiting time of distributed applications
by scheduling their forming tasks, or a subset of them at
the same time. Consequently, non-communicating distributed
applications (CPU bound ones) will not be favored by the
application of coscheduling. Only the performance of distributed
applications with remote communication can be increased
with coscheduling.

Coscheduling techniques follow two major trends: explicit
and implicit control. This classification is based on the
way the distributed tasks are managed and controlled. Basically,
in explicit-control, such work is carried out by specialized
processes and (or) processors. In contrast, in implicit-control,
coscheduling is performed by making local scheduling decisions
depending on the events occurring in each workstation.

Two coscheduling mechanisms which follow the two different
control trends are presented in this project. They also
provide additional features including usability, good performance
in the execution of distributed applications, simultaneous
execution of distributed applications, low overhead and
also low impact on local workload performance. The design
of the coscheduling techniques was mainly influenced by
the optimization of these features.

An implicit-control coscheduling model is also presented.
Some of the features it provides include collecting on-time
performance statistics and the usefulness as a basic scheme
for developing new coscheduling policies. The presented
implicit-control mechanism is based on this model.

The good scheduling behavior of the coscheduling models presented
is shown firstly by simulation, and their performance compared
with other coscheduling techniques in the literature. A
great effort is also made to implement the principal studied
coscheduling techniques in a real Cluster system. Thus,
it is possible to collect performance measurements of the
different coscheduling techniques and compare them in the
same environment. The study of the results obtained will
provide an important orientation for future research in
coscheduling because, to our knowledge, no similar work
(in the literature) has been done before.

Measurements in the real Cluster system were made by using
various distributed benchmarks with different message patterns:
regular and irregular communication patterns, token rings,
all-to-all and so on. Also, communication primitives such
as barriers and basic sending and receiving using one and
two directional links were separately measured. By using
this broad range of distributed applications, an accurate
analysis of the usefulness and applicability of the presented
coscheduling techniques in Cluster computing is performed.

The Large Hadron Collider (LHC), whose experiments include the Compact Muon Solenoid (CMS), produces over 30 million gigabytes of data annually, and implements a distributed computing architecture—a tiered hierarchy, from Tier-0 through Tier-3—in order to process and store all of this data. Out of all of the computing tiers, Tier-3 clusters allow scientists the most freedom and flexibility to perform their analyses of LHC data. Tier-3 clusters also provide local services such as login and storage services, provide a means to locally host and analyze LHC data, and allow both remote and local users to submit grid-based jobs. Using the Rocks cluster distribution software version 6.1.1, along with the Open Science Grid (OSG) roll version 3.2.35, a grid-enabled CMS Tier-3 computing cluster was deployed at Florida International University’s Modesto A. Maidique campus. Validation metric results from Ganglia, MyOSG, and CMS Dashboard verified a successful deployment.

Traditional cluster computing systems such as the supercomputers are equipped with specially designed high-performance hardware, which escalates the manufacturing cost and the energy cost of those systems. Due to such drawbacks and the diversified demand in computation, two new types of clusters are developed: the GPU clusters and the Hadoop clusters. The GPU cluster combines traditional CPU-only computing cluster with general purpose GPUs to accelerate the applications. Thanks to the massively-parallel architecture of the GPU, this type of system can deliver much higher performance-per-watt than the traditional computing clusters. The Hadoop cluster is another popular type of cluster computing system. It uses inexpensive off-the-shelf component and standard Ethernet to minimize manufacturing cost. The Hadoop systems are widely used throughout the industry. Alongside with the lowered cost, these new systems also bring their unique challenges. According to our study, the GPU clusters are prone to severe under-utilization due to the heterogeneous nature of its computation resources, and the Hadoop clusters are vulnerable to network congestion due to its limited network resources. In this research, we are trying to improve the throughput of these novel cluster computing systems by increasing the workload parallelism and network I/O parallelism.

L'Internet of Things è un concetto che è stato ora adottato in modo pervasivo per descrivere un vasto insieme di dispositivi connessi attraverso Internet. Comunemente, i sistemi IoT vengono creati con un approccio bottom-up e si concentrano principalmente sul singolo dispositivo, il quale è visto come la basilare unità programmabile. Da questo metodo può emergere un comportamento comune trovato in molti sistemi esistenti che deriva dall'interazione di singoli dispositivi. Tuttavia, questo crea un'applicazione distribuita spesso dove i componenti sono strettamente legati tra di loro. Quando tali applicazioni crescono in complessità, tendono a soffrire di problemi di progettazione, mancanza di modularità e riusabilità, difficoltà di implementazione e problemi di test e manutenzione. L'Aggregate Programming fornisce un approccio top-down a questi sistemi, in cui l'unità di calcolo di base è un'aggregazione anziché un singolo dispositivo. Questa tesi consiste nella progettazione e nella distribuzione di una piattaforma, basata su tecnologie open-source, per supportare l'Aggregate Computing nel cloud, in cui i dispositivi saranno in grado di scegliere dinamicamente se il calcolo si trova su se stessi o nel cloud. Anche se Aggregate Computing è intrinsecamente progettato per un calcolo distribuito, il Cloud Computing introduce un'alternativa scalabile, affidabile e altamente disponibile come strategia di esecuzione. Quest'opera descrive come sfruttare una Reactive Platform per creare un'applicazione scalabile nel cloud. Dopo che la struttura, l'interazione e il comportamento dell'applicazione sono stati progettati, viene descritto come la distribuzione dei suoi componenti viene effettuata attraverso un approccio di containerizzazione con Kubernetes come orchestratore per gestire lo stato desiderato del sistema con una strategia di Continuous Delivery.

One of the major challenges in the High Performance Computing (HPC) clusters, Data Centers, and Cloud Computing paradigms is intelligent power management to improve energy efficiency. The key contribution of the presented work is the modeling of a Power Aware Job Scheduler (PAJS) for HPC clusters, such that the: (a) threshold voltage is adjusted judiciously to achieve energy efficiency and (b) response time is minimized by scaling the supply voltage. The key novelty in our work is utilization of the Dynamic Threshold-Voltage Scaling (DTVS) for the reduction of cumulative power utilized by each node in the cluster. Moreover, to enhance the performance of the resource scheduling strategies in first part of the work, independent tasks within a job are scheduled to most suitable Computing Nodes (CNs). First, our research analyzes and compares eight scheduling techniques in terms of energy consumption and makespan. Primarily, the most suitable Dynamic Voltage Scaling (DVS) level adhering to the deadline is identified for each of the CNs by the scheduling heuristics. Afterwards, the DTVS is employed to scale down the static, as well as dynamic power by regulating the supply and bias voltages. Finally, the per node threshold scaling is used attain power saving. Our simulation results affirm that the proposed methodology significantly reduces the energy consumption using the DTVS. The work is further extended and the effect of task consolidation is studied and analyzed. By consolidating the tasks on a fewer number of servers the overall power consumed can be significantly reduced. The tasks are first allocated to suitable servers until all the tasks are exhausted. The idle servers are then turned off by using DTVS. The Virtual Machine (VM) monitor checks for under-utilized, partially filled, over-utilized, and empty servers. The VM monitor then migrates the tasks to suitable servers for execution if a set of conditions is met. By this way, many servers those were under-utilized get free and are turned off by using DTVS to save power. Simulations results confirm our study and a substantial reduction in the overall power consumption of the cloud data center is observed.

Modern applications are increasingly backed by large-scale data centres. Systems software in these data centre environments, however, faces substantial challenges: the lack of uniform resource abstractions makes sharing and resource management inefficient, infrastructure software lacks end-to-end access control mechanisms, and work placement ignores the effects of hardware heterogeneity and workload interference. In this dissertation, I argue that uniform, clean-slate operating system (OS) abstractions designed to support distributed systems can make data centres more efficient and secure. I present a novel distributed operating system for data centres, focusing on two OS components: the abstractions for resource naming, management and protection, and the scheduling of work to compute resources. First, I introduce a reference model for a decentralised, distributed data centre OS, based on pervasive distributed objects and inspired by concepts in classic 1980s distributed OSes. Translucent abstractions free users from having to understand implementation details, but enable introspection for performance optimisation. Fine-grained access control is supported by combining storable, communicable identifier capabilities, and context-dependent, ephemeral handle capabilities. Finally, multi-phase I/O requests implement optimistically concurrent access to objects while supporting diverse application-level consistency policies. Second, I present the DIOS operating system, an implementation of my model as an extension to Linux. The DIOS system call API is centred around distributed objects, globally resolvable names, and translucent references that carry context-sensitive object meta-data. I illustrate how these concepts support distributed applications, and evaluate the performance of DIOS in microbenchmarks and a data-intensive MapReduce application. I find that it offers improved, finegrained isolation of resources, while permitting flexible sharing. Third, I present the Firmament cluster scheduler, which generalises prior work on scheduling via minimum-cost flow optimisation. Firmament can flexibly express many scheduling policies using pluggable cost models; it makes high-quality placement decisions based on fine-grained information about tasks and resources; and it scales the flow-based scheduling approach to very large clusters. In two case studies, I show that Firmament supports policies that reduce colocation interference between tasks and that it successfully exploits flexibility in the workload to improve the energy efficiency of a heterogeneous cluster. Moreover, my evaluation shows that Firmament scales the minimum-cost flow optimisation to clusters of tens of thousands of machines while still making sub-second placement decisions.

Nell'ambito del calcolo distribuito, Microsoft HPC Pack offre un importante strumento per la gestione delle risorse computazionali in maniera efficiente orchestrando lo scheduling delle unità di lavoro all’interno dell’infrastruttura. HPC supporta nativamente l’integrazione sul cloud di Microsoft Azure tramite strategie di networking e virtualizzazione ben definite. Dopo una breve presentazione di Prometeia, presso cui il progetto ha avuto luogo, sono presentate nel dettaglio le tecnologie Microsoft utilizzate nel percorso. Segue una presentazione del progetto, che si compone di due passi: il primo consiste nella realizzazione dell’infrastruttura applicativa e HPC sul cloud Azure tramite template automatizzato (realizzazione di macchine virtuali, rete virtuale, installazione dei servizi e di HPC); il secondo passo è la realizzazione di un’applicazione che consenta, in base alle esigenze dell’utente, di creare ed eliminare risorse di calcolo dall’infrastruttura tramite comandi appositamente implementati. Questa soluzione apporta vantaggi di tempo ed economici sia rispetto agli scenari on-premise, in quanto non è più richiesto l’acquisto, la manutenzione e l’aggiornamento di server fisici, sia rispetto a soluzioni cloud più statiche, in cui la presenza di risorse di calcolo inattive per lunghi periodi di tempo producono costi molto più elevati. La parte finale dell’elaborato si concentra sull’analisi dei vantaggi economici che la soluzione presentata apporta, mostrando nel dettaglio le differenze tra i costi delle varie soluzioni offerte da Azure.

Cloud computing has become a powerful enabler for many IT services and new technolo-gies. It provides access to an unprecedented amount of resources in a fine-grained andon-demand manner. To deliver such a service, cloud providers should be able to efficientlyand reliably manage their available resources. This becomes a challenge for the manage-ment system as it should handle a large number of heterogeneous resources under diverseworkloads with fluctuations. In addition, it should also satisfy multiple operational require-ments and management objectives in large scale data centers.Autonomic computing techniques can be used to tackle cloud resource managementproblems. An autonomic system comprises of a number of autonomic elements, which arecapable of automatically organizing and managing themselves rather than being managedby external controllers. Therefore, they are well suited for decentralized control, as theydo not rely on a centrally managed state. A decentralized autonomic system benefits fromparallelization of control, faster decisions and better scalability. They are also more reliableas a failure of one will not affect the operation of the others, while there is also a lower riskof having faulty behaviors on all the elements, all at once. All these features are essentialrequirements of an effective cloud resource management.This thesis investigates algorithms, models, and techniques to autonomously managejobs, services, and virtual resources in a cloud data center. We introduce a decentralizedresource management framework, that automates resource allocation optimization and ser-vice consolidation, reliably schedules jobs considering probabilistic failures, and dynam-icly scales and repacks services to achieve cost efficiency.As part of the framework, we introduce a decentralized scheduler that provides andmaintains durable allocations with low maintenance costs for data centers with dynamicworkloads. The scheduler assigns resources in response to virtual machine requests andmaintains the packing efficiency while taking into account migration costs, topologicalconstraints, and the risk of resource contention, as well as fluctuations of the backgroundload.We also introduce a scheduling algorithm that considers probabilistic failures as part ofthe planning for scheduling. The aim of the algorithm is to achieve an overall job reliabil-ity, in presence of correlated failures in a data center. To do so, we study the impacts ofstochastic and correlated failures on job reliability in a virtual data center. We specificallyfocus on correlated failures caused by power outages or failure of network components onjobs running large number of replicas of identical tasks.Additionally, we investigate the trade-offs between vertical and horizontal scaling. Theresult of the investigations is used to introduce a repacking technique to automatically man-age the capacity required by an elastic service. The repacking technique combines thebenefits of both scaling strategies to improve its cost-efficiency.
Datormoln har kommit att bli kraftfulla möjliggörare för många nya IT-tjänster. De ger tillgång till mycket storskaliga datorresurser på ett finkornigt och omedelbart sätt. För att tillhandahålla sådana resurser krävs att de underliggande datorcentren kan hantera sina resurser på ett tillförlitligt och effektivt sätt. Frågan hur man ska designa deras resurshanteringssystem är en stor utmaning då de ska kunna hantera mycket stora mängder heterogena resurser som i sin tur ska klara av vitt skilda typer av belastning, ofta med väldigt stora variationer över tid. Därtill ska de typiskt kunna möta en mängd olika krav och målsättningar för hur resurserna ska nyttjas. Autonomiska system kan med fördel användas för att realisera sådana system. Ett autonomt system innehåller ett antal autonoma element som automatiskt kan organisera och hantera sig själva utan stöd av externa regulatorer. Förmågan att hantera sig själva gör dem mycket lämpliga som komponenter i distribuerade system, vilka i sin tur kan bidra till snabbare beslutsprocesser, bättre skalbarhet och högre feltolerans. Denna avhandling fokuserar på algoritmer, modeller och tekniker för autonom hantering av jobb och virtuella resurser i datacenter. Vi introducerar ett decentraliserat resurshanteringssystem som automatiserar resursallokering och konsolidering, schedulerar jobb tillförlitligt med hänsyn till korrelerade fel, samt skalar resurser dynamiskt för att uppnå kostnadseffektivitet. Som en del av detta ramverk introducerar vi en decentraliserad schedulerare som allokerar resurser med hänsyn till att tagna beslut ska vara bra för lång tid och ge låga resurshanteringskostnader för datacenter med dynamisk belastning. Scheduleraren allokerar virtuella maskiner utifrån aktuell belastning och upprätthåller ett effektivt nyttjande av underliggande servrar genom att ta hänsyn till migrationskostnader, topologiska bivillkor och risk för överutnyttjande. Vi introducerar också en resursallokeringsalgoritm som tar hänsyn till korrelerade fel som ett led i planeringen. Avsikten är att kunna uppnå specificerade tillgänglighetskrav för enskilda tjänster trots uppkomst av korrelerade fel. Vi fokuserar främst på korrelerade fel som härrör från problem med elförsörjning och från felande nätverkskomponenter samt deras påverkan på jobb bestående av många identiska del-jobb. Slutligen studerar vi även hur man bäst ska kombinera horisontell och vertikal skalning av resurser. Resultatet är en process som ökar kostnadseffektivitet genom att kombinera de två metoderna och därtill emellanåt förändra fördelning av storlekar på virtuella maskiner.

Cluster robust models are a kind of statistical models that attempt to estimate parameters considering potential heterogeneity in treatment effects. Absent heterogeneity in treatment effects, the partial and average treatment effect are the same. When heterogeneity in treatment effects occurs, the average treatment effect is a function of the various partial treatment effects and the composition of the population of interest. The first chapter explores the performance of common estimators as a function of the presence of heterogeneity in treatment effects and other characteristics that may influence their performance for estimating average treatment effects. The second chapter examines various approaches to evaluating and improving cluster structures as a way to obtain cluster-robust models. Both chapters are intended to be useful to practitioners as a how-to guide to examine and think about their applications and relevant factors. Empirical examples are provided to illustrate theoretical results, showcase potential tools, and communicate a suggested thought process. The third chapter relates to an open-source statistical software package for the Julia language. The content includes a description for the software functionality and technical elements. In addition, it features a critique and suggestions for statistical software development and the Julia ecosystem. These comments come from my experience throughout the development process of the package and related activities as an open-source and professional software developer. One goal of the paper is to make econometrics more accessible not only through accessibility to functionality, but understanding of the code, mathematics, and transparency in implementations.

The Transmission Line Matrix (TLM) method is an existing and established mathematical method for conducting computational electromagnetic (CEM) simulations. TLM models Maxwell s equations by discretising the contiguous nature of an environment and its contents into individual small-scale elements and it is a computationally intensive process. This thesis focusses on parallel processing optimisations to the TLM method when considering the opposing ends of the contemporary computing hardware spectrum, namely large-scale computing systems versus small-scale mobile computing devices. Theoretical aspects covered in this thesis are: The historical development and derivation of the TLM method. A discrete random variable (DRV) for rain-drop diameter,allowing generation of a rain-field with raindrops adhering to a Gaussian size distribution, as a case study for a 3-D TLM implementation. Investigations into parallel computing strategies for accelerating TLM on large and small-scale computing platforms. Implementation aspects covered in this thesis are: A script for modelling rain-fields using free-to-use modelling software. The first known implementation of 2-D TLM on mobile computing devices. A 3-D TLM implementation designed for simulating the effects of rain-fields on extremely high frequency (EHF) band signals. By optimising both TLM solver implementations for their respective platforms, new opportunities present themselves. Rain-field simulations containing individual rain-drop geometry can be simulated, which was previously impractical due to the lengthy computation times required. Also, computationally time-intensive methods such as TLM were previously impractical on mobile computing devices. Contemporary hardware features on these devices now provide the opportunity for CEM simulations at speeds that are acceptable to end users, as well as providing a new avenue for educating relevant user cohorts via dynamic presentations of EM phenomena.

As the biological databases are increasing rapidly, there is a challenge for both Biologists and Computer Scientists to develop algorithms and databases to manage the increasing data. There are many algorithms developed to align the sequences stored in biological databases - some take time to process the data while others are inefficient to produce reasonable results. As more data is generated, and time consuming algorithms are developed to handle them, there is a need for specialized computers to handle the computations. Researchers are typically limited by the computational power of their computers. High Performance Computing (HPC) field addresses this challenge and can be used in a cost-effective manner where there is no need for expensive equipment, instead old computers can be used together to form a powerful system. This is the premise of this research, wherein the setup of a low-cost Beowulf cluster is explored, with the subsequent evaluation of its performance for processing sequent alignment algorithms. A mixed method methodology is used in this dissertation, which consists of literature study, theoretical and practise based system. This mixed method methodology also have a proof and concept where the Beowulf cluster is designed and implemented to perform the sequence alignment algorithms and also the performance test. This dissertation firstly gives an overview of sequence alignment algorithms that are already developed and also highlights their timeline. A presentation of the design and implementation of the Beowulf Cluster is highlighted and this is followed by the experiments on the baseline performance of the cluster. A detailed timeline of the sequence alignment algorithms is given and also the comparison between ClustalW-MPI and T-Coffee (Tree-based Consistency Objective Function For alignment Evaluation) algorithm is presented as part of the findings in the research study. The efficiency of the cluster was observed to be 19.8%, this percentage is unexpected because the predicted efficiency is 83.3%, which is found in the theoretical cluster calculator. The theoretical performance of the cluster showed a high performance as compared with the experimental performance, this is attributable to the slow network, which was 100Mbps, low processor speed of 2.50 GHz, and low memory of 2 Gigabytes.

L'algorithme Branch-and-Bound (B&B) est une méthode de recherche arborescente fréquemment utilisé pour la résolution exacte de problèmes d'optimisation combinatoire (POC). Néanmoins, seules des petites instances peuvent être effectivement résolues sur une machine séquentielle, le nombre de sous-problèmes à évaluer étant souvent très grand. Visant la resolution de POC de grande taille, nous réexaminons la conception et l'implémentation d'algorithmes B&B massivement parallèles sur de larges plateformes hétérogènes de calcul, intégrant des processeurs multi-coeurs, many-cores et et processeurs graphiques (GPUs). Pour une représentation compacte en mémoire des sous-problèmes une structure de données originale (IVM), dédiée aux problèmes de permutation est utilisée. En raison de la forte irrégularité de l'arbre de recherche, l'équilibrage de charge dynamique entre processus d'exploration parallèles occupe une place centrale dans cette thèse. Basés sur un encodage compact de l'espace de recherche sous forme d'intervalles, des stratégies de vol de tâches sont proposées pour processeurs multi-core et GPU, ainsi une approche hiérarchique pour l'équilibrage de charge dans les systèmes multi-GPU et multi-CPU à mémoire distribuée. Trois problèmes d'optimisation définis sur l'ensemble des permutations, le problème d'ordonnancement Flow-Shop (FSP), d'affectation quadratique (QAP) et le problème des n-dames sont utilisés comme cas d'étude. La resolution en 9 heures d'une instance du FSP dont le temps de résolution séquentiel est estimé à 22 ans demontre la capacité de passage à l'échelle des algorithmes proposés sur une grappe de calcul composé de 36 GPUs
Branch-and-Bound (B&B) is a frequently used tree-search exploratory method for the exact resolution of combinatorial optimization problems (COPs). However, in practice, only small problem instances can be solved on a sequential computer, as B&B generates often generates a huge amount of subproblems to be evaluated. In order to solve large COPs, we revisit the design and implementation of massively parallel B&B on top of large heterogeneous clusters, integrating multi-core CPUs, many-core processors and GPUs. For the efficient storage and management of subproblems an original data structure (IVM) dedicated to permutation problems is used. Because of the highly irregular and unpredictable shape of the B&B tree, dynamic load balancing between parallel exploration processes is one of the main issues addressed in this thesis. Based on a compact encoding of the search space in the form of intervals, work stealing strategies for multi-core and GPU are proposed, as well as hierarchical approaches for load balancing in distributed memory multi-CPU/multi-GPU systems. Three permutation problems, the Flowshop Scheduling Problem (FSP), the Quadratic Assignment Problem (QAP) and the n-Queens puzzle problem are used as test-cases. The resolution, in 9 hours, of a FSP instance with an estimated sequential execution time of 22 years demonstrates the scalability of the proposed algorithms on a cluster composed of 36 GPUs

PROCESSAMENTO DE DADOS DO ESTADO DO PARÁ
O objetivo deste trabalho é estudar a integração de mecanismos de reserva de recursos computacionais em infra-estruturas de software para aglomerados de computadores. Para realizar esse estudo, foi utilizado o framework CSBase, que é uma infra-estrutura de software concebida com o intuito de dar apoio à implementação e integração de aplicações científicas em ambientes distribuídos e heterogêneos. O CSBase oferece suporte à execução de aplicações em ambientes distribuídos e ao gerenciamento de usuários e de recursos computacionais, tais como computadores, arquivos de dados e aplicações. Entretanto, as primeiras aplicações desenvolvidas com o CSBase já demonstraram que são necessários mecanismos que permitam um melhor gerenciamento e controle dos recursos computacionais disponíveis em ambientes distribuídos, e especialmente em aglomerados de computadores dedicados à execução de aplicações de alto desempenho. Neste trabalho, apresentamos uma extensão ao framework CSBase que possibilita a reserva de processador para aplicações de usuários do sistema, e assim permitindo um gerenciamento mais eficiente dos recursos computacionais disponíveis. Essa extensão também garante que serão efetuadas as adaptações necessárias para acomodar eventuais variações no perfil de uso do processador por parte das aplicações. Como resultado dessa extensão, obteve-se a integração entre a monitoração de recursos distribuídos, a iniciação remota de aplicações, e um mecanismo de reserva de processador que proporcionou uma melhor utilização das máquinas disponíveis.
The goal of this work is to study the integration of resource reservation mechanisms with software infrastructures for cluster computing. To perform this study, we used the CSBase framework, which is a software infrastructure conceived to support the implementation and integration of scientific applications in heterogeneous and distributed environments. CSBase offers support for application execution in distributed environments, as well as support for management of users and computational resources, such as computers, data files and applications. However, the first applications developed with CSBase showed that it requires mechanisms to allow a better management of resources available in distributed environments, and especially in clusters of computers dedicated to execute high performance applications. In this work, we present an extension to CSBase that provides the reservation of processor time to user applications, thus allowing a more efficient resource management. This extension also guarantees that all required adaptations will be performed to accommodate variations in the applications´ processor usage profile. As a result of this extension, we achieved the integration of mechanisms for distributed resource monitoring, remote application execution, and processor reservation, providing a better utilization among the available machines.

Parallel computing techniques have become more important especially now that we have effectively reached the limit on individual processor speeds due to unacceptable levels of heat generation. Multi-core processors are already the norm and will continue to rise in terms of number of cores in the near future. However clusters of machines remain the next major step up in system performance effectively allowing vast numbers of cores to be devoted to any given problem. It is in that context that this Professional Doctorate thesis and Portfolio exists. Most parallel or cluster based software is custom built for an application using techniques such as OpenMP or MPI. But what if the capability of writing such software does not exist, what if the very act of writing a new piece of software compromises the integrity of an industry standard piece of software currently being used in a research project? The first outcome was to explore how grid/cluster computing teaching and learning facilities could be made accessible to students and teaching staff alike within the Department of Computing, Engineering & Technology in order to enhance the student experience. This was achieved through the development of VCNet, a virtual technology cluster solution, based on the design of the University of Sunderland Cluster Computer (USCC) and capable of running behind a dual boot arrangement on standard teaching machines. The second outcome of this Professional Doctorate was to produce a framework for efficient cluster computing services in a collaborative university environment. Although small by national and international standards, the USCC, with its forty machines and 160 cores, packs a mighty punch in computing terms. Through the work of this doctorate, ‘supercomputer class’ performance has been successfully used in cross- disciplinary research through the development and use of the Application Framework for Computational Chemistry (AFCC). In addition, I will also discuss the contribution this doctorate has made within the context of my community of practice by enhancing both my teaching and learning contribution as well as cross-disciplinary research and application.

The discipline of remote sensing is concerned with observing the earth's suface using different portions of the electro-magnetic spectrum. Earth orbiting satellites will soon collect terabytes of data per day with increased accuracy. Automated parallel algorithms are essential to quickly process this large amount of data. Data parallel languages have been used effectively for the diverse algorithms found in such systems. With improved network technology, it is now feasible to build data parallel supercomputers using traditional RISC-based workstations connected by a high-speed network. This dissertation presents Cluster-C$\sp\*$, an architecture that implements the data parallel language C$\sp\*$ on a cluster of workstations. A specialized language run-time system and network protocols effectively integrate the cluster components to form a dedicated, efficient multiprocessor environment. A series of analytic, empirical, and simulation techniques quantify the cluster's performance. A nine program test suite, derived from remote sensing and image understanding algorithms, provides a basis for cluster evaluation. An in-depth look at the communication behavior of the test suite supports prediction of algorithm performance on the cluster, as well as important architectural design insights. The test suite is executed on a cluster of 8 HP 720 workstations and a 32-node (128 vector unit) CM-5 to establish a concrete performance baseline. The result is that, under some conditions, the cluster is faster on an absolute scale, and that on a relative, per-node scale, the cluster delivers superior performance in all cases. Finally, a trace-driven simulator, based on these empirical measurements, supports predictions of the cluster's scalability and performance when equipped with next generation workstation and network technologies. Simulations show that Gigabit networks have the necessary bandwidth to build clusters with hundreds of nodes. Furthermore, even a modestly enhanced cluster, consisting of 16 high-end workstations connected by a 600 Mbps token ring will out-perform a 32-node CM-5 in all but a few cases.

Nowadays, although MapReduce is applied to the parallel processing on big data, it has some limitations: for instance, lack of generic but efficient and richly functional primitive parallel methods, incapability of entering multiple input parameters on the entry of parallel methods, and inefficiency in the way of handling iterative algorithms. Spark and Stratosphere are developed to deal with (partly) the shortcoming of MapReduce. The goal of this thesis is to evaluate Spark and Stratosphere both from the point of view of theoretical programming model and practical execution on specified application algorithms. In the introductory section of comparative programming models, we mainly explore and compare the features of Spark and Stratosphere that overcome the limitation of MapReduce. After the comparison in theoretical programming model, we further evaluate their practical performance by running three different classes of applications and assessing usage of computing resources and execution time. It is concluded that Spark has promising features for iterative algorithms in theory but it may not achieve the expected performance improvement to run iterative applications if the amount of memory used for cached operations is close to the actual available memory in the cluster environment. In that case, the reason for the poor results in performance is because larger amount of memory participates in the caching operation and in turn, only a small amount memory is available for computing operations of actual algorithms. Stratosphere shows favorable characteristics as a general parallel computing framework, but it has no support for iterative algorithms and spends more computing resources than Spark for the same amount of work. In another aspect, applications based on Stratosphere can achieve benefits by manually setting compiler hints when developing the code, whereas Spark has no corresponding functionality.

Scientific applications generally require large computational requirements, memory and data management for their execution. Such applications have traditionally used high-performance resources, such as shared memory supercomputers, clusters of PCs with distributed memory, or resources from Grid infrastructures on which the application needs to be adapted to run successfully. In recent years, the advent of virtualization techniques, together with the emergence of Cloud Computing, has caused a major shift in the way these applications are executed. However, the execution management of scientific applications on high performance elastic platforms is not a trivial task. In this doctoral thesis, Elastic Cloud Computing Cluster (EC3) has been developed. EC3 is an open-source tool able to execute high performance scientific applications by creating self-managed cost-efficient virtual hybrid elastic clusters on top of IaaS Clouds. These self-managed clusters have the capability to adapt the size of the cluster, i.e. the number of nodes, to the workload, thus creating the illusion of a real cluster without requiring an investment beyond the actual usage. They can be fully customized and migrated from one provider to another, in an automatically and transparent process for the users and jobs running in the cluster. EC3 can also deploy hybrid clusters across on-premises and public Cloud resources, where on-premises resources are supplemented with public Cloud resources to accelerate the execution process. Different instance types and the use of spot instances combined with on-demand resources are also cluster configurations supported by EC3. Moreover, using spot instances, together with checkpointing techniques, the tool can significantly reduce the total cost of executions while introducing automatic fault tolerance. EC3 is conceived to facilitate the use of virtual clusters to users, that might not have an extensive knowledge about these technologies, but they can benefit from them. Thus, the tool offers two different interfaces for its users, a web interface where EC3 is exposed as a service for non-experienced users and a powerful command line interface. Moreover, this thesis explores the field of light-weight virtualization using containers as an alternative to the traditional virtualization solution based on virtual machines. This study analyzes the suitable scenario for the use of containers and proposes an architecture for the deployment of elastic virtual clusters based on this technology. Finally, to demonstrate the functionality and advantages of the tools developed during this thesis, this document includes several use cases covering different scenarios and fields of knowledge, such as structural analysis of buildings, astrophysics or biodiversity.
Las aplicaciones científicas generalmente precisan grandes requisitos de cómputo, memoria y gestión de datos para su ejecución. Este tipo de aplicaciones tradicionalmente ha empleado recursos de altas prestaciones, como supercomputadores de memoria compartida, clústers de PCs de memoria distribuida, o recursos provenientes de infraestructuras Grid, sobre los que se adaptaba la aplicación para que se ejecutara satisfactoriamente. El auge que han tenido las técnicas de virtualización en los últimos años, propiciando la aparición de la computación en la nube (Cloud Computing), ha provocado un importante cambio en la forma de ejecutar este tipo de aplicaciones. Sin embargo, la gestión de la ejecución de aplicaciones científicas sobre plataformas de computación elásticas de altas prestaciones no es una tarea trivial. En esta tesis doctoral se ha desarrollado Elastic Cloud Computing Cluster (EC3), una herramienta de código abierto capaz de llevar a cabo la ejecución de aplicaciones científicas de altas prestaciones creando para ello clústers virtuales, híbridos y elásticos, autogestionados y eficientes en cuanto a costes, sobre plataformas Cloud de tipo Infraestructura como Servicio (IaaS). Estos clústers autogestionados tienen la capacidad de adaptar su tamaño, es decir, el número de nodos, a la carga de trabajo, creando así la ilusión de un clúster real sin requerir una inversión por encima del uso actual. Además, son completamente configurables y pueden ser migrados de un proveedor a otro de manera automática y transparente a los usuarios y trabajos en ejecución en el cluster. EC3 también permite desplegar clústers híbridos sobre recursos Cloud públicos y privados, donde los recursos privados son complementados con recursos Cloud públicos para acelerar el proceso de ejecución. Otras configuraciones híbridas, como el empleo de diferentes tipos de instancias y el uso de instancias puntuales combinado con instancias bajo demanda son también soportadas por EC3. Además, el uso de instancias puntuales junto con técnicas de checkpointing permite a EC3 reducir significantemente el coste total de las ejecuciones a la vez que proporciona tolerancia a fallos. EC3 está concebido para facilitar el uso de clústers virtuales a los usuarios, que, aunque no tengan un conocimiento extenso sobre este tipo de tecnologías, pueden beneficiarse fácilmente de ellas. Por ello, la herramienta ofrece dos interfaces diferentes a sus usuarios, una interfaz web donde se expone EC3 como servicio para usuarios no experimentados y una potente interfaz de línea de comandos. Además, esta tesis doctoral se adentra en el campo de la virtualización ligera, mediante el uso de contenedores como alternativa a la solución tradicional de virtualización basada en máquinas virtuales. Este estudio analiza el escenario propicio para el uso de contenedores y propone una arquitectura para el despliegue de clusters virtuales elásticos basados en esta tecnología. Finalmente, para demostrar la funcionalidad y ventajas de las herramientas desarrolladas durante esta tesis, esta memoria recoge varios casos de uso que abarcan diferentes escenarios y campos de conocimiento, como estudios estructurales de edificios, astrofísica o biodiversidad.
Les aplicacions científiques generalment precisen grans requisits de còmput, de memòria i de gestió de dades per a la seua execució. Este tipus d'aplicacions tradicionalment hi ha empleat recursos d'altes prestacions, com supercomputadors de memòria compartida, clústers de PCs de memòria distribuïda, o recursos provinents d'infraestructures Grid, sobre els quals s'adaptava l'aplicació perquè s'executara satisfactòriament. L'auge que han tingut les tècniques de virtualitzaciò en els últims anys, propiciant l'aparició de la computació en el núvol (Cloud Computing), ha provocat un important canvi en la forma d'executar este tipus d'aplicacions. No obstant això, la gestió de l'execució d'aplicacions científiques sobre plataformes de computació elàstiques d'altes prestacions no és una tasca trivial. En esta tesi doctoral s'ha desenvolupat Elastic Cloud Computing Cluster (EC3), una ferramenta de codi lliure capaç de dur a terme l'execució d'aplicacions científiques d'altes prestacions creant per a això clústers virtuals, híbrids i elàstics, autogestionats i eficients quant a costos, sobre plataformes Cloud de tipus Infraestructura com a Servici (IaaS). Estos clústers autogestionats tenen la capacitat d'adaptar la seua grandària, es dir, el nombre de nodes, a la càrrega de treball, creant així la il·lusió d'un cluster real sense requerir una inversió per damunt de l'ús actual. A més, són completament configurables i poden ser migrats d'un proveïdor a un altre de forma automàtica i transparent als usuaris i treballs en execució en el cluster. EC3 també permet desplegar clústers híbrids sobre recursos Cloud públics i privats, on els recursos privats són complementats amb recursos Cloud públics per a accelerar el procés d'execució. Altres configuracions híbrides, com l'us de diferents tipus d'instàncies i l'ús d'instàncies puntuals combinat amb instàncies baix demanda són també suportades per EC3. A més, l'ús d'instàncies puntuals junt amb tècniques de checkpointing permet a EC3 reduir significantment el cost total de les execucions al mateix temps que proporciona tolerància a fallades. EC3e stà concebut per a facilitar l'ús de clústers virtuals als usuaris, que, encara que no tinguen un coneixement extensiu sobre este tipus de tecnologies, poden beneficiar-se fàcilment d'elles. Per això, la ferramenta oferix dos interfícies diferents dels seus usuaris, una interfície web on s'exposa EC3 com a servici per a usuaris no experimentats i una potent interfície de línia d'ordres. A més, esta tesi doctoral s'endinsa en el camp de la virtualitzaciò lleugera, per mitjà de l'ús de contenidors com a alternativa a la solució tradicional de virtualitzaciò basada en màquines virtuals. Este estudi analitza l'escenari propici per a l'ús de contenidors i proposa una arquitectura per al desplegament de clusters virtuals elàstics basats en esta tecnologia. Finalment, per a demostrar la funcionalitat i avantatges de les ferramentes desenrotllades durant esta tesi, esta memòria arreplega diversos casos d'ús que comprenen diferents escenaris i camps de coneixement, com a estudis estructurals d'edificis, astrofísica o biodiversitat.
Calatrava Arroyo, A. (2016). High Performance Scientific Computing over Hybrid Cloud Platforms [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/75265
TESIS

Cluster-weighted modeling (CWM) is a mixture approach to modeling the joint probability of data coming from a heterogeneous population. In this thesis first we investigate statistical properties of CWM from both theoretical and numerical point of view for both Gaussian and Student-t CWM. Then we introduce a novel family of twelve mixture models, all nested in the linear-t cluster weighted model (CWM). This family of models provides a unified framework that also includes the linear Gaussian CWM as a special case. Parameters estimation is carried out through algorithms based on maximum likelihood estimation and both the BIC and ICL are used for model selection. Finally, based on these algorithms, a software package for the R language has been implemented.

The advances in computing and communication technologies and software tools have resulted in an explosive growth in networked applications and information services that cover all aspects of our life. These services and applications are inherently complex, dynamic and heterogeneous. In a similar way, the underlying information infrastructure, e.g. the Internet, is large, complex, heterogeneous and dynamic, globally aggregating large numbers of independent computing and communication resources. The combination of the two results in application development and management complexities that break current computing paradigms, which are based on static behaviors. As a result, applications, programming environments and information infrastructures are rapidly becoming fragile, unmanageable and insecure. This has led researchers to consider alternative programming paradigms and management techniques that are based on strategies used by biological systems. Autonomic programming paradigm is inspired by the human autonomic nervous system that handles complexity, uncertainties and abnormality. The overarching goal of the autonomic programming paradigm is to help building systems and applications capable of self-management. Firstly, we investigated the large-scale scientific computing applications which generally experience different execution phases at run time and each phase has different computational, communication and storage requirements as well as different physical characteristics. In this dissertation, we present Physics Aware Optimization (PAO) paradigm that enables programmers to identify the appropriate solution methods to exploit the heterogeneity and the dynamism of the application execution states. We implement a Physics Aware Optimization Manager to exploit the PAO paradigm. On the other hand we present a self configuration paradigm based on the principles of autonomic computing that can handle efficiently complexity, dynamism and uncertainty in configuring server and networked systems and their applications. Our approach is based on making any resource/application to operate as an Autonomic Component (that means it can be self-managed component) by using our autonomic programming paradigm. Our POA technique for medical application yielded about 3X improvement of performance with 98.3% simulation accuracy compared to traditional techniques for performance optimization. Also, our Self-configuration management for power and performance management in GPU cluster demonstrated 53.7% power savings for CUDAworkload while maintaining the cluster performance within given acceptable thresholds.

The results of the survey about usage of scalable environment, peak workloads management and automatic scaling configuration among IT companies are presented and discussed in this paper. The hypothesis that most companies use automatic scaling based on static thresholds is checked. The insight into the most popular setups of manual and automatic scalable systems on the market is given.

In recent years program transformation technology has matured into a practical solution for many software reengineering and migration tasks. FermaT, an industrial strength program transformation system, has demonstrated that legacy systems can be successfully transformed into efficient and maintainable structured C or COBOL code. Its core, a transformation engine, is based on mathematically proven program transformations and ensures that transformed programs are semantically equivalent to its original state. Its engine facilitates a Wide Spectrum Language (WSL), with low-level as well as high-level constructs, to capture as much information as possible during transformation steps. FermaT’s methodology and technique lack in provision of concurrent migration and analysis. This provision is crucial if the transformation process is to be further automated. As the constraint based program migration theory has demonstrated, it is inefficient and time consuming, trying to satisfy the enormous computation of the generated transformation sequence search-space and its constraints. With the objective to solve the above problems and to extend the operating range of the FermaT transformation system, this thesis proposes a Parallel Transformations Framework which makes parallel transformations processing within the FermaT environment not only possible but also beneficial for its migration process. During a migration process, many thousands of program transformations have to be applied. For example a 1 million line of assembler to C migration takes over 21 hours to be processed on a single PC. Various approaches of search, prediction techniques and a constraint-based approach to address the presented issues already exist but they solve them unsatisfactorily. To remedy this situation, this dissertation proposes a framework to extend transformation processing systems with parallel processing capabilities. The parallel system can analyse specified parallel transformation tasks and produce appropriate parallel transformations processing outlines. To underpin an automated objective, a formal language is introduced. This language can be utilised to describe and outline parallel transformation tasks whereas parallel processing constraints underpin the parallel objective. This thesis addresses and explains how transformation processing steps can be automatically parallelised within a reengineering domain. It presents search and prediction tactics within this field. The decomposition and parallelisation of transformation sequence search-spaces is outlined. At the end, the presented work is evaluated on practical case studies, to demonstrate different parallel transformations processing techniques and conclusions are drawn.

Thesis (M.S. in Information Technology Management)--Naval Postgraduate School, September 2007.
Thesis Advisor(s): Sanchez, Paul. "September 2007." Description based on title screen as viewed on October 23, 2007. Includes bibliographical references (p. 87-89). Also available in print.

In this body of work, we present the details of a novel method for passive resource discovery in cluster grid environments where resources constantly utilize inter-node communication. Our method offers the ability to non-intrusively identify resources that have available memory or CPU cycles; this is critical for lowering queue wait times in large cluster grid networks, and for memory-intensive cluster grid applica-tions such as Gaussian (computational chemistry package) and the Weather Research and Forecasting (WRF) modeling package. The benefits include: (1) low message complexity, (2) scalability, (3) load bal-ancing support, and (4) low maintainability. Using several test-beds (i.e., a small local test-bed and a 50-node Deterlab test-bed), we demonstrate the feasibility of our method with experiments utilizing TCP, UDP and ICMP network traffic. Using this technique, we observed a correlation between memory or CPU load and the timely response of network traffic. In such situations, we have observed that in highly utilized (due to multi-programming) nodes there will be numerous, active processes which require context switching or paging. The latency associated with numerous context switches or paging manifests as a de-lay signature within the packet transmission process. Our method detects this delay signature to determine the utilization of network resources. The aforementioned delay signature is the keystone that provides a correlation between network traffic and the internal state of the source node. We characterize this delay signature due to CPU utilization by (1) identifying the different types of assembly language instructions that source this delay and (2) describing how performance-enhancing techniques (e.g., instruction pipelin-ing, caching) impact this delay signature by using the LEON3, implemented as a 40 MHz development board. At the software level, results for medium sized networks show that our method can consistently and accurately identify nodes with available memory or CPU cycles (< 70% availability). At the hardware level, our results show that excessive context switching in active applications increases the average mem-ory access time, thus adding additional delay to the execution of LD instructions. Additionally, internal use of these instructions in heavily utilized situations to send network packets induces the delay signature into network traffic.

The era of big data has led to problems of unprecedented scale and complexity that are challenging the computing capability of conventional computer systems. One way to address the computational and communication challenges of such demanding applications is to incorporate the use of non-conventional hardware accelerators such as FPGAs into existing systems. By providing a mix of FPGAs and conventional CPUs as computing resources in a heterogeneous cluster, a distributed computing environment can be achieved to address the need of both compute-intensive and data-intensive applications. However, utilizing heterogeneous clusters requires application developers’ comprehensive knowledge on both hardware and software. In order to assist programmers to take advantage of the synergy between hardware and software easily, an easy-to-use framework for virtualizing the underlying FPGA computing resources of the heterogeneous cluster is motivated. In this work, a heterogeneous cluster consisting of both FPGAs and CPUs was built and a framework for managing multiple FPGAs across the cluster was designed. The major contribution of the framework is to provide an abstraction layer between the application developer and the underlying FPGA computing resources, so as to improve the overall design productivity. An inter-FPGA communication system was implemented such that gateware executing on FPGAs can communicate with each other autonomously to the CPU. Furthermore, to demonstrate a real-life application on the heterogeneous cluster, a generic k-means clustering application was implemented, using the MapReduce programming model. The implementation of the k-means application on multiple FPGAs was compared with a software-only version that was run on a Hadoop multi-core computer cluster. The performance results show that the FPGA version outperforms the Hadoop version across various parameters. An in-depth study on the communication bottleneck presented in the system was also carried out. A number of experiments were specifically designed to benchmark the performance of each I/O channel. The study shows that the major source of I/O bottleneck lies at the communication between the host system and the FPGA. This gives insight into programming considerations of potential applications on the cluster as well as improvement to the framework. Moreover, the benefit of multiple FPGAs was investigated through a series of experiments. Compared with putting all mappers on a single FPGA, it was found that distributing the same amount of mappers across more FPGAs can provide a tradeoff between FPGA resources and I/O performance.
published_or_final_version
Electrical and Electronic Engineering
Master
Master of Philosophy

This paper explores the possibilities and tools for creating High-Performace Computing cluster. It contains a project for his creation and a detailed description of the setup and configuration in a virtual environment.

The size and complexity of many scientific and enterprise-level applications require a high degree of parallelization in order to produce outputs within an acceptable period of time. This often necessitates the uses of high performance computing clusters (HPCCs) and parallelized applications which are carefully designed and optimized. A myriad of papers study the various factors which influence performance and then attempt to quantify the maximum theoretical speedup that can be achieved by a cluster relative to a sequential processor. The studies tend to only investigate the influences in isolation, but in practice these factors tend to be interdependent. It is the interaction rather than any solitary influence which normally creates the bounds of the design trade space. In the attempt to address this disconnect, this thesis blends the studies into an expanded speedup model which captures the interplay. The model is intended to help the cluster engineer make initial estimates during the early phases of design while the system is not mature enough for refinement using timing studies. The model pulls together factors such as problem scaling, resource allocation, critical sections, and the problem's inherent parallelizability. The derivation was examined theoretically and then validated by timing studies on a physical HPCC. The validation studies found that the model was an adequate generic first approximation. However, it was also found that customizations may be needed in order to account for application-specific influences such as bandwidth limitations and communication delays which are not readily incorporated into a generic model.
Master of Science

Big data has received a great deal attention from many sectors, including academia, industry and government. The Hadoop framework has emerged for supporting its storage and analysis using the MapReduce programming module. However, this framework is a complex system that has more than 150 parameters and some of them can exert a considerable effect on the performance of a Hadoop job. The optimum tuning of the Hadoop parameters is a difficult task as well as being time consuming. In this thesis, an optimisation approach is presented to improve the performance of a Hadoop framework by setting the values of the Hadoop parameters automatically. Specifically, genetic programming is used to construct a fitness function that represents the interrelations among the Hadoop parameters. Then, a genetic algorithm is employed to search for the optimum or near the optimum values of the Hadoop parameters. A Hadoop cluster is configured on two severe at Brunel University London to evaluate the performance of the proposed optimisation approach. The experimental results show that the performance of a Hadoop MapReduce job for 20 GB on Word Count Application is improved by 69.63% and 30.31% when compared to the default settings and state of the art, respectively. Whilst on Tera sort application, it is improved by 73.39% and 55.93%. For better optimisation, SDN is also employed to improve the performance of a Hadoop job. The experimental results show that the performance of a Hadoop job in SDN network for 50 GB is improved by 32.8% when compared to traditional network. Whilst on Tera sort application, the improvement for 50 GB is on average 38.7%. An effective computing platform is also presented in this thesis to support solar irradiation data analytics. It is built based on RHIPE to provide fast analysis and calculation for solar irradiation datasets. The performance of RHIPE is compared with the R language in terms of accuracy, scalability and speedup. The speed up of RHIPE is evaluated by Gustafson's Law, which is revised to enhance the performance of the parallel computation on intensive irradiation data sets in a cluster computing environment like Hadoop. The performance of the proposed work is evaluated using a Hadoop cluster based on the Microsoft azure cloud and the experimental results show that RHIPE provides considerable improvements over the R language. Finally, an effective routing algorithm based on SDN to improve the performance of a Hadoop job in a large scale cluster in a data centre network is presented. The proposed algorithm is used to improve the performance of a Hadoop job during the shuffle phase by allocating efficient paths for each shuffling flow, according to the network resources demand of each flow as well as their size and number. Furthermore, it is also employed to allocate alternative paths for each shuffling flow in the case of any link crashing or failure. This algorithm is evaluated by two network topologies, namely, fat tree and leaf-spine, built by EstiNet emulator software. The experimental results show that the proposed approach improves the performance of a Hadoop job in a data centre network.