Dissertations / Theses on the topic 'Conjugate Gradient Algorithm'

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2002.
Includes bibliographical references (p. 145-147).
Optimal control problems often arise in engineering applications when a known desired behavior is to be imposed on a dynamical system. Typically, there is a performance and controller use trade-off that can be quantified as a total cost functional of the state and control histories. Problems stated in such a manner are not required to follow an exact desired behavior, alleviating potential controllability issues. We present a method for solving large deterministic optimal control problems defined by quadratic cost functionals, nonlinear state equations, and box-type constraints on the control variables. The algorithm has been developed so that systems governed by general parabolic partial differential equations can be solved. The problems addressed are of the regulator-terminal type, in which deviations from specified state variable behavior are minimized over the entire trajectory as well as at the final time. The core of the algorithm consists of an extension of the Hilbert Uniqueness Method which, we show, can be considered a statement of the dual. With the definition of a problem-specific inner-product space, a formulation is constructed around a well-conditioned, stable, SPD operator, thus leading to fast rates of convergence when solved by, for instance, a conjugate gradient procedure (denoted here TRCG). Total computational time scales roughly as twice the order of magnitude of the computational cost of a single initial-value problem.
(cont.) Standard logarithmic barrier functions and Newton methods are employed to address the hard constraints on control variables of the type Umin < U < Umax. We have shown that the TRCG algorithm allows for the incorporation of these techniques, and that convergence results maintain advantageous properties found in the standard (linear programming) literature. The TRCG operator is shown to maintain its symmetric positive-definiteness for temporal discretizations, a property that is crucial to the practical implementation of the proposed algorithm. Sample calculations are presented which illustrate the performance of the method when applied to a nonlinear heat transfer problem governed by partial differential equations.
by Ivan B. Oliveira.
Ph.D.

A blackbody optical fiber thermometer consists of an optical fiber whose sensing tip is given a metallic coating. The sensing tip of the fiber forms an isothermal cavity, and the emission from this cavity is approximately equal to the emission from a blackbody. Standard two-color optical fiber thermometry involves measuring the spectral intensity at the end of the fiber at two wavelengths. The temperature at the sensing tip of the fiber can then be inferred using Planck's law and the ratio of the spectral intensities. If, however, the length of the optical fiber is exposed to elevated temperatures, erroneous temperature measurements will occur due to emission by the fiber. This thesis presents a method to account for emission by the fiber and accurately infer the temperature at the tip of the optical fiber. Additionally, an estimate of the temperature profile along the fiber may be obtained. A mathematical relation for radiation transfer down the optical fiber is developed. The radiation exiting the fiber and the temperature profile along the fiber are related to the detector signal by a signal measurement equation. Since the temperature profile cannot be solved for directly using the signal measurement equation, two inverse minimization techniques are developed to find the temperature profile. Simulated temperature profile reconstructions show the techniques produce valid and unique results. Tip temperatures are reconstructed to within 1.0%. Experimental results are also presented. Due to the limitations of the detection system and the optical fiber probe, the uncertainty in the signal measurement equation is high. Also, due to the limitations of the laboratory furnace and the optical detector, the measurement uncertainty is also high. This leads to reconstructions that are not always accurate. Even though the temperature profiles are not completely accurate, the tip-temperatures are reconstructed to within 1%—a significant improvement over the standard two-color technique under the same conditions. Improvements are recommended that will lead to decreased measurement and signal measurement equation uncertainty. This decreased uncertainty will lead to the development of a reliable and accurate temperature measurement device.

Electrical impedance tomography is a rapidly growing discipline with an increasing number of medical and nonmedical applications. Many recent studies indicate that while the technique shows promise, improvements must be made before impedance imaging systems take their place beside more mature imaging technologies in the clinic and in the laboratory. This dissertation is an effort to address two of the shortcomings of currently available impedance tomography systems. First, a new numerical solution to the governing partial differential equation is presented which allows the user a fast, easy means of making geometrical changes. Treating the domain of interest as an input to the problem, recent results from the field of wavelet theory provide a simple means of identifying the boundary as well as giving a method for solving the partial differential equation in a fast, efficient manner. Since the algorithm only requires a pixel representation of the geometry and does not use a grid generation program, it may be of interest in applications where the geometry varies with time or the user may not be familiar with the complexities of typical finite element method grid generation programs. Second, an application of the conjugate gradient method to the problem of regularizing the nonlinear Newton-Raphson conductivity update leads to significant improvement over the popular Levenberg-Marquardt trust region regularization. The use of the conjugate gradient method as a regularization technique allows for convergence of the conductivity reconstruction in far fewer iterations and can perform reconstructions with an initial assumption of uniform conductivity in situations where other methods require either a priori knowledge or internal measurement of voltages.

A new neural network approach for performing matrix computations is presented. The idea of this approach is to construct a feed-forward neural network (FNN) and then train it by matching a desired set of patterns. The solution of the problem is the converged weight of the FNN. Accordingly, unlike the conventional FNN research that concentrates on external properties (mappings) of the networks, this study concentrates on the internal properties (weights) of the network. The present network is linear and its weights are usually strongly constrained; hence, complicated overlapped network needs to be construct. It should be noticed, however, that the present approach depends highly on the training algorithm of the FNN. Unfortunately, the available training methods; such as, the original Back-propagation (BP) algorithm, encounter many deficiencies when applied to matrix algebra problems; e. g., slow convergence due to improper choice of learning rates (LR). Thus, this study will focus on the development of new efficient and accurate FNN training methods. One improvement suggested to alleviate the problem of LR choice is the use of a line search with steepest descent method; namely, bracketing with golden section method. This provides an optimal LR as training progresses. Another improvement proposed in this study is the use of conjugate gradient (CG) methods to speed up the training process of the neural network. The computational feasibility of these methods is assessed on two matrix problems; namely, the LU-decomposition of both band and square ill-conditioned unsymmetric matrices and the inversion of square ill-conditioned unsymmetric matrices. In this study, two performance indexes have been considered; namely, learning speed and convergence accuracy. Extensive computer simulations have been carried out using the following training methods: steepest descent with line search (SDLS) method, conventional back propagation (BP) algorithm, and conjugate gradient (CG) methods; specifically, Fletcher Reeves conjugate gradient (CGFR) method and Polak Ribiere conjugate gradient (CGPR) method. The performance comparisons between these minimization methods have demonstrated that the CG training methods give better convergence accuracy and are by far the superior with respect to learning time; they offer speed-ups of anything between 3 and 4 over SDLS depending on the severity of the error goal chosen and the size of the problem. Furthermore, when using Powell's restart criteria with the CG methods, the problem of wrong convergence directions usually encountered in pure CG learning methods is alleviated. In general, CG methods with restarts have shown the best performance among all other methods in training the FNN for LU-decomposition and matrix inversion. Consequently, it is concluded that CG methods are good candidates for training FNN of matrix computations, in particular, Polak-Ribidre conjugate gradient method with Powell's restart criteria.

The parallel version of precondition techniques is developed for matrices arising from the Galerkin boundary element method for two-dimensional domains with Dirichlet boundary conditions. Results were obtained for implementations on a transputer network as well as on an nCUBE-2 parallel computer showing that iterative solution methods are very well suited for a MIMD computer. A comparison of numerical results for iterative and direct solution methods is presented and underlines the superiority of iterative methods for large systems.

A programação paralela tem sido freqüentemente adotada para o desenvolvimento de aplicações que demandam alto desempenho computacional. Com o advento das arquiteturas multi-cores e a existência de diversos níveis de paralelismo é importante definir estratégias de programação paralela que tirem proveito desse poder de processamento nessas arquiteturas. Neste contexto, este trabalho busca avaliar o desempenho da utilização das arquiteturas multi-cores, principalmente o oferecido pelas unidades de processamento gráfico (GPUs) e CPUs multi-cores na resolução de um problema térmico inverso. Algoritmos paralelos para a GPU e CPU foram desenvolvidos utilizando respectivamente as ferramentas de programação em arquiteturas de memória compartilhada NVIDIA CUDA (Compute Unified Device Architecture) e a API POSIX Threads. O algoritmo do método do gradiente conjugado pré-condicionado para resolução de sistemas lineares esparsos foi implementado totalmente no espaço da memória global da GPU em CUDA. O algoritmo desenvolvido foi avaliado em dois modelos de GPU, os quais se mostraram mais eficientes, apresentando um speedup de quatro vezes que a versão serial do algoritmo. A aplicação paralela em POSIX Threads foi avaliada em diferentes CPUs multi-cores com distintas microarquiteturas. Buscando um maior desempenho do código paralelizado foram utilizados flags de otimização as quais se mostraram muito eficientes na aplicação desenvolvida. Desta forma o código paralelizado com o auxílio das flags de otimização chegou a apresentar tempos de processamento cerca de doze vezes mais rápido que a versão serial no mesmo processador sem nenhum tipo de otimização. Assim tanto a abordagem utilizando a GPU como um co-processador genérico a CPU como a aplicação paralela empregando as CPUs multi-cores mostraram-se ferramentas eficientes para a resolução do problema térmico inverso.
Parallel programming has been frequently adopted for the development of applications that demand high-performance computing. With the advent of multi-cores architectures and the existence of several levels of parallelism are important to define programming strategies that take advantage of parallel processing power in these architectures. In this context, this study aims to evaluate the performance of architectures using multi-cores, mainly those offered by the graphics processing units (GPUs) and CPU multi-cores in the resolution of an inverse thermal problem. Parallel algorithms for the GPU and CPU were developed respectively, using the programming tools in shared memory architectures, NVIDIA CUDA (Compute Unified Device Architecture) and the POSIX Threads API. The algorithm of the preconditioned conjugate gradient method for solving sparse linear systems entirely within the global memory of the GPU was implemented by CUDA. It evaluated the two models of GPU, which proved more efficient by having a speedup was four times faster than the serial version of the algorithm. The parallel application in POSIX Threads was evaluated in different multi-core CPU with different microarchitectures. Optimization flags were used to achieve a higher performance of the parallelized code. As those were efficient in the developed application, the parallelized code presented processing times about twelve times faster than the serial version on the same processor without any optimization. Thus both the approach using GPU as a coprocessor to the CPU as a generic parallel application using the multi-core CPU proved to be more efficient tools for solving the inverse thermal problem.

There are algorithms for finding zeros or fixed points of nonlinear systems of equations that are globally convergent for almost all starting points, i.e., with probability one. The essence of all such algorithms is the construction of an appropriate homotopy map and then tracking some smooth curve in the zero set of this homotopy map. HOMPACK is a mathematical software package implementing globally convergent homotopy algorithms with three different techniques for tracking a homotopy zero curve, and has separate routines for dense and sparse Jacobian matrices. The HOMPACK algorithms for sparse Jacobian matrices use a preconditioned conjugate gradient algorithm for the computation of the kernel of the homotopy Jacobian matrix, a required linear algebra step for homotopy curve tracking. Variants of the conjugate gradient algorithm along with different preconditioners are implemented in the context of homotopy curve tracking and compared with Craig's preconditioned conjugate gradient method used in HOMPACK. In addition, a parallel version of Craig's method with incomplete LU factorization preconditioning is implemented on a shared memory parallel computer with various levels and degrees of parallelism (e.g., linear algebra, function and Jacobian matrix evaluation, etc.). An in-depth study is presented for each of these levels with respect to the speedup in execution time obtained with the parallelism, the time spent implementing the parallel code and the extra memory allocated by the parallel algorithm.
Master of Science

Orientador: Denis José Schiozer
Tese (doutorado) - Universidade Estadual de Campinas, Faculdade de Engenharia Mecânica e Instituto de Geociências
Made available in DSpace on 2018-08-24T00:34:10Z (GMT). No. of bitstreams: 1 Pinto_MarcioAugustoSampaio_D.pdf: 5097853 bytes, checksum: bc8b7f6300987de2beb9a57c26ad806a (MD5) Previous issue date: 2013
Resumo: Neste trabalho, um método de otimização assistido é proposto para estabelecer uma comparação refinada entre poços convencionais e inteligentes, considerando incertezas geológicas e econômicas. Para isto é apresentada uma metodologia dividida em quatro etapas: (1) representação e operação dos poços no simulador; (2) otimização das camadas/ou blocos completados nos poços convencionais e do número e posicionamento das válvulas nos poços inteligentes; (3) otimização da operação dos poços convencionais e das válvulas nos poços inteligentes, através de um método híbrido de otimização, composto pelo algoritmo genético rápido, para realizar a otimização global, e pelo método de gradiente conjugado, para realizar a otimização local; (4) uma análise de decisão considerando os resultados de todos os cenários geológicos e econômicos. Esta metodologia foi validada em modelos de reservatórios mais simples e com configuração de poços verticais do tipo five-spot, para em seguida ser aplicada em modelos de reservatórios mais complexos, com quatro poços produtores e quatro injetores, todos horizontais. Os resultados mostram uma clara diferença ao aplicar a metodologia proposta para estabelecer a comparação entre os dois tipos de poços. Apresenta também a comparação entre os resultados dos poços inteligentes com três tipos de controle, o reativo e mais duas formas de controle proativo. Os resultados mostram, para os casos utilizados nesta tese, uma ampla vantagem em se utilizar pelo menos uma das formas de controle proativo, ao aumentar a recuperação de óleo e VPL, reduzindo a produção e injeção de água na maioria dos casos
Abstract: In this work, an assisted optimization method is proposed to establish a refined comparison between conventional and intelligent wells, considering geological and economic uncertainties. For this, it is presented a methodology divided into four steps: (1) representation and operation of wells in the simulator, (2) optimization of the layers /blocks with completion in conventional wells and the number and placement of the valves in intelligent wells; (3) optimization of the operation of the conventional and valves in the intelligent, through a hybrid optimization method, comprising by fast genetic algorithm, to perform global optimization, and the conjugate gradient method, to perform local optimization; (4) decision analysis considering the results of all geological and economic scenarios. This method was validated in simple reservoir models and configuration of vertical wells with five-spot type, and then applied to a more complex reservoir model, with four producers and four injectors wells, all horizontal. The results show a clear difference in applying the proposed methodology to establish a comparison between the two types of wells. It also shows the comparison between the results of intelligent wells with three types of control, reactive and two ways of proactive control. The results show, for the cases used in this work, a large advantage to use intelligent wells with at least one form of proactive control, to enhance oil recovery and NPV, reducing water production and injection in most cases
Doutorado
Reservatórios e Gestão
Doutor em Ciências e Engenharia de Petróleo

The main contribution of this thesis is the concept of Fenchel duality with a focus on its application in the field of machine learning problems and image restoration tasks. We formulate a general optimization problem for modeling support vector machine tasks and assign a Fenchel dual problem to it, prove weak and strong duality statements as well as necessary and sufficient optimality conditions for that primal-dual pair. In addition, several special instances of the general optimization problem are derived for different choices of loss functions for both the regression and the classifification task. The convenience of these approaches is demonstrated by numerically solving several problems. We formulate a general nonsmooth optimization problem and assign a Fenchel dual problem to it. It is shown that the optimal objective values of the primal and the dual one coincide and that the primal problem has an optimal solution under certain assumptions. The dual problem turns out to be nonsmooth in general and therefore a regularization is performed twice to obtain an approximate dual problem that can be solved efficiently via a fast gradient algorithm. We show how an approximate optimal and feasible primal solution can be constructed by means of some sequences of proximal points closely related to the dual iterates. Furthermore, we show that the solution will indeed converge to the optimal solution of the primal for arbitrarily small accuracy. Finally, the support vector regression task is obtained to arise as a particular case of the general optimization problem and the theory is specialized to this problem. We calculate several proximal points occurring when using difffferent loss functions as well as for some regularization problems applied in image restoration tasks. Numerical experiments illustrate the applicability of our approach for these types of problems.

La thèse porte sur une étude à la fois théorique et pratique des méthodes de points intérieurs pour la programmation linéaire et la programmation quadratique convexe. Dans une première partie, elle donne une introduction aux méthodes de points intérieurs pour la programmation linéaire, décrit les outils de base, classifie et présente d'une façon unifiée les différentes méthodes. Elle présente dans la suite un exposé des algorithmes de trajectoire centrale pour la programmation linéaire et la programmation quadratique convexe. Dans une seconde partie sont étudiées des procédures de purification en programmation linéaire. Il s'agit des procédures qui déterminent, via une méthode de points intérieurs, un sommet (ou face) optimal. Dans cette partie, nous avons introduit et développé une nouvelle procédure de purification qui permet de mener dans tous les cas à un sommet optimal et de réduire le temps de calcul. La dernière partie est consacrée aux illustrations et aux expériences numériques.

Power system operation and planning relies heavily on computer simulation programs such as load flow, transient stability, contingency analysis, state estimation, short-circuit studies and optimal power flow. All of the above mentioned methodologies for planning and operation involve simulation programs which require the solution of numerous sets of simultaneous linear equations, Ax = b, whose coefficient matrices are in general very large and sparse. The main part of the computational effort involved in these algorithms is dedicated to solving such systems of linear equations.
This thesis investigates the properties of the coefficient matrix A that arises in power system analysis, as well as the application of more efficient alternative solution techniques for Ax = b which exploit these special properties. In particular, in this thesis, pre-conditioned conjugate gradient (PCG) methods have been applied and extensively tested for the first time to the solution of systems of linear equations arising in many power system operations and planning tasks.
In this vein, first, it is theoretically proven that some important power network coefficient matrices are positive definite and comply with the requirements for the convergence of the PCG method.
The PCG algorithm is then applied to the Fast Decoupled load flow and to the DC load flow. Its performance is numerically compared with a Frontal band-width direct solver (Frontal solver) as well as with a Sparspak solver (B5) with minimum degree ordering. The experimental results are based on a wide spectrum of power networks up to 5000 buses and about 10000 lines for two different types of networks: grid-type networks and star-type networks. These results demonstrate that the PCG method is clearly superior to direct solvers for certain types of large power networks.
The performance of the PCG solver within other load flow algorithms is also numerically investigated.
A detailed investigation into the eigenvalue clustering effect of alternative pre-conditioners which utilize the intrinsic properties of power networks is also presented. In addition, the effect of their eigenvalue clusterings on the convergence of the PCG algorithm is analyzed and compared with that of the classical incomplete Cholesky pre-conditioner.
Furthermore, the usefulness of the PCG solvers is investigated for complex or indefinite power network matrices. A modified PCG method was applied to the IEEE test networks as well as to large synthetically generated networks (up to 6500 buses and 13000 lines) for the solution of systems of equations Y x = b, where Y is the complex admittance matrix. Comparison with direct solvers is provided.
Finally, a new technique is developed to synthetically generate realistic data sets which characterize power networks of arbitrary size and complexity. While these networks are randomly generated, the software allows the user to specify the system dimension, type of network, connectivity configurations and other network characteristics. This software was developed to overcome the difficulties associated with the collection of network data, especially for large scale systems.

Orientador: Eduardo Cardoso de Abreu
Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Matemática Estatística e Computação Científica
Made available in DSpace on 2018-08-26T22:28:13Z (GMT). No. of bitstreams: 1 Ferraz_PaolaCunha_M.pdf: 6535346 bytes, checksum: 5f9c9ba53cd3e63fc60c09c90ad2c625 (MD5) Previous issue date: 2015
Resumo: O foco deste trabalho é aproximação numérica de problemas envolvendo equações diferenciais parciais (EDPs), de natureza elíptica, no contexto de aplicações em dinâmica de fluidos em meios porosos. Especificamente, a dissertação pretende contribuir com uma implementação de um algoritmo multiescala e multigrid, recentemente introduzido na literatura, para resolução aproximada de sistemas de equações lineares de grande porte e mal condicionados, proveniente dessa classe de EDPs, tipicamente associada a problemas de Poisson de pressão-velocidade com condições de contornos típicas de fluxo em meios porosos. O problema concreto de Poisson discutido neste trabalho será desacoplado do sistema de transporte de EDPs de convecção-difusão, com convecção dominante, e linearizado por meio do emprego de uma técnica de decomposição de operadores. A metodologia para a discretização do problema elíptico de Poisson é elementos finitos mistos híbridos. A resolução numérica do sistema linear resultante deste procedimento será realizado via um método do tipo Gradientes Conjugados com Pré-condicionamento (PCG) multiescala e multigrid. Combinamos as metodologias multi-escala e multigrid de modo a capturar os distintos comprimentos de onda associados aos diferentes comprimentos de onda do operador diferencial auto-adjunto de Poisson, fortemente influenciado pela heterogeneidade das propriedades geológicas do meio poroso, em particular da permeabilidade absoluta, que pode exibir flutuações em várias ordens de grandeza. Experimentos computacionais em aplicações de problemas de dinâmica de fluidos em meios porosos são apresentados e discutidos para verificação dos resultados obtidos
Abstract: The focus of this work is the numerical approximation of differential problems involving partial differential equations (PDE's) of elliptic nature, in the context of modelling and simulation in fluid dynamics in porous media. The dissertation aims to contribute with an implementation of a multiscale multigrid algorithm, recently introduced in the literature, designed for solving ill-conditioned large linear systems of equations derived from those classes of PDE's, typically associated with Poisson problems of pressure-velocity with boundary conditions typical of flow in porous media. The Poisson problem discussed here is identified from the coupled convection-diffusion transport system counterpart of PDE's, with dominated convection, and by a linearization by means the use of an operator splitting approach. The methodology used for the discretization of the Poisson elliptic problem is by mixed hybrid finite elements. The numerical solution of the resulting linear system will be addressed by a multiscale multigrid preconditioned conjugate gradient (PCG) method. We combine both methodologies in order to capture the distinct wavelengths associated with the different wavelengths from the assosiated self-adjoint Poisson operator, strongly influenced by the heterogeneity of the geological properties of the porous media, in particular to the absolute permeability tensor, which in turn might exhibit very large fluctuations of orders of magnitude. Numerical experiments in applications of fluid dynamics problems in porous media are presented and discussed for a verification of the results obtained by direct numerical simulations with the multiscale multigrid algorithm under consideration
Mestrado
Matematica Aplicada
Mestra em Matemática Aplicada

We present a new method for regularization of ill-conditioned problems that extends the traditional trust-region approach. Ill-conditioned problems arise, for example, in image restoration or mathematical processing of medical data, and involve matrices that are very ill-conditioned. The method makes use of the L-curve and L-curve maximum curvature criterion as a strategy recently proposed to find a good regularization parameter. We describe the method and show its application to an image restoration problem. We also provide a MATLAB code for the algorithm. Finally, a comparison to the CGLS approach is given and analyzed, and future research directions are proposed.

Este trabalho apresenta um método de reconstrução de imagens de ultrassom por problemas inversos que tem como penalidade para o erro entre solução e dados a norma L2, ou euclidiana, e como penalidade de regularização a norma L1. A motivação para o uso da regularização L1 é que se trata de um tipo de regularização promotora de esparsidade na solução. A esparsidade da regularização L1 contorna o problema de excesso do artefatos, observado em outras implementações de reconstrução por problemas inversos em ultrassom. Este problema é consequência principalmente da limitação da representação discreta do objeto contínuo no modelo de aquisição. Por conta desta limitação, objetos refletores na área imageada quase sempre localizam-se em posições que não correspondem precisamente a uma das posições do modelo discreto, gerando dados que não correspondem aos dados modelados. As formulações do problema com regularização L2 e com regularização L1 são apresentadas e comparadas dos pontos de vista geométrico e Bayesiano. O algoritmo de otimização proposto é uma implementação do algoritmo Iteratively Reweighted Least Squares (IRLS) e utiliza o método do Gradiente Conjugado (CG - Conjugate Gradient) a cada iteração, sendo chamado de IRLS-CG. São realizadas simulações com phantoms computacionais que mostram que o método permite reconstruir imagens a partir da aquisição de dados com refletores em posições não modeladas sem a observação de artefatos. As simulações também mostram melhor resolução espacial do método proposto com relação ao algoritmo delay-and-sum (DAS). Também se observou melhor desempenho computacional do CG com relação à matriz inversa nas iterações do IRLS.
This work presents an inverse problem based method for ultrasound image reconstruction which uses the L2-norm (or euclidean norm) as a penalty for the error between the data and the solution, and the L1-norm as a regularization penalty. The motivation for the use of of L1 regularization is the sparsity promoting property of this type of regularization. The sparsity of L1 regularization circumvents the problem of excess of artifatcts that is observed in other approaches of inverse problem based reconstrucion in ultrasound. Such problem is mainly a consequence of the limitation in the discrete representation of a continuous object in the acquisition model. Due to this limitation, reflecting objects in the imaged area are often localized in positions that do not correspond precisely to one of the positions in the discrete model, therefore generating data that do not correspond to the model data. The formulations of the problem with L2 regularization and with L1 regularization are presented and compared in geometric and Bayesian terms. The optimization algorithm proposed is an implementation of Iteratively Reweighted Least Squares (IRLS) and uses the Conjugate Gradient (CG) method inside each iteration, thus being called IRLS-CG. Simulations with computer phantoms are realized showing that the proposed method allows for the reconstruction of images, without observable artifacts, from data with reflectors located in non-modeled positions. Simulations also show a better spatial resolution in the proposed method when compared to the delay-and-sum (DAS) algorithm. It was also observed better computational performance of CG when compared to the matrix inversion in the iterations of IRLS.

Submitted by Renata Lopes (renatasil82@gmail.com) on 2017-04-24T12:12:06Z No. of bitstreams: 1 eduardopestanadeaguiar.pdf: 1937921 bytes, checksum: 0472ffffb70cabf120dc5de86d6626b1 (MD5)
Approved for entry into archive by Adriana Oliveira (adriana.oliveira@ufjf.edu.br) on 2017-04-24T16:55:00Z (GMT) No. of bitstreams: 1 eduardopestanadeaguiar.pdf: 1937921 bytes, checksum: 0472ffffb70cabf120dc5de86d6626b1 (MD5)
Made available in DSpace on 2017-04-24T16:55:00Z (GMT). No. of bitstreams: 1 eduardopestanadeaguiar.pdf: 1937921 bytes, checksum: 0472ffffb70cabf120dc5de86d6626b1 (MD5) Previous issue date: 2011-08-30
A presente dissertação tem como objetivo discutir o uso de técnicas de otimização baseadas no gradiente conjugado e de informações de segunda ordem para o treinamento de sistemas de inferência fuzzy singleton e non-singleton. Além disso, as soluções computacionais derivadas são aplicadas aos problemas de classificação de distúrbios múltiplos e isolados em sinais elétricos. Os resultados computacionais, obtidos a partir de dados sintéticos de distúrbios em sinais de tensão, indicam que os sistemas de inferência fuzzy singleton e non-singleton treinados pelos algoritmos de otimização considerados apresentam maior velocidade de convergência e melhores taxas de classificação quando comparados com aqueles treinados pelo algoritmo de otimização baseada em informações de primeira ordem e é bastante competitivo em relação à rede neural artificial perceptron multicamadas - multilayer perceptron (MLP) e ao classificador de Bayes.
This master dissertation aims to discuss the use of optimization techniques based on the conjugated gradient and on second order information for the training of singleton or non-singleton fuzzy inference systems. In addition, the computacional solutions obtained are applied to isolated a multiple disturbances classification problems in electric signals. Computational results obtained from synthetic data from disturbances in electric signals indicate that singleton or non-singleton fuzzy inference systems trained by the considered optimization algorithms present greater convergence speed and better classification rates when compared to those data trained by an optimization algorithm based on first order information and is quite competitive with multilayer perceptron neural network and Bayesian classifier.

La resolution de l'equation de la radiosite obtenue avec la methode galerkin est traditionnellement effectuee par l'algorithme de gauss - seidel, qui n'est pas proportionnelle par suite de son caractere recursif. Afin de depasser ces limitations pratiques, nous avons monte comment il est possible de resoudre l'equation de la radiosite a l'aide des algorithmes du type gradient conjugue generalise, qui sont non parametriques et robustes. Afin de simuler l'eclairement a l'interieur des batiments, nous avons propose une approche algorithmique originale qui resout localement cette equation. Il est desormais possible de resoudre cette equation de radiosite par la simulation de l'eclairement.

Dissertação (mestrado) - Pontifícia Universidade Católica do Paraná, Curitiba, 2006
Inclui bibliografia
Este trabalho trata do problema de minimizar uma função não linear sujeita a restrições lineares de igualdade, min f(x) s.a Ax = b onde f(x) é uma função duas vezes continuamente diferenciável, A é uma matriz m × n, com m < n, de posto m, e b um vetor de

Dans cette thèse, on a développé une méthode numérique pour la simulation des écoulements de fluides à nombre de Reynolds élevé, utilisant deux types d'éléments finis. On a établi la convergence de l'algorithme d'Uzawa en formulation de Petrov-Galerkin et on a étudié l'élément fini de Crouzeix-Raviart en formulation de Petrov-Galerkin. Pour finir, on a construit un préconditionneur du CGS pour une formulation couplée

The topic of this dissertation is the derivation, development, and evaluation of novel hybrid algorithms for process control that use a limited number of measurements and that are suitable to operate in the presence of large amounts of process noise. As an initial step, affine and neural network statistical process models are developed in order to simulate the steady-state system behavior. Such models are vitally important in the evaluation, testing, and improvement of all other process controllers referred to in this work. Afterwards, fuzzy logic controller rules are assimilated into a mathematical characterization of a model that includes the modes and mode transition rules that define a hybrid hierarchical process control. The main processing entity in such framework is a closed-loop control algorithm that performs global and then local optimizations in order to asymptotically reach minimum bias error; this is done while requiring a minimum number of iterations in order to promptly reach a desired operational window. The results of this research are applied to surface mount technology manufacturing-lines yield optimization. This work achieves a practical degree of control over the solder-paste volume deposition in the Stencil Printing Process (SPP). Results show that it is possible to change the operating point of the process by modifying certain machine parameters and even compensate for the difference in height due to change in print direction.

Visual tracking is a computer vision problem where the task is to follow a targetthrough a video sequence. Tracking has many important real-world applications in several fields such as autonomous vehicles and robot-vision. Since visual tracking does not assume any prior knowledge about the target, it faces different challenges such occlusion, appearance change, background clutter and scale change. In this thesis we try to improve the capabilities of tracking frameworks using discriminative correlation filters by incorporating scene depth information. We utilize scene depth information on three main levels. First, we use raw depth information to segment the target from its surroundings enabling occlusion detection and scale estimation. Second, we investigate different visual features calculated from depth data to decide which features are good at encoding geometric information available solely in depth data. Third, we investigate handling missing data in the depth maps using a modified version of the normalized convolution framework. Finally, we introduce a novel approach for parameter search using genetic algorithms to find the best hyperparameters for our tracking framework. Experiments show that depth data can be used to estimate scale changes and handle occlusions. In addition, visual features calculated from depth are more representative if they were combined with color features. It is also shown that utilizing normalized convolution improves the overall performance in some cases. Lastly, the usage of genetic algorithms for hyperparameter search leads to accuracy gains as well as some insights on the performance of different components within the framework.

碩士
國立成功大學
資訊管理研究所
104
Data mining, also known as knowledge discovery in database (KDD), is the computational process of discovering patterns from observed data, for which classification is one of the most important tasks in data mining. Many classification techniques, including Support Vector Machines (SVMs), have been developed over the years. Recently, SVMs have become state-of-art classifiers due to their excellent ability in solving classification problems. However, SVMs also have drawbacks, such as high computing cost with large amounts of data and high susceptibility to noisy data. Various efforts have been made to improve SVMs based on different scenarios of real world problems. Among them, taking into account experts' knowledge has been confirmed to help SVMs deal with noisy data to gain more useful results. For example, SVMs with monotonicity constraints and with the Tikhonov regularization method, also known as Regularized Monotonic SVM (RMC-SVM) incorporates inequality constraints into SVMs based on the monotonic property of real-world problems and use the Tikhonov regularization method is further applied to ensure that the solution is unique and bounded. These kinds of SVMs are also referred to as knowledge-oriented SVMs. However, solving SVMs with monotonicity constraints will require even more computational time than SVMs. In the era of big data, information is ubiquitous. The progress of data processing and analyzing the ability of computer hardware has fallen behind the growth of information. With the size of dataset becoming larger and larger, the efficiency of SVMs decreased gradually. Therefore, in this research, a parallelized Conjugate Gradient (CG) strategy is proposed to solve the regularized monotonicity constrained SVMs. Due to the characteristics of the CG method, the dataset can be divided into n parts for parallel computing at different times. This study proposed an RMC-SVMs with a parallel strategy to reduce the required training time and to increase the feasibility of using RMC-SVMs in real world applications.

碩士
國立成功大學
航空太空工程學系
89
The purpose of this study is to combine the conjugate gradient method(CG) and the genetic algorithm(GA) for the training of artificial neural networks(ANN). The back-propagation artificial neural network is a broadly used artificial neural network in many areas. It usually adopts the steepest descent method(SD) to search for a set of connection weights that minimizes the training error. But the convergence of the steepest descent method is very slow and easy to trap into a local optimal. In order to speed up the convergence, the conjugate gradient method searches the optimal weights along a set of conjugate directions in stead of steepest descent ones. But it still has the drawback of trapping into local optimals. The genetic algorithm is a global optimization method based on the Darwin’s principle of ‘’Survival of the fittest’’. The genetic algorithm always searches for the global optimal. In this study, we develop a hybrid method which combines the conjugate gradient method and the genetic algorithm to improve the convergence and successful rate for the training of artificial neural networks.

碩士
國立海洋大學
電機工程學系
89
In this thesis, we use the method of moment (MoM) to solve the electromagnetic scattering problems. A three-dimension arbitrary-shaped conductive object is divided into triangular patches, and the integral equation is discretized by MoM. Then a conjugate gradient method (CGM) is used to iteratively solve the resulting matrix equation for unknown expansion coefficients for the surface current. But when the number of unknowns is large, the CGM takes more time at each iteration. In view of this, we use the fast multipole method (FMM) to speed up the matrix-vector multiply in the CGM. The FMM reduces the complexity of a matrix-vector multiply from to , where N is the number of unknowns. The program makes use of the object-oriented programming technique and uses visual C++ as a tool to design some practical classes, which are convenient to expand programs further. This FMM algorithm also requires less memory, and hence, large and more practical problems can be solved on a PC computer.

碩士
國立中山大學
電機工程學系研究所
91
The direct-sequence code division multiple access (DS-CDMA) is one of the significant techniques for wireless communication systems with multiple simultaneous transmissions. The main concern of this thesis is to propose a new linearly constrained constant modulus modified conjugate gradient (LCCM-MCG) adaptive filtering algorithm to deal with problem of channel mismatch associated with the multiple access interference (MAI) in DS-CDMA system over multipath fading channel. In fact, the adaptive filtering algorithm based on the CM criterion is known to be very attractive for the case when the channel parameters are not estimated perfectly. The proposed LCCM-MCG algorithm is derived based on the so-called generalized sidelobe canceller (GSC). It has the advantage of having better stability and less computational complexity compared with conventional recursive least-squares (RLS) algorithm, and can be used to achieve desired performance for multiuser RAKE receiver. Moreover, with the MCG algorithm it requires only one recursive iteration per incoming sample data for updating the weight vector, but still maintains performance comparable to the RLS algorithm. From computer simulation results, we show that the proposed LCCM-MCG algorithm has fast convergence rate and could be used to circumvent the effect due to channel mismatch. Also, the performance, in terms of bit error rate (BER), is quite close to the LCCM-RLS algorithm suggested in [18], and is superior to the stochastic gradient descent (SGD) algorithm proposed in [7].

碩士
國立臺南大學
機電系統工程研究所碩士班
99
Abstract The development of high power Light Emitting Diodes (LEDs) is limited due to occurrence of heat concentration on Metal Core Printed Circuit Board (MCPCB). Reducing and eliminating the heat concentration is believed to be an excellent design for a heat sink. In this research, an optimization method was used to improve the performance of a LED light bulb in the heat sink. In addition, this study also adopted the simplified conjugate-gradient method (SCGM) and genetic algorithm (GA) combined with the finite element method (FEM) to optimize a fin heat sink. Both SCGM and GA was used to change the variable value and call the finite element package to obtain the results which later would send. These results will feedback to the SCGM, GA to evaluate the objective function and find the better variable value. This proposed method is robust and efficient for the optimal process. Finally, the results of this present study are shown that the simulation results obtained were fairly in agreement with the published data. It is proved that the proposed method is an effective and reliable tool for studying the effect of various geometrical configurations on the optimal performance design of the LED lamp holder. It is easy and convenient to design and adjust the function of the equipment through this method. Keywords: LED, S.C.G.M, G.A,

Η παρούσα διδακτορική διατριβή πραγματεύεται το θέμα της εκπαίδευσης εμπρόσθιων τροφοδοτούμενων τεχνητών νευρωνικών δικτύων και τις εφαρμογές τους. Η παρουσίαση των θεμάτων και των αποτελεσμάτων της διατριβής οργανώνεται ως εξής: Στο Κεφάλαιο 1 παρουσιάζονται τα τεχνητά νευρωνικά δίκτυα , τα οφέλη της χρήσης τους, η δομή και η λειτουργία τους. Πιο συγκεκριμένα, παρουσιάζεται πως από τους βιολογικούς νευρώνες μοντελοποιούνται οι τεχνητοί νευρώνες, που αποτελούν το θεμελιώδες στοιχείο των τεχνητών νευρωνικών δικτύων. Στη συνέχεια αναφέρονται οι βασικές αρχιτεκτονικές των εμπρόσθιων τροφοδοτούμενων τεχνητών νευρωνικών δικτύων. Το κεφάλαιο ολοκληρώνεται με μια ιστορική αναδρομή για τα τεχνητά νευρωνικά δίκτυα και με την παρουσίαση κάποιων εφαρμογών τους. Στο Κεφάλαιο 2 παρουσιάζονται μερικοί από τους υπάρχοντες αλγορίθμους εκπαίδευσης τεχνητών νευρωνικών δικτύων. Γίνεται μια περιληπτική αναφορά του προβλήματος της εκπαίδευσης των τεχνητών νευρωνικών δικτύων με επίβλεψη και δίνεται η μαθηματική μοντελοποίηση που αντιστοιχεί στην ελαχιστοποίηση του κόστους. Στην συνέχεια γίνεται μια περιληπτική αναφορά στις μεθόδους που βασίζονται στην κατεύθυνση της πιο απότομης καθόδου, στις μεθόδους δευτέρας τάξεως όπου απαιτείται ο υπολογισμός του Εσσιανού πίνακα της συνάρτησης κόστους, στις μεθόδους μεταβλητής μετρικής, και στις μεθόδους συζυγών κλίσεων. Κατόπιν, παρουσιάζεται ο χώρος των βαρών, η επιφάνεια σφάλματος και οι διάφορες τεχνικές αρχικοποίησης των βαρών των τεχνητών νευρωνικών δικτύων και περιγράφονται οι επιπτώσεις που έχουν στην εκπαίδευση τους. Στο Κεφάλαιο 3 παρουσιάζεται ένας νέος αλγόριθμος εκπαίδευσης τεχνητών νευρωνικών δικτύων βασισμένος στον αλγόριθμο της οπισθοδιάδοσης του σφάλματος και στην αυτόματη προσαρμογή του ρυθμού εκπαίδευσης χρησιμοποιώντας πληροφορία δυο σημείων. Η κατεύθυνση αναζήτησης του νέου αλγορίθμου είναι η κατεύθυνση της πιο απότομης καθόδου, αλλά για τον προσδιορισμό του ρυθμού εκπαίδευσης χρησιμοποιούνται προσεγγίσεις δυο σημείων της εξίσωσης χορδής των μεθόδων ψεύδο-Newton. Επιπλέον, παράγεται ένας νέος ρυθμός εκπαίδευσης προσεγγίζοντας την νέα εξίσωση χορδής, που προτάθηκε από τον Zhang, η οποία χρησιμοποιεί πληροφορία παραγώγων και συναρτησιακών τιμών. Στη συνέχεια, ένας κατάλληλος μηχανισμός επιλογής του ρυθμού εκπαίδευσης ενσωματώνεται στον αλγόριθμο εκπαίδευσης ώστε να επιλέγεται κάθε φορά ο κατάλληλος ρυθμός εκπαίδευσης. Τέλος, γίνεται μελέτη της σύγκλισης του αλγορίθμου εκπαίδευσης και παρουσιάζονται τα πειραματικά αποτελέσματα για διάφορα προβλήματα εκπαίδευσης. Στο Κεφάλαιο 4 παρουσιάζονται μερικοί αποτελεσματικοί αλγόριθμοι εκπαίδευσης οι οποίοι βασίζονται στις μεθόδους βελτιστοποίησης συζυγών κλίσεων. Στους υπάρχοντες αλγόριθμους εκπαίδευσης συζυγών κλίσεων προστίθεται ένας αλγόριθμος εκπαίδευσης που βασίζεται στη μέθοδο συζυγών κλίσεων του Perry. Επιπρόσθετα, προτείνονται νέοι αλγόριθμοι συζυγών κλίσεων που προκύπτουν από τις ίδιες αρχές που προέρχονται οι γνωστοί αλγόριθμοι συζυγών κλίσεων των Hestenes-Stiefel, Fletcher-Reeves, Polak-Ribiere και Perry, και ονομάζονται κλιμακωτοί αλγόριθμοι συζυγών κλίσεων. Αυτή η κατηγορία αλγορίθμων βασίζεται στην φασματική παράμετρο κλιμάκωσης του προτάθηκε από τους Barzilai και Borwein. Επιπλέον, ενσωματώνεται στους αλγόριθμους εκπαίδευσης συζυγών κλίσεων μια αποδοτική τεχνική γραμμικής αναζήτησης, που βασίζεται στις συνθήκες του Wolfe και στην διασφαλισμένη κυβική παρεμβολή. Ακόμη, η παράμετρος του αρχικού ρυθμού εκπαίδευσης προσαρμόζεται αυτόματα σε κάθε επανάληψη σύμφωνα με ένα κλειστό τύπο. Στη συνέχεια, εφαρμόζεται μια αποτελεσματική διαδικασία επανεκκίνησης, έτσι ώστε να βελτιωθούν περαιτέρω οι αλγόριθμοι εκπαίδευσης συζυγών κλίσεων και να αποδειχθεί η ολική τους σύγκλιση. Τέλος, παρουσιάζονται τα πειραματικά αποτελέσματα για διάφορα προβλήματα εκπαίδευσης. Στο τελευταίο Κεφάλαιο της παρούσας διδακτορικής διατριβής, απομονώνεται και τροποποιείται ο κλιμακωτός αλγόριθμος του Perry, που παρουσιάστηκε στο προηγούμενο κεφάλαιο. Πιο συγκεκριμένα, ενώ διατηρούνται τα κύρια χαρακτηριστικά του αλγορίθμου εκπαίδευσης, εφαρμόζεται μια διαφορετική τεχνική γραμμικής αναζήτησης η οποία βασίζεται στις μη μονότονες συνθήκες του Wolfe. Επίσης προτείνεται ένας νέος αρχικός ρυθμός εκπαίδευσης για χρήση με τον κλιμακωτό αλγόριθμο εκπαίδευσης συζυγών κλίσεων, ο οποίος φαίνεται να είναι αποδοτικότερος από τον αρχικό ρυθμό εκπαίδευσης που προτάθηκε από τον Shanno όταν χρησιμοποιείται σε συνδυασμό με την μη μονότονη τεχνική γραμμικής αναζήτησης. Στη συνέχεια παρουσιάζονται τα πειραματικά αποτελέσματα για διάφορα προβλήματα εκπαίδευσης. Τέλος, ως εφαρμογή εκπαιδεύεται ένα πολυεπίπεδο εμπρόσθια τροφοδοτούμενο τεχνητό νευρωνικό δίκτυο με τον προτεινόμενο αλγόριθμο για το πρόβλημα της ταξινόμησης καρκινικών κυττάρων του εγκεφάλου και συγκρίνεται η απόδοση του με την απόδοση ενός πιθανοτικού τεχνητού νευρωνικού δικτύου. Η διατριβή ολοκληρώνεται με το Παράρτημα Α’, όπου παρουσιάζονται τα προβλήματα εκπαίδευσης τεχνητών νευρωνικών δικτύων που χρησιμοποιήθηκαν για την αξιολόγηση των προτεινόμενων αλγορίθμων εκπαίδευσης.
In this dissertation the problem of the training of feedforward artificial neural networks and its applications are considered. The presentation of the topics and the results are organized as follows: In the first chapter, the artificial neural networks are introduced. Initially, the benefits of the use of artificial neural networks are presented. In the sequence, the structure and their functionality are presented. More specifically, the derivation of the artificial neurons from the biological ones is presented followed by the presentation of the architecture of the feedforward neural networks. The historical notes and the use of neural networks in real world problems are concluding the first chapter. In Chapter 2, the existing training algorithms for the feedforward neural networks are considered. First, a summary of the training problem and its mathematical formulation, that corresponds to the uncostrained minimization of a cost function, are given. In the sequence, training algorithms based on the steepest descent, Newton, variable metric and conjugate gradient methods are presented. Furthermore, the weight space, the error surface and the techniques of the initialization of the weights are described. Their influence in the training procedure is discussed. In Chapter 3, a new training algorithm for feedforward neural networks based on the backpropagation algorithm and the automatic two-point step size (learning rate) is presented. The algorithm uses the steepest descent search direction while the learning rate parameter is calculated by minimizing the standard secant equation. Furthermore, a new learning rate parameter is derived by minimizing the modified secant equation introduced by Zhang, that uses both gradient and function value information. In the sequece a switching mechanism is incorporated into the algorithm so that the appropriate stepsize to be chosen according to the status of the current iterative point. Finaly, the global convergence of the proposed algorithm is studied and the results of some numerical experiments are presented. In Chapter 4, some efficient training algorithms, based on conjugate gradient optimization methods, are presented. In addition to the existing conjugate gradient training algorithms, we introduce Perry's conjugate gradient method as a training algorithm. Furthermore, a new class of conjugate gradient methods is proposed, called self-scaled conjugate gradient methods, which are derived from the principles of Hestenes-Stiefel, Fletcher-Reeves, Polak-Ribiere and Perry's method. This class is based on the spectral scaling parameter. Furthermore, we incorporate to the conjugate gradient training algorithms an efficient line search technique based on the Wolfe conditions and on safeguarded cubic interpolation. In addition, the initial learning rate parameter, fed to the line search technique, was automatically adapted at each iteration by a closed formula. Finally, an efficient restarting procedure was employed in order to further improve the effectiveness of the conjugate gradient training algorithms and prove their global convergence. Experimental results show that, in general, the new class of methods can perform better with a much lower computational cost and better success performance. In the last chapter of this dissertation, the Perry's self-scaled conjugate gradient training algorithm that was presented in the previous chapter was isolated and modified. More specifically, the main characteristics of the training algorithm were maintained but in this case a different line search strategy based on the nonmonotone Wolfe conditions was utilized. Furthermore, a new initial learning rate parameter was introduced for use in conjunction with the self-scaled conjugate gradient training algorithm that seems to be more effective from the initial learning rate parameter, proposed by Shanno, when used with the nonmonotone line search technique. In the sequence the experimental results for differrent training problems are presented. Finally, a feedforward neural network with the proposed algorithm for the problem of brain astrocytomas grading was trained and compared the results with those achieved by a probabilistic neural network. The dissertation is concluded with the Appendix A', where the training problems used for the evaluation of the proposed training algorithms are presented.

The main contribution of this thesis is the concept of Fenchel duality with a focus on its application in the field of machine learning problems and image restoration tasks. We formulate a general optimization problem for modeling support vector machine tasks and assign a Fenchel dual problem to it, prove weak and strong duality statements as well as necessary and sufficient optimality conditions for that primal-dual pair. In addition, several special instances of the general optimization problem are derived for different choices of loss functions for both the regression and the classifification task. The convenience of these approaches is demonstrated by numerically solving several problems. We formulate a general nonsmooth optimization problem and assign a Fenchel dual problem to it. It is shown that the optimal objective values of the primal and the dual one coincide and that the primal problem has an optimal solution under certain assumptions. The dual problem turns out to be nonsmooth in general and therefore a regularization is performed twice to obtain an approximate dual problem that can be solved efficiently via a fast gradient algorithm. We show how an approximate optimal and feasible primal solution can be constructed by means of some sequences of proximal points closely related to the dual iterates. Furthermore, we show that the solution will indeed converge to the optimal solution of the primal for arbitrarily small accuracy. Finally, the support vector regression task is obtained to arise as a particular case of the general optimization problem and the theory is specialized to this problem. We calculate several proximal points occurring when using difffferent loss functions as well as for some regularization problems applied in image restoration tasks. Numerical experiments illustrate the applicability of our approach for these types of problems.

Η χρήση επαναληπτικών τεχνικών προσαρμοστικής ισοστάθμισης διαύλου αποτελεί μια σχετικά πρόσφατη και πολλά υποσχόμενη μέθοδο αντιμετώπισης του φαινομένου της διασυμβολικής παρεμβολής που εισάγεται από το κανάλι λόγω του φαινομένου της πολυδιόδευσης. Ο αλγόριθμος που έχει επικρατήσει στις περισσότερες προσαρμοστικές εφαρμογές είναι ο ελαχίστων μέσων τετραγώνων (LMS). Διακρίνεται για την απλότητά του, έχει όμως φτωχές ιδιότητες σύγκλισης. Η μέθοδος των αναδρομικών ελαχίστων τετραγώνων (RLS) είναι επίσης αρκετά διαδεδομένη και κατέχει υπερέχουσες ιδιότητες σύγκλισης. Ωστόσο παρουσιάζει μεγάλη υπολογιστική πολυπλοκότητα και αυξημένες απαιτήσεις σε μνήμη. Στα πλαίσια της εργασίας αυτής εγίνε μια προσπάθεια ανάλυσης των τεχνικών που βασίζονται στη μέθοδο των συζυγών παραγώγων (Conjugate Gradient), χρησιμοποιούνται σε προβλήματα προσαρμοστικού φιλτραρίσματος και πιο ειδικά στο πρόβλημα της προσαρμοστικής ισοστάθμισης διαύλου. Οι τεχνικές αυτές επεξεργάζονται τα δεδομένα και ανά μπλοκ. Είναι ικανές να παρέχουν ιδιότητες σύγκλισης συγκρίσιμες με αυτές της (RLS) μεθόδου, εισάγοντας υπολογιστική πολυπλοκότητα ενδιάμεσων απαιτήσεων μεταξύ των μεθόδων LMS και RLS χωρίς να παρουσιάζουν προβλήματα αριθμητικής ευστάθειας.
The use of iteration methods for adaptive equalization has received considerable attention during the past several decades. The Least Mean Squares (LMS) method, which has found widespread use owing to its simplicity, has poor convergence properties. The Recursive Least Squares (RLS) method possess superior convergence properties, but it is computationally intensive and has high storage requirements for matrix manipulations. In this MSc thesis the technique of conjugate gradients is applied for the adaptive filtering problem. Conjugate gradient algorithms for adaptive filtering applications suitable for efficient implementation has been developed and has been applied for the design of an adaptive transversal equalizer. Low cost block algorithms using the preconditioned conjugate gradient method are also discussed. The algorithms are capable of providing convergence comparable to RLS schemes at a computational complexity between the LMS and the RLS methods and does not suffer from any known instability problems.

Kwok-po Ng.
Thesis (Ph.D.)--Chinese University of Hong Kong, 1995.
Includes bibliographical references (leaves 138-139).
Abstracts
Summary
Introduction --- p.1
Summary of the papers A-F --- p.2
Paper A --- p.15
Paper B --- p.36
Paper C --- p.63
Paper D --- p.87
Paper E --- p.109
Paper F --- p.122

Thesis (Ph. D.)--York University, 2001. Graduate Programme in Mathematics.
Typescript. Includes bibliographical references (leaves 114-119). Also available on the Internet. MODE OF ACCESS via web browser by entering the following URL: http://wwwlib.umi.com/cr/yorku/fullcit?pNQ66358.

Η συνεισφορά της παρούσας διατριβής επικεντρώνεται στην ανάπτυξη και στη Μαθηματική θεμελίωση νέων μεθόδων συζυγών κλίσεων για βελτιστοποίηση χωρίς περιορισμούς και στη μελέτη νέων μεθόδων εκπαίδευσης νευρωνικών δικτύων και εφαρμογών τους. Αναπτύσσουμε δύο νέες μεθόδους βελτιστοποίησης, οι οποίες ανήκουν στην κλάση των μεθόδων συζυγών κλίσεων. Οι νέες μέθοδοι βασίζονται σε νέες εξισώσεις της τέμνουσας με ισχυρά θεωρητικά πλεονεκτήματα, όπως η προσέγγιση με μεγαλύτερη ακρίβεια της επιφάνεια της αντικειμενικής συνάρτησης. Επιπλέον, μία σημαντική ιδιότητα και των δύο προτεινόμενων μεθόδων είναι ότι εγγυώνται επαρκή μείωση ανεξάρτητα από την ακρίβεια της γραμμικής αναζήτησης, αποφεύγοντας τις συχνά αναποτελεσματικές επανεκκινήσεις. Επίσης, αποδείξαμε την ολική σύγκλιση των προτεινόμενων μεθόδων για μη κυρτές συναρτήσεις. Με βάση τα αριθμητικά μας αποτελέσματα καταλήγουμε στο συμπέρασμα ότι οι νέες μέθοδοι έχουν πολύ καλή υπολογιστική αποτελεσματικότητα, όπως και καλή ταχύτητα επίλυσης των προβλημάτων, υπερτερώντας σημαντικά των κλασικών μεθόδων συζυγών κλίσεων. Το δεύτερο μέρος της διατριβής είναι αφιερωμένο στην ανάπτυξη και στη μελέτη νέων μεθόδων εκπαίδευσης νευρωνικών δικτύων. Προτείνουμε νέες μεθόδους, οι οποίες διατηρούν τα πλεονεκτήματα των κλασικών μεθόδων συζυγών κλίσεων και εξασφαλίζουν τη δημιουργία κατευθύνσεων μείωσης αποφεύγοντας τις συχνά αναποτελεσματικές επανεκκινήσεις. Επιπλέον, αποδείξαμε ότι οι προτεινόμενες μέθοδοι συγκλίνουν ολικά για μη κυρτές συναρτήσεις. Τα αριθμητικά αποτελέσματα επαληθεύουν ότι οι προτεινόμενες μέθοδοι παρέχουν γρήγορη, σταθερότερη και πιο αξιόπιστη σύγκλιση, υπερτερώντας των κλασικών μεθόδων εκπαίδευσης. Η παρουσίαση του ερευνητικού μέρους της διατριβής ολοκληρώνεται με μία νέα μέθοδο εκπαίδευσης νευρωνικών δικτύων, η οποία βασίζεται σε μία καμπυλόγραμμη αναζήτηση. Η μέθοδος χρησιμοποιεί τη BFGS ενημέρωση ελάχιστης μνήμης για τον υπολογισμό των κατευθύνσεων μείωσης, η οποία αντλεί πληροφορία από την ιδιοσύνθεση του προσεγγιστικού Eσσιανού πίνακα, αποφεύγοντας οποιαδήποτε αποθήκευση ή παραγοντοποίηση πίνακα, έτσι ώστε η μέθοδος να μπορεί να εφαρμοστεί για την εκπαίδευση νευρωνικών δικτύων μεγάλης κλίμακας. Ο αλγόριθμος εφαρμόζεται σε προβλήματα από το πεδίο της τεχνητής νοημοσύνης και της βιοπληροφορικής καταγράφοντας πολύ καλά αποτελέσματα. Επίσης, με σκοπό την αύξηση της ικανότητας γενίκευσης των εκπαιδευόμενων δικτύων διερευνήσαμε πειραματικά και αξιολογήσαμε την εφαρμογή τεχνικών μείωσης της διάστασης δεδομένων στην απόδοση της γενίκευσης των τεχνητών νευρωνικών δικτύων σε μεγάλης κλίμακας δεδομένα βιοϊατρικής.
The contribution of this thesis focuses on the development and the Mathematical foundation of new conjugate gradient methods for unconstrained optimization and on the study of new neural network training methods and their applications. We propose two new conjugate gradient methods for unconstrained optimization. The proposed methods are based on new secant equations with strong theoretical advantages i.e. they approximate the surface of the objective function with higher accuracy. Moreover, they have the attractive property of ensuring sufficient descent independent of the accuracy of the line search, avoiding thereby the usual inefficient restarts. Further, we have established the global convergence of the proposed methods for general functions under mild conditions. Based on our numerical results we conclude that our proposed methods outperform classical conjugate gradient methods in both efficiency and robustness. The second part of the thesis is devoted on the study and development of new neural network training algorithms. More specifically, we propose some new training methods which preserve the advantages of classical conjugate gradient methods while simultaneously ensure sufficient descent using any line search, avoiding thereby the usual inefficient restarts. Moreover, we have established the global convergence of our proposed methods for general functions. Encouraging numerical experiments on famous benchmarks verify that the presented methods provide fast, stable and reliable convergence, outperforming classical training methods. Finally, the presentation of the research work of this dissertation is fulfilled with the presentation of a new curvilinear algorithm for training large neural networks which is based on the analysis of the eigenstructure of the memoryless BFGS matrices. The proposed method preserves the strong convergence properties provided by the quasi-Newton direction while simultaneously it exploits the nonconvexity of the error surface through the computation of the negative curvature direction without using any storage and matrix factorization. Our numerical experiments have shown that the proposed method outperforms other popular training methods on famous benchmarks. Furthermore, for improving the generalization capability of trained ANNs, we explore the incorporation of several dimensionality reduction techniques as a pre-processing step. To this end, we have experimentally evaluated the application of dimensional reduction techniques for increasing the generalization capability of neural network in large biomedical datasets.