Relevant bibliographies by topics / Sparse deep neural networks

Contents

  1. Journal articles
  2. Dissertations / Theses
  3. Books
  4. Book chapters
  5. Conference papers
  6. Reports

Academic literature on the topic 'Sparse deep neural networks'

Author: Grafiati
Published: 7 July 2024
Last updated: 7 July 2024

Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles

Select a source type:

Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Sparse deep neural networks.'

Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.

You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.

Journal articles on the topic "Sparse deep neural networks"

1

Scardapane, Simone, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. "Group sparse regularization for deep neural networks." Neurocomputing 241 (June 2017): 81–89. http://dx.doi.org/10.1016/j.neucom.2017.02.029.

Full text
APA, Harvard, Vancouver, ISO, and other styles
2

Zang, Ke, Wenqi Wu, and Wei Luo. "Deep Sparse Learning for Automatic Modulation Classification Using Recurrent Neural Networks." Sensors 21, no. 19 (September 25, 2021): 6410. http://dx.doi.org/10.3390/s21196410.

Full text
Abstract:
Deep learning models, especially recurrent neural networks (RNNs), have been successfully applied to automatic modulation classification (AMC) problems recently. However, deep neural networks are usually overparameterized, i.e., most of the connections between neurons are redundant. The large model size hinders the deployment of deep neural networks in applications such as Internet-of-Things (IoT) networks. Therefore, reducing parameters without compromising the network performance via sparse learning is often desirable since it can alleviates the computational and storage burdens of deep learning models. In this paper, we propose a sparse learning algorithm that can directly train a sparsely connected neural network based on the statistics of weight magnitude and gradient momentum. We first used the MNIST and CIFAR10 datasets to demonstrate the effectiveness of this method. Subsequently, we applied it to RNNs with different pruning strategies on recurrent and non-recurrent connections for AMC problems. Experimental results demonstrated that the proposed method can effectively reduce the parameters of the neural networks while maintaining model performance. Moreover, we show that appropriate sparsity can further improve network generalization ability.
APA, Harvard, Vancouver, ISO, and other styles
3

Wu, Kailun, Yiwen Guo, and Changshui Zhang. "Compressing Deep Neural Networks With Sparse Matrix Factorization." IEEE Transactions on Neural Networks and Learning Systems 31, no. 10 (October 2020): 3828–38. http://dx.doi.org/10.1109/tnnls.2019.2946636.

Full text
APA, Harvard, Vancouver, ISO, and other styles
4

Gangopadhyay, Briti, Pallab Dasgupta, and Soumyajit Dey. "Safety Aware Neural Pruning for Deep Reinforcement Learning (Student Abstract)." Proceedings of the AAAI Conference on Artificial Intelligence 37, no. 13 (June 26, 2023): 16212–13. http://dx.doi.org/10.1609/aaai.v37i13.26966.

Full text
Abstract:
Neural network pruning is a technique of network compression by removing weights of lower importance from an optimized neural network. Often, pruned networks are compared in terms of accuracy, which is realized in terms of rewards for Deep Reinforcement Learning (DRL) networks. However, networks that estimate control actions for safety-critical tasks, must also adhere to safety requirements along with obtaining rewards. We propose a methodology to iteratively refine the weights of a pruned neural network such that we get a sparse high-performance network without significant side effects on safety.
APA, Harvard, Vancouver, ISO, and other styles
5

Petschenig, Horst, and Robert Legenstein. "Quantized rewiring: hardware-aware training of sparse deep neural networks." Neuromorphic Computing and Engineering 3, no. 2 (May 26, 2023): 024006. http://dx.doi.org/10.1088/2634-4386/accd8f.

Full text
Abstract:
Abstract Mixed-signal and fully digital neuromorphic systems have been of significant interest for deploying spiking neural networks in an energy-efficient manner. However, many of these systems impose constraints in terms of fan-in, memory, or synaptic weight precision that have to be considered during network design and training. In this paper, we present quantized rewiring (Q-rewiring), an algorithm that can train both spiking and non-spiking neural networks while meeting hardware constraints during the entire training process. To demonstrate our approach, we train both feedforward and recurrent neural networks with a combined fan-in/weight precision limit, a constraint that is, for example, present in the DYNAP-SE mixed-signal analog/digital neuromorphic processor. Q-rewiring simultaneously performs quantization and rewiring of synapses and synaptic weights through gradient descent updates and projecting the trainable parameters to a constraint-compliant region. Using our algorithm, we find trade-offs between the number of incoming connections to neurons and network performance for a number of common benchmark datasets.
APA, Harvard, Vancouver, ISO, and other styles
6

Belay, Kaleab. "Gradient and Mangitude Based Pruning for Sparse Deep Neural Networks." Proceedings of the AAAI Conference on Artificial Intelligence 36, no. 11 (June 28, 2022): 13126–27. http://dx.doi.org/10.1609/aaai.v36i11.21699.

Full text
Abstract:
Deep Neural Networks have memory and computational demands that often render them difficult to use in low-resource environments. Also, highly dense networks are over-parameterized and thus prone to overfitting. To address these problems, we introduce a novel algorithm that prunes (sparsifies) weights from the network by taking into account their magnitudes and gradients taken against a validation dataset. Unlike existing pruning methods, our method does not require the network model to be retrained once initial training is completed. On the CIFAR-10 dataset, our method reduced the number of paramters of MobileNet by a factor of 9X, from 14 million to 1.5 million, with just a 3.8% drop in accuracy.
APA, Harvard, Vancouver, ISO, and other styles
7

Kaur, Mandeep, and Pradip Kumar Yadava. "A Review on Classification of Images with Convolutional Neural Networks." International Journal for Research in Applied Science and Engineering Technology 11, no. 7 (July 31, 2023): 658–63. http://dx.doi.org/10.22214/ijraset.2023.54704.

Full text
Abstract:
Abstract: Deep learning has recently been applied to scene labelling, object tracking, pose estimation, text detection and recognition, visual saliency detection, and image categorization. Deep learning typically uses models like Auto Encoder, Sparse Coding, Restricted Boltzmann Machine, Deep Belief Networks, and Convolutional Neural Networks. Convolutional neural networks have exhibited good performance in picture categorization when compared to other types of models. A straightforward Convolutional neural network for image categorization was built in this paper. The image classification was finished by this straightforward Convolutional neural network. On the foundation of the Convolutional neural network, we also examined several learning rate setting techniques and different optimisation algorithms for determining the ideal parameters that have the greatest influence on image categorization
APA, Harvard, Vancouver, ISO, and other styles
8

Bi, Jia, and Steve R. Gunn. "Sparse Deep Neural Network Optimization for Embedded Intelligence." International Journal on Artificial Intelligence Tools 29, no. 03n04 (June 2020): 2060002. http://dx.doi.org/10.1142/s0218213020600027.

Full text
Abstract:
Deep neural networks become more popular as its ability to solve very complex pattern recognition problems. However, deep neural networks often need massive computational and memory resources, which is main reason resulting them to be difficult efficiently and entirely running on embedded platforms. This work addresses this problem by saving the computational and memory requirements of deep neural networks by proposing a variance reduced (VR)-based optimization with regularization techniques to compress the requirements of memory of models within fast training process. It is shown theoretically and experimentally that sparsity-inducing regularization can be effectively worked with the VR-based optimization whereby in the optimizer the behaviors of the stochastic element is controlled by a hyper-parameter to solve non-convex problems.
APA, Harvard, Vancouver, ISO, and other styles
9

Gallicchio, Claudio, and Alessio Micheli. "Fast and Deep Graph Neural Networks." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 04 (April 3, 2020): 3898–905. http://dx.doi.org/10.1609/aaai.v34i04.5803.

Full text
Abstract:
We address the efficiency issue for the construction of a deep graph neural network (GNN). The approach exploits the idea of representing each input graph as a fixed point of a dynamical system (implemented through a recurrent neural network), and leverages a deep architectural organization of the recurrent units. Efficiency is gained by many aspects, including the use of small and very sparse networks, where the weights of the recurrent units are left untrained under the stability condition introduced in this work. This can be viewed as a way to study the intrinsic power of the architecture of a deep GNN, and also to provide insights for the set-up of more complex fully-trained models. Through experimental results, we show that even without training of the recurrent connections, the architecture of small deep GNN is surprisingly able to achieve or improve the state-of-the-art performance on a significant set of tasks in the field of graphs classification.
APA, Harvard, Vancouver, ISO, and other styles
10

Tartaglione, Enzo, Andrea Bragagnolo, Attilio Fiandrotti, and Marco Grangetto. "LOss-Based SensiTivity rEgulaRization: Towards deep sparse neural networks." Neural Networks 146 (February 2022): 230–37. http://dx.doi.org/10.1016/j.neunet.2021.11.029.

Full text
APA, Harvard, Vancouver, ISO, and other styles
More sources

Dissertations / Theses on the topic "Sparse deep neural networks"

1

Tavanaei, Amirhossein. "Spiking Neural Networks and Sparse Deep Learning." Thesis, University of Louisiana at Lafayette, 2019. http://pqdtopen.proquest.com/#viewpdf?dispub=10807940.

Full text
Abstract:

This document proposes new methods for training multi-layer and deep spiking neural networks (SNNs), specifically, spiking convolutional neural networks (CNNs). Training a multi-layer spiking network poses difficulties because the output spikes do not have derivatives and the commonly used backpropagation method for non-spiking networks is not easily applied. Our methods use novel versions of the brain-like, local learning rule named spike-timing-dependent plasticity (STDP) that incorporates supervised and unsupervised components. Our method starts with conventional learning methods and converts them to spatio-temporally local rules suited for SNNs.

The training uses two components for unsupervised feature extraction and supervised classification. The first component refers to new STDP rules for spike-based representation learning that trains convolutional filters and initial representations. The second introduces new STDP-based supervised learning rules for spike pattern classification via an approximation to gradient descent by combining the STDP and anti-STDP rules. Specifically, the STDP-based supervised learning model approximates gradient descent by using temporally local STDP rules. Stacking these components implements a novel sparse, spiking deep learning model. Our spiking deep learning model is categorized as a variation of spiking CNNs of integrate-and-fire (IF) neurons with performance comparable with the state-of-the-art deep SNNs. The experimental results show the success of the proposed model for image classification. Our network architecture is the only spiking CNN which provides bio-inspired STDP rules in a hierarchy of feature extraction and classification in an entirely spike-based framework.

APA, Harvard, Vancouver, ISO, and other styles
2

Le, Quoc Tung. "Algorithmic and theoretical aspects of sparse deep neural networks." Electronic Thesis or Diss., Lyon, École normale supérieure, 2023. http://www.theses.fr/2023ENSL0105.

Full text
Abstract:
Les réseaux de neurones profonds parcimonieux offrent une opportunité pratique convaincante pour réduire le coût de l'entraînement, de l'inférence et du stockage, qui augmente de manière exponentielle dans l'état de l'art de l'apprentissage profond. Dans cette présentation, nous introduirons une approche pour étudier les réseaux de neurones profonds parcimonieux à travers le prisme d'un autre problème : la factorisation de matrices sous constraints de parcimonie, c'est-à-dire le problème d'approximation d'une matrice (dense) par le produit de facteurs (multiples) parcimonieux. En particulier, nous identifions et étudions en détail certains aspects théoriques et algorithmiques d'une variante de la factorisation de matrices parcimonieux appelée factorisation de matrices à support fixe (FSMF), dans laquelle l'ensemble des entrées non nulles des facteurs parcimonieux est connu. Plusieurs questions fondamentales des réseaux de neurones profonds parcimonieux, telles que l'existence de solutions optimales du problème d'entraînement ou les propriétés topologiques de son espace fonctionnel, peuvent être abordées à l'aide des résultats de la (FSMF). De plus, en appliquant les résultats de la (FSMF), nous étudions également la paramétrisation du type "butterfly", une approche qui consiste à remplacer les matrices de poids (larges) par le produit de matrices extrêmement parcimonieuses et structurées dans les réseaux de neurones profonds parcimonieux
Sparse deep neural networks offer a compelling practical opportunity to reduce the cost of training, inference and storage, which are growing exponentially in the state of the art of deep learning. In this presentation, we will introduce an approach to study sparse deep neural networks through the lens of another related problem: sparse matrix factorization, i.e., the problem of approximating a (dense) matrix by the product of (multiple) sparse factors. In particular, we identify and investigate in detail some theoretical and algorithmic aspects of a variant of sparse matrix factorization named fixed support matrix factorization (FSMF) in which the set of non-zero entries of sparse factors are known. Several fundamental questions of sparse deep neural networks such as the existence of optimal solutions of the training problem or topological properties of its function space can be addressed using the results of (FSMF). In addition, by applying the results of (FSMF), we also study the butterfly parametrization, an approach that consists of replacing (large) weight matrices by the products of extremely sparse and structured ones in sparse deep neural networks
APA, Harvard, Vancouver, ISO, and other styles
3

Hoori, Ammar O. "MULTI-COLUMN NEURAL NETWORKS AND SPARSE CODING NOVEL TECHNIQUES IN MACHINE LEARNING." VCU Scholars Compass, 2019. https://scholarscompass.vcu.edu/etd/5743.

Full text
Abstract:
Accurate and fast machine learning (ML) algorithms are highly vital in artificial intelligence (AI) applications. In complex dataset problems, traditional ML methods such as radial basis function neural network (RBFN), sparse coding (SC) using dictionary learning, and particle swarm optimization (PSO) provide trivial results, large structure, slow training, and/or slow testing. This dissertation introduces four novel ML techniques: the multi-column RBFN network (MCRN), the projected dictionary learning algorithm (PDL) and the multi-column adaptive and non-adaptive particle swarm optimization techniques (MC-APSO and MC-PSO). These novel techniques provide efficient alternatives for traditional ML techniques. Compared to traditional ML techniques, the novel ML techniques demonstrate more accurate results, faster training and testing timing, and parallelized structured solutions. MCRN deploys small RBFNs in a parallel structure to speed up both training and testing. Each RBFN is trained with a subset of the dataset and the overall structure provides results that are more accurate. PDL introduces a conceptual dictionary learning method in updating the dictionary atoms with the reconstructed input blocks. This method improves the sparsity of extracted features and hence, the image denoising results. MC-PSO and MC-APSO provide fast and more accurate alternatives to the PSO and APSO slow evolutionary techniques. MC-PSO and MC-APSO use multi-column parallelized RBFN structure to improve results and speed with a wide range of classification dataset problems. The novel techniques are trained and tested using benchmark dataset problems and the results are compared with the state-of-the-art counterpart techniques to evaluate their performance. Novel techniques’ results show superiority over techniques in accuracy and speed in most of the experimental results, which make them good alternatives in solving difficult ML problems.
APA, Harvard, Vancouver, ISO, and other styles
4

Vekhande, Swapnil Sudhir. "Deep Learning Neural Network-based Sinogram Interpolation for Sparse-View CT Reconstruction." Thesis, Virginia Tech, 2019. http://hdl.handle.net/10919/90182.

Full text
Abstract:
Computed Tomography (CT) finds applications across domains like medical diagnosis, security screening, and scientific research. In medical imaging, CT allows physicians to diagnose injuries and disease more quickly and accurately than other imaging techniques. However, CT is one of the most significant contributors of radiation dose to the general population and the required radiation dose for scanning could lead to cancer. On the other hand, a shallow radiation dose could sacrifice image quality causing misdiagnosis. To reduce the radiation dose, sparse-view CT, which includes capturing a smaller number of projections, becomes a promising alternative. However, the image reconstructed from linearly interpolated views possesses severe artifacts. Recently, Deep Learning-based methods are increasingly being used to interpret the missing data by learning the nature of the image formation process. The current methods are promising but operate mostly in the image domain presumably due to lack of projection data. Another limitation is the use of simulated data with less sparsity (up to 75%). This research aims to interpolate the missing sparse-view CT in the sinogram domain using deep learning. To this end, a residual U-Net architecture has been trained with patch-wise projection data to minimize Euclidean distance between the ground truth and the interpolated sinogram. The model can generate highly sparse missing projection data. The results show improvement in SSIM and RMSE by 14% and 52% respectively with respect to the linear interpolation-based methods. Thus, experimental sparse-view CT data with 90% sparsity has been successfully interpolated while improving CT image quality.
Master of Science
Computed Tomography is a commonly used imaging technique due to the remarkable ability to visualize internal organs, bones, soft tissues, and blood vessels. It involves exposing the subject to X-ray radiation, which could lead to cancer. On the other hand, the radiation dose is critical for the image quality and subsequent diagnosis. Thus, image reconstruction using only a small number of projection data is an open research problem. Deep learning techniques have already revolutionized various Computer Vision applications. Here, we have used a method which fills missing highly sparse CT data. The results show that the deep learning-based method outperforms standard linear interpolation-based methods while improving the image quality.
APA, Harvard, Vancouver, ISO, and other styles
5

Carvalho, Micael. "Deep representation spaces." Electronic Thesis or Diss., Sorbonne université, 2018. http://www.theses.fr/2018SORUS292.

Full text
Abstract:
Ces dernières années, les techniques d’apprentissage profond ont fondamentalement transformé l'état de l'art de nombreuses applications de l'apprentissage automatique, devenant la nouvelle approche standard pour plusieurs d’entre elles. Les architectures provenant de ces techniques ont été utilisées pour l'apprentissage par transfert, ce qui a élargi la puissance des modèles profonds à des tâches qui ne disposaient pas de suffisamment de données pour les entraîner à partir de zéro. Le sujet d'étude de cette thèse couvre les espaces de représentation créés par les architectures profondes. Dans un premier temps, nous étudions les propriétés de leurs espaces, en prêtant un intérêt particulier à la redondance des dimensions et la précision numérique de leurs représentations. Nos résultats démontrent un fort degré de robustesse, pointant vers des schémas de compression simples et puissants. Ensuite, nous nous concentrons sur le l'affinement de ces représentations. Nous choisissons d'adopter un problème multi-tâches intermodal et de concevoir une fonction de coût capable de tirer parti des données de plusieurs modalités, tout en tenant compte des différentes tâches associées au même ensemble de données. Afin d'équilibrer correctement ces coûts, nous développons également un nouveau processus d'échantillonnage qui ne prend en compte que des exemples contribuant à la phase d'apprentissage, c'est-à-dire ceux ayant un coût positif. Enfin, nous testons notre approche sur un ensemble de données à grande échelle de recettes de cuisine et d'images associées. Notre méthode améliore de 5 fois l'état de l'art sur cette tâche, et nous montrons que l'aspect multitâche de notre approche favorise l'organisation sémantique de l'espace de représentation, lui permettant d'effectuer des sous-tâches jamais vues pendant l'entraînement, comme l'exclusion et la sélection d’ingrédients. Les résultats que nous présentons dans cette thèse ouvrent de nombreuses possibilités, y compris la compression de caractéristiques pour les applications distantes, l'apprentissage multi-modal et multitâche robuste et l'affinement de l'espace des caractéristiques. Pour l'application dans le contexte de la cuisine, beaucoup de nos résultats sont directement applicables dans une situation réelle, en particulier pour la détection d'allergènes, la recherche de recettes alternatives en raison de restrictions alimentaires et la planification de menus
In recent years, Deep Learning techniques have swept the state-of-the-art of many applications of Machine Learning, becoming the new standard approach for them. The architectures issued from these techniques have been used for transfer learning, which extended the power of deep models to tasks that did not have enough data to fully train them from scratch. This thesis' subject of study is the representation spaces created by deep architectures. First, we study properties inherent to them, with particular interest in dimensionality redundancy and precision of their features. Our findings reveal a strong degree of robustness, pointing the path to simple and powerful compression schemes. Then, we focus on refining these representations. We choose to adopt a cross-modal multi-task problem, and design a loss function capable of taking advantage of data coming from multiple modalities, while also taking into account different tasks associated to the same dataset. In order to correctly balance these losses, we also we develop a new sampling scheme that only takes into account examples contributing to the learning phase, i.e. those having a positive loss. Finally, we test our approach in a large-scale dataset of cooking recipes and associated pictures. Our method achieves a 5-fold improvement over the state-of-the-art, and we show that the multi-task aspect of our approach promotes a semantically meaningful organization of the representation space, allowing it to perform subtasks never seen during training, like ingredient exclusion and selection. The results we present in this thesis open many possibilities, including feature compression for remote applications, robust multi-modal and multi-task learning, and feature space refinement. For the cooking application, in particular, many of our findings are directly applicable in a real-world context, especially for the detection of allergens, finding alternative recipes due to dietary restrictions, and menu planning
APA, Harvard, Vancouver, ISO, and other styles
6

Pawlowski, Filip igor. "High-performance dense tensor and sparse matrix kernels for machine learning." Thesis, Lyon, 2020. http://www.theses.fr/2020LYSEN081.

Full text
Abstract:
Dans cette thèse, nous développons des algorithmes à haute performance pour certains calculs impliquant des tenseurs denses et des matrices éparses. Nous abordons les opérations du noyau qui sont utiles pour les tâches d'apprentissage de la machine, telles que l'inférence avec les réseaux neuronaux profonds. Nous développons des structures de données et des techniques pour réduire l'utilisation de la mémoire, pour améliorer la localisation des données et donc pour améliorer la réutilisation du cache des opérations du noyau. Nous concevons des algorithmes parallèles à mémoire séquentielle et à mémoire partagée.Dans la première partie de la thèse, nous nous concentrons sur les noyaux tenseurs denses. Les noyaux tenseurs comprennent la multiplication tenseur-vecteur (TVM), la multiplication tenseur-matrice (TMM) et la multiplication tenseur-tendeur (TTM). Parmi ceux-ci, la MVT est la plus liée à la largeur de bande et constitue un élément de base pour de nombreux algorithmes. Nous proposons une nouvelle structure de données qui stocke le tenseur sous forme de blocs, qui sont ordonnés en utilisant la courbe de remplissage de l'espace connue sous le nom de courbe de Morton (ou courbe en Z). L'idée clé consiste à diviser le tenseur en blocs suffisamment petits pour tenir dans le cache et à les stocker selon l'ordre de Morton, tout en conservant un ordre simple et multidimensionnel sur les éléments individuels qui les composent. Ainsi, des routines BLAS haute performance peuvent être utilisées comme micro-noyaux pour chaque bloc. Les résultats démontrent non seulement que l'approche proposée est plus performante que les variantes de pointe jusqu'à 18%, mais aussi que l'approche proposée induit 71% de moins d'écart-type d'échantillon pour le MVT dans les différents modes possibles. Enfin, nous étudions des algorithmes de mémoire partagée parallèles pour la MVT qui utilisent la structure de données proposée. Nos résultats sur un maximum de 8 systèmes de prises montrent une performance presque maximale pour l'algorithme proposé pour les tenseurs à 2, 3, 4 et 5 dimensions.Dans la deuxième partie de la thèse, nous explorons les calculs épars dans les réseaux de neurones en nous concentrant sur le problème d'inférence profonde épars à haute performance. L'inférence sparse DNN est la tâche d'utiliser les réseaux sparse DNN pour classifier un lot d'éléments de données formant, dans notre cas, une matrice de caractéristiques sparse. La performance de l'inférence clairsemée dépend de la parallélisation efficace de la matrice clairsemée - la multiplication matricielle clairsemée (SpGEMM) répétée pour chaque couche dans la fonction d'inférence. Nous introduisons ensuite l'inférence modèle-parallèle, qui utilise un partitionnement bidimensionnel des matrices de poids obtenues à l'aide du logiciel de partitionnement des hypergraphes. Enfin, nous introduisons les algorithmes de tuilage modèle-parallèle et de tuilage hybride, qui augmentent la réutilisation du cache entre les couches, et utilisent un module de synchronisation faible pour cacher le déséquilibre de charge et les coûts de synchronisation. Nous évaluons nos techniques sur les données du grand réseau du IEEE HPEC 2019 Graph Challenge sur les systèmes à mémoire partagée et nous rapportons jusqu'à 2x l'accélération par rapport à la ligne de base
In this thesis, we develop high performance algorithms for certain computations involving dense tensors and sparse matrices. We address kernel operations that are useful for machine learning tasks, such as inference with deep neural networks (DNNs). We develop data structures and techniques to reduce memory use, to improve data locality and hence to improve cache reuse of the kernel operations. We design both sequential and shared-memory parallel algorithms. In the first part of the thesis we focus on dense tensors kernels. Tensor kernels include the tensor--vector multiplication (TVM), tensor--matrix multiplication (TMM), and tensor--tensor multiplication (TTM). Among these, TVM is the most bandwidth-bound and constitutes a building block for many algorithms. We focus on this operation and develop a data structure and sequential and parallel algorithms for it. We propose a novel data structure which stores the tensor as blocks, which are ordered using the space-filling curve known as the Morton curve (or Z-curve). The key idea consists of dividing the tensor into blocks small enough to fit cache, and storing them according to the Morton order, while keeping a simple, multi-dimensional order on the individual elements within them. Thus, high performance BLAS routines can be used as microkernels for each block. We evaluate our techniques on a set of experiments. The results not only demonstrate superior performance of the proposed approach over the state-of-the-art variants by up to 18%, but also show that the proposed approach induces 71% less sample standard deviation for the TVM across the d possible modes. Finally, we show that our data structure naturally expands to other tensor kernels by demonstrating that it yields up to 38% higher performance for the higher-order power method. Finally, we investigate shared-memory parallel TVM algorithms which use the proposed data structure. Several alternative parallel algorithms were characterized theoretically and implemented using OpenMP to compare them experimentally. Our results on up to 8 socket systems show near peak performance for the proposed algorithm for 2, 3, 4, and 5-dimensional tensors. In the second part of the thesis, we explore the sparse computations in neural networks focusing on the high-performance sparse deep inference problem. The sparse DNN inference is the task of using sparse DNN networks to classify a batch of data elements forming, in our case, a sparse feature matrix. The performance of sparse inference hinges on efficient parallelization of the sparse matrix--sparse matrix multiplication (SpGEMM) repeated for each layer in the inference function. We first characterize efficient sequential SpGEMM algorithms for our use case. We then introduce the model-parallel inference, which uses a two-dimensional partitioning of the weight matrices obtained using the hypergraph partitioning software. The model-parallel variant uses barriers to synchronize at layers. Finally, we introduce tiling model-parallel and tiling hybrid algorithms, which increase cache reuse between the layers, and use a weak synchronization module to hide load imbalance and synchronization costs. We evaluate our techniques on the large network data from the IEEE HPEC 2019 Graph Challenge on shared-memory systems and report up to 2x times speed-up versus the baseline
APA, Harvard, Vancouver, ISO, and other styles
7

Thom, Markus [Verfasser]. "Sparse neural networks / Markus Thom." Ulm : Universität Ulm. Fakultät für Ingenieurwissenschaften und Informatik, 2015. http://d-nb.info/1067496319/34.

Full text
APA, Harvard, Vancouver, ISO, and other styles
8

Liu, Qian. "Deep spiking neural networks." Thesis, University of Manchester, 2018. https://www.research.manchester.ac.uk/portal/en/theses/deep-spiking-neural-networks(336e6a37-2a0b-41ff-9ffb-cca897220d6c).html.

Full text
Abstract:
Neuromorphic Engineering (NE) has led to the development of biologically-inspired computer architectures whose long-term goal is to approach the performance of the human brain in terms of energy efficiency and cognitive capabilities. Although there are a number of neuromorphic platforms available for large-scale Spiking Neural Network (SNN) simulations, the problem of programming these brain-like machines to be competent in cognitive applications still remains unsolved. On the other hand, Deep Learning has emerged in Artificial Neural Network (ANN) research to dominate state-of-the-art solutions for cognitive tasks. Thus the main research problem emerges of understanding how to operate and train biologically-plausible SNNs to close the gap in cognitive capabilities between SNNs and ANNs. SNNs can be trained by first training an equivalent ANN and then transferring the tuned weights to the SNN. This method is called ‘off-line’ training, since it does not take place on an SNN directly, but rather on an ANN instead. However, previous work on such off-line training methods has struggled in terms of poor modelling accuracy of the spiking neurons and high computational complexity. In this thesis we propose a simple and novel activation function, Noisy Softplus (NSP), to closely model the response firing activity of biologically-plausible spiking neurons, and introduce a generalised off-line training method using the Parametric Activation Function (PAF) to map the abstract numerical values of the ANN to concrete physical units, such as current and firing rate in the SNN. Based on this generalised training method and its fine tuning, we achieve the state-of-the-art accuracy on the MNIST classification task using spiking neurons, 99.07%, on a deep spiking convolutional neural network (ConvNet). We then take a step forward to ‘on-line’ training methods, where Deep Learning modules are trained purely on SNNs in an event-driven manner. Existing work has failed to provide SNNs with recognition accuracy equivalent to ANNs due to the lack of mathematical analysis. Thus we propose a formalised Spike-based Rate Multiplication (SRM) method which transforms the product of firing rates to the number of coincident spikes of a pair of rate-coded spike trains. Moreover, these coincident spikes can be captured by the Spike-Time-Dependent Plasticity (STDP) rule to update the weights between the neurons in an on-line, event-based, and biologically-plausible manner. Furthermore, we put forward solutions to reduce correlations between spike trains; thereby addressing the result of performance drop in on-line SNN training. The promising results of spiking Autoencoders (AEs) and Restricted Boltzmann Machines (SRBMs) exhibit equivalent, sometimes even superior, classification and reconstruction capabilities compared to their non-spiking counterparts. To provide meaningful comparisons between these proposed SNN models and other existing methods within this rapidly advancing field of NE, we propose a large dataset of spike-based visual stimuli and a corresponding evaluation methodology to estimate the overall performance of SNN models and their hardware implementations.
APA, Harvard, Vancouver, ISO, and other styles
9

Squadrani, Lorenzo. "Deep neural networks and thermodynamics." Bachelor's thesis, Alma Mater Studiorum - Università di Bologna, 2020.

Find full text
Abstract:
Deep learning is the most effective and used approach to artificial intelligence, and yet it is far from being properly understood. The understanding of it is the way to go to further improve its effectiveness and in the best case to gain some understanding of the "natural" intelligence. We attempt a step in this direction with the aim of physics. We describe a convolutional neural network for image classification (trained on CIFAR-10) within the descriptive framework of Thermodynamics. In particular we define and study the temperature of each component of the network. Our results provides a new point of view on deep learning models, which may be a starting point towards a better understanding of artificial intelligence.
APA, Harvard, Vancouver, ISO, and other styles
10

Mancevo, del Castillo Ayala Diego. "Compressing Deep Convolutional Neural Networks." Thesis, KTH, Skolan för datavetenskap och kommunikation (CSC), 2017. http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-217316.

Full text
Abstract:
Deep Convolutional Neural Networks and "deep learning" in general stand at the cutting edge on a range of applications, from image based recognition and classification to natural language processing, speech and speaker recognition and reinforcement learning. Very deep models however are often large, complex and computationally expensive to train and evaluate. Deep learning models are thus seldom deployed natively in environments where computational resources are scarce or expensive. To address this problem we turn our attention towards a range of techniques that we collectively refer to as "model compression" where a lighter student model is trained to approximate the output produced by the model we wish to compress. To this end, the output from the original model is used to craft the training labels of the smaller student model. This work contains some experiments on CIFAR-10 and demonstrates how to use the aforementioned techniques to compress a people counting model whose precision, recall and F1-score are improved by as much as 14% against our baseline.
APA, Harvard, Vancouver, ISO, and other styles
More sources

Books on the topic "Sparse deep neural networks"

1

A, Renzetti N., and Jet Propulsion Laboratory (U.S.), eds. The Deep Space Network as an instrument for radio science research: Power system stability applications of artificial neural networks. Pasadena, Calif: National Aeronautics and Space Administration, Jet Propulsion Laboratory, California Institute of Technology, 1993.

Find full text
APA, Harvard, Vancouver, ISO, and other styles
2

Aggarwal, Charu C. Neural Networks and Deep Learning. Cham: Springer International Publishing, 2018. http://dx.doi.org/10.1007/978-3-319-94463-0.

Full text
APA, Harvard, Vancouver, ISO, and other styles
3

Aggarwal, Charu C. Neural Networks and Deep Learning. Cham: Springer International Publishing, 2023. http://dx.doi.org/10.1007/978-3-031-29642-0.

Full text
APA, Harvard, Vancouver, ISO, and other styles
4

Moolayil, Jojo. Learn Keras for Deep Neural Networks. Berkeley, CA: Apress, 2019. http://dx.doi.org/10.1007/978-1-4842-4240-7.

Full text
APA, Harvard, Vancouver, ISO, and other styles
5

Caterini, Anthony L., and Dong Eui Chang. Deep Neural Networks in a Mathematical Framework. Cham: Springer International Publishing, 2018. http://dx.doi.org/10.1007/978-3-319-75304-1.

Full text
APA, Harvard, Vancouver, ISO, and other styles
6

Razaghi, Hooshmand Shokri. Statistical Machine Learning & Deep Neural Networks Applied to Neural Data Analysis. [New York, N.Y.?]: [publisher not identified], 2020.

Find full text
APA, Harvard, Vancouver, ISO, and other styles
7

Fingscheidt, Tim, Hanno Gottschalk, and Sebastian Houben, eds. Deep Neural Networks and Data for Automated Driving. Cham: Springer International Publishing, 2022. http://dx.doi.org/10.1007/978-3-031-01233-4.

Full text
APA, Harvard, Vancouver, ISO, and other styles
8

Modrzyk, Nicolas. Real-Time IoT Imaging with Deep Neural Networks. Berkeley, CA: Apress, 2020. http://dx.doi.org/10.1007/978-1-4842-5722-7.

Full text
APA, Harvard, Vancouver, ISO, and other styles
9

Iba, Hitoshi. Evolutionary Approach to Machine Learning and Deep Neural Networks. Singapore: Springer Singapore, 2018. http://dx.doi.org/10.1007/978-981-13-0200-8.

Full text
APA, Harvard, Vancouver, ISO, and other styles
10

Lu, Le, Yefeng Zheng, Gustavo Carneiro, and Lin Yang, eds. Deep Learning and Convolutional Neural Networks for Medical Image Computing. Cham: Springer International Publishing, 2017. http://dx.doi.org/10.1007/978-3-319-42999-1.

Full text
APA, Harvard, Vancouver, ISO, and other styles
More sources

Book chapters on the topic "Sparse deep neural networks"

1

Moons, Bert, Daniel Bankman, and Marian Verhelst. "ENVISION: Energy-Scalable Sparse Convolutional Neural Network Processing." In Embedded Deep Learning, 115–51. Cham: Springer International Publishing, 2018. http://dx.doi.org/10.1007/978-3-319-99223-5_5.

Full text
APA, Harvard, Vancouver, ISO, and other styles
2

Wang, Xin, Zhiqiang Hou, Wangsheng Yu, and Zefenfen Jin. "Online Fast Deep Learning Tracker Based on Deep Sparse Neural Networks." In Lecture Notes in Computer Science, 186–98. Cham: Springer International Publishing, 2017. http://dx.doi.org/10.1007/978-3-319-71607-7_17.

Full text
APA, Harvard, Vancouver, ISO, and other styles
3

Huang, Zehao, and Naiyan Wang. "Data-Driven Sparse Structure Selection for Deep Neural Networks." In Computer Vision – ECCV 2018, 317–34. Cham: Springer International Publishing, 2018. http://dx.doi.org/10.1007/978-3-030-01270-0_19.

Full text
APA, Harvard, Vancouver, ISO, and other styles
4

Fakhfakh, Mohamed, Bassem Bouaziz, Lotfi Chaari, and Faiez Gargouri. "Efficient Bayesian Learning of Sparse Deep Artificial Neural Networks." In Lecture Notes in Computer Science, 78–88. Cham: Springer International Publishing, 2022. http://dx.doi.org/10.1007/978-3-031-01333-1_7.

Full text
APA, Harvard, Vancouver, ISO, and other styles
5

Dey, Sourya, Yinan Shao, Keith M. Chugg, and Peter A. Beerel. "Accelerating Training of Deep Neural Networks via Sparse Edge Processing." In Artificial Neural Networks and Machine Learning – ICANN 2017, 273–80. Cham: Springer International Publishing, 2017. http://dx.doi.org/10.1007/978-3-319-68600-4_32.

Full text
APA, Harvard, Vancouver, ISO, and other styles
6

Huang, Junzhou, and Zheng Xu. "Cell Detection with Deep Learning Accelerated by Sparse Kernel." In Deep Learning and Convolutional Neural Networks for Medical Image Computing, 137–57. Cham: Springer International Publishing, 2017. http://dx.doi.org/10.1007/978-3-319-42999-1_9.

Full text
APA, Harvard, Vancouver, ISO, and other styles
7

Matsumoto, Wataru, Manabu Hagiwara, Petros T. Boufounos, Kunihiko Fukushima, Toshisada Mariyama, and Zhao Xiongxin. "A Deep Neural Network Architecture Using Dimensionality Reduction with Sparse Matrices." In Neural Information Processing, 397–404. Cham: Springer International Publishing, 2016. http://dx.doi.org/10.1007/978-3-319-46681-1_48.

Full text
APA, Harvard, Vancouver, ISO, and other styles
8

Xu, Ting, Bo Zhang, Baoju Zhang, Taekon Kim, and Yi Wang. "Sparse Deep Neural Network Based Directional Modulation Design." In Lecture Notes in Electrical Engineering, 503–11. Singapore: Springer Nature Singapore, 2024. http://dx.doi.org/10.1007/978-981-99-7545-7_51.

Full text
APA, Harvard, Vancouver, ISO, and other styles
9

Dai, Qionghai, and Yue Gao. "Neural Networks on Hypergraph." In Artificial Intelligence: Foundations, Theory, and Algorithms, 121–43. Singapore: Springer Nature Singapore, 2023. http://dx.doi.org/10.1007/978-981-99-0185-2_7.

Full text
Abstract:
AbstractWith the development of deep learning on high-order correlations, hypergraph neural networks have received much attention in recent years. Generally, the neural networks on hypergraph can be divided into two categories, including the spectral-based methods and the spatial-based methods. For the spectral-based methods, the convolution operation is formulated in the spectral domain of graph, and we introduce the typical spectral-based methods, including hypergraph neural networks (HGNN), hypergraph convolution with attention (Hyper-Atten), and hyperbolic hypergraph neural network (HHGNN), which extend hypergraph computation to hyperbolic spaces beyond the Euclidean space. For the spatial-based methods, the convolution operation is defined in groups of spatially close vertices. We then present spatial-based hypergraph neural networks of the general hypergraph neural networks (HGNN+) and the dynamic hypergraph neural networks (DHGNN). Additionally, there are several convolution methods that attempt to reduce the hypergraph structure to the graph structure, so that the existing graph convolution methods can be directly deployed. Lastly, we analyze the association and comparison between hypergraph and graph in the two areas described above (spectral-based, spatial-based), further demonstrating the ability and advantages of hypergraph on constructing and computing higher-order correlations in the data.
APA, Harvard, Vancouver, ISO, and other styles
10

Marinò, Giosuè Cataldo, Gregorio Ghidoli, Marco Frasca, and Dario Malchiodi. "Reproducing the Sparse Huffman Address Map Compression for Deep Neural Networks." In Reproducible Research in Pattern Recognition, 161–66. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-76423-4_12.

Full text
APA, Harvard, Vancouver, ISO, and other styles

Conference papers on the topic "Sparse deep neural networks"

1

Keyvanrad, Mohammad Ali, and Mohammad Mehdi Homayounpour. "Normal sparse Deep Belief Network." In 2015 International Joint Conference on Neural Networks (IJCNN). IEEE, 2015. http://dx.doi.org/10.1109/ijcnn.2015.7280688.

Full text
APA, Harvard, Vancouver, ISO, and other styles
2

Huang, Sitao, Carl Pearson, Rakesh Nagi, Jinjun Xiong, Deming Chen, and Wen-mei Hwu. "Accelerating Sparse Deep Neural Networks on FPGAs." In 2019 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2019. http://dx.doi.org/10.1109/hpec.2019.8916419.

Full text
APA, Harvard, Vancouver, ISO, and other styles
3

Obmann, Daniel, Johannes Schwab, and Markus Haltmeier. "Sparse synthesis regularization with deep neural networks." In 2019 13th International conference on Sampling Theory and Applications (SampTA). IEEE, 2019. http://dx.doi.org/10.1109/sampta45681.2019.9030953.

Full text
APA, Harvard, Vancouver, ISO, and other styles
4

Wen, Weijing, Fan Yang, Yangfeng Su, Dian Zhou, and Xuan Zeng. "Learning Sparse Patterns in Deep Neural Networks." In 2019 IEEE 13th International Conference on ASIC (ASICON). IEEE, 2019. http://dx.doi.org/10.1109/asicon47005.2019.8983429.

Full text
APA, Harvard, Vancouver, ISO, and other styles
5

Bi, Jia, and Steve R. Gunn. "Sparse Deep Neural Networks for Embedded Intelligence." In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2018. http://dx.doi.org/10.1109/ictai.2018.00016.

Full text
APA, Harvard, Vancouver, ISO, and other styles
6

Jing, How, and Yu Tsao. "Sparse maximum entropy deep belief nets." In 2013 International Joint Conference on Neural Networks (IJCNN 2013 - Dallas). IEEE, 2013. http://dx.doi.org/10.1109/ijcnn.2013.6706749.

Full text
APA, Harvard, Vancouver, ISO, and other styles
7

Xu, Lie, Chiu-Sing Choy, and Yi-Wen Li. "Deep sparse rectifier neural networks for speech denoising." In 2016 IEEE International Workshop on Acoustic Signal Enhancement (IWAENC). IEEE, 2016. http://dx.doi.org/10.1109/iwaenc.2016.7602891.

Full text
APA, Harvard, Vancouver, ISO, and other styles
8

Toth, Laszlo. "Phone recognition with deep sparse rectifier neural networks." In ICASSP 2013 - 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2013. http://dx.doi.org/10.1109/icassp.2013.6639016.

Full text
APA, Harvard, Vancouver, ISO, and other styles
9

Pironkov, Gueorgui, Stephane Dupont, and Thierry Dutoit. "Investigating sparse deep neural networks for speech recognition." In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015. http://dx.doi.org/10.1109/asru.2015.7404784.

Full text
APA, Harvard, Vancouver, ISO, and other styles
10

Mitsuno, Kakeru, Junichi Miyao, and Takio Kurita. "Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks." In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020. http://dx.doi.org/10.1109/ijcnn48605.2020.9207531.

Full text
APA, Harvard, Vancouver, ISO, and other styles

Reports on the topic "Sparse deep neural networks"

1

Yu, Haichao, Haoxiang Li, Honghui Shi, Thomas S. Huang, and Gang Hua. Any-Precision Deep Neural Networks. Web of Open Science, December 2020. http://dx.doi.org/10.37686/ejai.v1i1.82.

Full text
Abstract:
We present Any-Precision Deep Neural Networks (Any- Precision DNNs), which are trained with a new method that empowers learned DNNs to be flexible in any numerical precision during inference. The same model in runtime can be flexibly and directly set to different bit-width, by trun- cating the least significant bits, to support dynamic speed and accuracy trade-off. When all layers are set to low- bits, we show that the model achieved accuracy compara- ble to dedicated models trained at the same precision. This nice property facilitates flexible deployment of deep learn- ing models in real-world applications, where in practice trade-offs between model accuracy and runtime efficiency are often sought. Previous literature presents solutions to train models at each individual fixed efficiency/accuracy trade-off point. But how to produce a model flexible in runtime precision is largely unexplored. When the demand of efficiency/accuracy trade-off varies from time to time or even dynamically changes in runtime, it is infeasible to re-train models accordingly, and the storage budget may forbid keeping multiple models. Our proposed framework achieves this flexibility without performance degradation. More importantly, we demonstrate that this achievement is agnostic to model architectures. We experimentally validated our method with different deep network backbones (AlexNet-small, Resnet-20, Resnet-50) on different datasets (SVHN, Cifar-10, ImageNet) and observed consistent results.
APA, Harvard, Vancouver, ISO, and other styles
2

Koh, Christopher Fu-Chai, and Sergey Igorevich Magedov. Bond Order Prediction Using Deep Neural Networks. Office of Scientific and Technical Information (OSTI), August 2019. http://dx.doi.org/10.2172/1557202.

Full text
APA, Harvard, Vancouver, ISO, and other styles
3

Shevitski, Brian, Yijing Watkins, Nicole Man, and Michael Girard. Digital Signal Processing Using Deep Neural Networks. Office of Scientific and Technical Information (OSTI), April 2023. http://dx.doi.org/10.2172/1984848.

Full text
APA, Harvard, Vancouver, ISO, and other styles
4

Landon, Nicholas. A survey of repair strategies for deep neural networks. Ames (Iowa): Iowa State University, August 2022. http://dx.doi.org/10.31274/cc-20240624-93.

Full text
APA, Harvard, Vancouver, ISO, and other styles
5

Talathi, S. S. Deep Recurrent Neural Networks for seizure detection and early seizure detection systems. Office of Scientific and Technical Information (OSTI), June 2017. http://dx.doi.org/10.2172/1366924.

Full text
APA, Harvard, Vancouver, ISO, and other styles
6

Armstrong, Derek Elswick, and Joseph Gabriel Gorka. Using Deep Neural Networks to Extract Fireball Parameters from Infrared Spectral Data. Office of Scientific and Technical Information (OSTI), May 2020. http://dx.doi.org/10.2172/1623398.

Full text
APA, Harvard, Vancouver, ISO, and other styles
7

Thulasidasan, Sunil, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah E. Michalak. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. Office of Scientific and Technical Information (OSTI), June 2019. http://dx.doi.org/10.2172/1525811.

Full text
APA, Harvard, Vancouver, ISO, and other styles
8

Ellis, John, Attila Cangi, Normand Modine, John Stephens, Aidan Thompson, and Sivasankaran Rajamanickam. Accelerating Finite-temperature Kohn-Sham Density Functional Theory\ with Deep Neural Networks. Office of Scientific and Technical Information (OSTI), October 2020. http://dx.doi.org/10.2172/1677521.

Full text
APA, Harvard, Vancouver, ISO, and other styles
9

Ellis, Austin, Lenz Fielder, Gabriel Popoola, Normand Modine, John Stephens, Aidan Thompson, and Sivasankaran Rajamanickam. Accelerating Finite-Temperature Kohn-Sham Density Functional Theory with Deep Neural Networks. Office of Scientific and Technical Information (OSTI), June 2021. http://dx.doi.org/10.2172/1817970.

Full text
APA, Harvard, Vancouver, ISO, and other styles
10

Chronopoulos, Ilias, Katerina Chrysikou, George Kapetanios, James Mitchell, and Aristeidis Raftapostolos. Deep Neural Network Estimation in Panel Data Models. Federal Reserve Bank of Cleveland, July 2023. http://dx.doi.org/10.26509/frbc-wp-202315.

Full text
Abstract:
In this paper we study neural networks and their approximating power in panel data models. We provide asymptotic guarantees on deep feed-forward neural network estimation of the conditional mean, building on the work of Farrell et al. (2021), and explore latent patterns in the cross-section. We use the proposed estimators to forecast the progression of new COVID-19 cases across the G7 countries during the pandemic. We find significant forecasting gains over both linear panel and nonlinear time-series models. Containment or lockdown policies, as instigated at the national level by governments, are found to have out-of-sample predictive power for new COVID-19 cases. We illustrate how the use of partial derivatives can help open the "black box" of neural networks and facilitate semi-structural analysis: school and workplace closures are found to have been effective policies at restricting the progression of the pandemic across the G7 countries. But our methods illustrate significant heterogeneity and time variation in the effectiveness of specific containment policies.
APA, Harvard, Vancouver, ISO, and other styles
You might also be interested in the extended bibliographies on the topic 'Sparse deep neural networks' for particular source types:
Journal articles Dissertations / Theses Books
We offer discounts on all premium plans for authors whose works are included in thematic literature selections. Contact us to get a unique promo code!

To the bibliography