The Difficulty of Learning Long-Term Dependencies with Gradient Flow in Recurrent Nets

Engineering International

View Publication Info
Field Value
Title The Difficulty of Learning Long-Term Dependencies with Gradient Flow in Recurrent Nets
Creator Bynagari , Naresh Babu
Subject Recurrent Networks
Back-Propagation through Time
Learning Long-Term Dependencies
Gradient Flow
Description In theory, recurrent networks (RN) can leverage their feedback connections to store activations as representations of recent input events. The most extensively used methods for learning what to put in short-term memory, on the other hand, take far too long to be practicable or do not work at all, especially when the time lags between inputs and instructor signals are long. They do not provide significant practical advantages over, the backdrop in feedforward networks with limited time windows, despite being theoretically fascinating. The goal of this article is to have a succinct overview of this rapidly evolving topic, with a focus on recent advancements. Also, we examine the asymptotic behavior of error gradients as a function of time lags to provide a hypothetical treatment of this topic. The methodology adopted in the study was to review some scholarly research papers on the subject matter to address the difficulty of learning long-term dependencies with gradient flow in recurrent nets. RNNs are the most general and powerful sequence learning algorithm currently available. Unlike Hidden Markov Models (HMMs), which have proven to be the most successful technique in a variety of sequence processing applications, they are not limited to discrete internal states and can represent continuous, dispersed sequences. As a result, they can address problems that no other method can.    Conventional RNNs, on the other hand, are difficult to train due to the problem of vanishing gradients.
Publisher Asian Business Consortium
Date 2020-12-22
Type info:eu-repo/semantics/article
Format application/pdf
Source Engineering International; Vol. 8 No. 2 (2020): July - December Issue; 127-138
Language eng
Rights Copyright (c) 2020 Naresh Babu Bynagari

Contact Us

The PKP Index is an initiative of the Public Knowledge Project.

For PKP Publishing Services please use the PKP|PS contact form.

For support with PKP software we encourage users to consult our wiki for documentation and search our support forums.

For any other correspondence feel free to contact us using the PKP contact form.

Find Us


Copyright © 2015-2018 Simon Fraser University Library