The Difficulty of Learning Long-Term Dependencies with Gradient Flow in Recurrent Nets

Engineering International

View Publication Info
 
 
Field Value
 
Title The Difficulty of Learning Long-Term Dependencies with Gradient Flow in Recurrent Nets
 
Creator Bynagari , Naresh Babu
 
Subject Recurrent Networks
Back-Propagation through Time
Learning Long-Term Dependencies
Gradient Flow
 
Description In theory, recurrent networks (RN) can leverage their feedback connections to store activations as representations of recent input events. The most extensively used methods for learning what to put in short-term memory, on the other hand, take far too long to be practicable or do not work at all, especially when the time lags between inputs and instructor signals are long. They do not provide significant practical advantages over, the backdrop in feedforward networks with limited time windows, despite being theoretically fascinating. The goal of this article is to have a succinct overview of this rapidly evolving topic, with a focus on recent advancements. Also, we examine the asymptotic behavior of error gradients as a function of time lags to provide a hypothetical treatment of this topic. The methodology adopted in the study was to review some scholarly research papers on the subject matter to address the difficulty of learning long-term dependencies with gradient flow in recurrent nets. RNNs are the most general and powerful sequence learning algorithm currently available. Unlike Hidden Markov Models (HMMs), which have proven to be the most successful technique in a variety of sequence processing applications, they are not limited to discrete internal states and can represent continuous, dispersed sequences. As a result, they can address problems that no other method can.    Conventional RNNs, on the other hand, are difficult to train due to the problem of vanishing gradients.
 
Publisher Asian Business Consortium
 
Date 2020-12-22
 
Type info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
 
Format application/pdf
 
Identifier https://abc.us.org/ojs/index.php/ei/article/view/570
10.18034/ei.v8i2.570
 
Source Engineering International; Vol. 8 No. 2 (2020): July - December Issue; 127-138
2409-3629
10.18034/ei.v8i2
 
Language eng
 
Relation https://abc.us.org/ojs/index.php/ei/article/view/570/1072
 
Rights Copyright (c) 2020 Naresh Babu Bynagari
https://creativecommons.org/licenses/by-nc/4.0
 

Contact Us

The PKP Index is an initiative of the Public Knowledge Project.

For PKP Publishing Services please use the PKP|PS contact form.

For support with PKP software we encourage users to consult our wiki for documentation and search our support forums.

For any other correspondence feel free to contact us using the PKP contact form.

Find Us

Twitter

Copyright © 2015-2018 Simon Fraser University Library