Heuristics-based compartmentalization of Replay memory in simple environments

Abstract

En viktig komponent av moderne forsterkningslæringsalgoritmer er repriseminnet. En rekke foreslåtte endringer i virkemåten til repriseminnet har blitt utforsket, men de fleste har med samplingsmekanismen å gjøre. Denne rapporten utforsker muligheten for å utvide en annen side ved repriseminnealgoritmen: det å avgjøre intelligent hvilken erfaring som skal erstattes når en ny erfaring legges til i et fullt minne. Metoden som utforskes er å dele repriseminnet i to og bruke en heuristikk til å fordele erfaringen mellom de to delene.

A ubiquitous component of state of the art reinforcement learning algorithms is the replay memory. Numerous proposed alterations to the operation of the replay memory have been explored, but they deal with the sampling mechanism. This report explores the possibility of augmenting another faucet of the replay memory algorithm: intelligently deciding on which experience to evict when adding new experience to a full memory. The explored method is to compartmentalize the replay memory into two buffers and direct experience to either based on a heuristic.