A Julia Implementation of the Differentiable Neural Computer

Abstract

I differensierbar programmering-paradigmet modelleres programmer som parametriserte løsninger på et problem som deretter trenes med gradientbaserte optimeringsmetoder. DeepMinds Differentiable Neural Computer (DNC) er et eksempel på overgangen fra tradisjonell dyp læring til differensierbar programmering. I denne oppgaven argumenteres det for at Julias maskinlæringsrammeverk, med Flux i sentrum og med Zygote som algoritmisk differensiator, er et mer passende verktøy for implementasjon av programmer som DNC enn TensorFlow. Implementasjonen settes i kontekst av algoritmisk differensiering sin rolle i differensierbar programmering. DNC reimplementers i Julia og dens interne metoder måles opp mot den originale TensorFlow-implementasjonen, der den oppnår lik treningstid med enkelte interne metoder som kjører opp mot syv ganger så raskt.

In the paradigm of differentiable programming, programs are modeled as a parameterized solution prototype, which is trained by gradient-based optimization. DeepMind's Differential Neural Computer (DNC) is an example of the transition from traditional deep learning models to differential programs. In this thesis, a Julia implementation of the DNC is presented. An argument is made for the applicability of the Julia programming language and its machine learning ecosystem, with Flux at its core and Zygote as the algorithmic differentiation engine, on differentiable programs. The focus of the thesis is on the role of algorithmic differentiation in machine learning toolkits. The reimplementation is benchmarked, achieving 10\% speedup on a simple training task with a simpler implementation, with some internal methods running up to seven times faster.