Auto Generated Documentation

Preprocessing

imageProcessing.zeroCenterGrayData(image)

Preprocesses image, using our Specular Removal Preprocessing strategy.

Parameters:image – Matrix containing an image Shape = [224, 224, 3]
Returns:Preprocessed image
imageProcessing.zeroCenterOFData(image, datasetType)
Normalizes and zero centers Optical Flow data
Parameters:
  • image – pixel matrix (224 x 224 x 3)
  • datasetType – the sampling rate of our data
Returns:

zero-centered and normalized image, based on sampling rate used

Spatial Stream

SpatialStream.VGG16_tensorflow.trainSpatialStream()

Fine tunes an improved VGG-16 from tflearn checkpoint.

  1. Define learning rate and epsilon
  2. Define Network Architecture
  3. Define loss and optimizer
  4. Load checkpoint weights
  5. Load dataset
  6. Start training
  7. Store best performing model
Returns:Nothing
class SpatialStream.Spatial_File_Handler.File_Handler(feed_folder_path, dataType, nonfeed_folder_path, numTimesteps, datasetType)
get_GRAY_Batch(batch_size, startPos)

Builds a new mini-batch through selecting images from a list of video frame paths.

Each mini-batch consists of an equal amount of Feeding and NonFeeding frames.

Each image is preprocessed using the Specular Removal Preprocesing strategy.

Parameters:
  • batch_size – size for the mini-batch
  • startPos – the start position for the list of video frames
Returns:

A preprocessed mini-batch, labels and the new start position

read_image_list(folder_path)

Read image paths from the folder path

Parameters:folder_path – String Path to the folder containing the images
Returns:list of image paths
shuffle_GRAY_data()

Shuffles the lists to make them ready for the next batch

Returns:NOTHING

Temporal Stream

TemporalStream.3DConvMultiStream.trainMultiStream()

Trains a MultiStream 3D-CNN from scratch.

  1. Define Network Architecture
  2. Define loss and optimizer
  3. Load dataset
  4. Start training
  5. Store best performing model
Returns:Nothing
TemporalStream.3DConvTwinStream.trainTwinStream()

Trains a TwinStream 3D-CNN from scratch.

  1. Define Network Architecture
  2. Define loss and optimizer
  3. Load dataset
  4. Start training
  5. Store best performing model
Returns:Nothing
TemporalStream.3DResnet.train3DResNet()

Trains a 3D-Residual Neural Network from scratch.

  1. Define Network Architecture
  2. Define loss and optimizer
  3. Load dataset
  4. Start training
  5. Store best performing model
Returns:Nothing
TemporalStream.ResNetModels3D.longResNet101(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 101-layer long 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResNetModels3D.longResNet18(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 18-layer long 3D-Residual Neural Network from scratch, using 3D-Residual Blocks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResNetModels3D.longResNet34(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 34-layer long 3D-Residual Neural Network from scratch, using 3D-Residual Blocks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResNetModels3D.longResNet50Bottle(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 50-layer long 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResNetModels3D.resNet101(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 101-layer 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResNetModels3D.resNet152(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 152-layer 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResNetModels3D.resNet18(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 18-layer 3D-Residual Neural Network from scratch, using 3D-Residual Blocks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResNetModels3D.resNet34(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 34-layer 3D-Residual Neural Network from scratch, using 3D-Residual Blocks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResNetModels3D.resNet50(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')

Defines a 50-layer 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks

Parameters:
  • x – network input
  • keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
  • printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
  • type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns:

the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore

TemporalStream.ResnetLayers.LongResidual_block3D(incoming, nb_blocks, out_channels, downsample=False, keep_time_dim=False, print_Layer=False, type='B', downsample_strides=2, activation='relu', batch_norm=True, bias=True, weights_init='variance_scaling', bias_init='zeros', regularizer='L2', weight_decay=0.0001, trainable=True, restore=True, reuse=False, scope=None, name='LongResidualBlock3D')

A long 3D-residual block extended from the residual block described in MSRA’s Deep Residual Network paper. Full pre-activation architecture is used here. (This implementation is an extension of tflearn.layers.conv.residual_block)

Parameters:
  • incoming – 5-D Tensor [batch, numFrames, height, width, in_channels].
  • nb_blocks – 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
  • bottleneck_size – int`. The number of convolutional filter of the bottleneck convolutional layer.
  • out_channelsint. The number of convolutional filters of the layers surrounding the bottleneck layer.
  • keep_time_dim – ‘boolean’. Tells us if we will keep temporal dimension in this layer.
  • print_Layer – ‘boolean’. Tells us if we will printin this layer.
  • type – ‘char’. Tells us which downsampling strategy we will use.
  • downsamplebool. If True, apply downsampling using ‘downsample_strides’ for strides.
  • downsample_strides – int`. The strides to use when downsampling.
  • activationstr (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: ‘linear’.
  • batch_normbool. If True, apply batch normalization.
  • biasbool. If True, a bias is used.
  • weights_initstr (name) or Tensor. Weights initialization. (see tflearn.initializations) Default: ‘uniform_scaling’.
  • bias_initstr (name) or tf.Tensor. Bias initialization. (see tflearn.initializations) Default: ‘zeros’.
  • regularizerstr (name) or Tensor. Add a regularizer to this layer weights (see tflearn.regularizers). Default: None.
  • weight_decayfloat. Regularizer decay parameter. Default: 0.001.
  • trainablebool. If True, weights will be trainable.
  • restorebool. If True, this layer weights will be restored when loading a model.
  • reusebool. If True and ‘scope’ is provided, this layer variables will be reused (shared).
  • scopestr. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
  • name – A name for this layer (optional). Default: ‘DeepBottleneck’.
Returns:

5-D Tensor [batch, numFrames, new height, new width, nb_filter].

TemporalStream.ResnetLayers.global_avg_pool3D(incoming, name='GlobalAvgPool3D')

3D-Global Average pooling

Parameters:
  • incoming – 5-D Tensor [batch, numFrames,height, width, in_channels].
  • name – A name for this layer (optional). Default: ‘GlobalAvgPool’.
Returns:

2-D Tensor [batch, pooled dim]

TemporalStream.ResnetLayers.long_residual_bottleneck3D(incoming, nb_blocks, bottleneck_size, out_channels, keep_time_dim=False, print_Layer=False, type='B', downsample=False, downsample_strides=2, activation='relu', batch_norm=True, bias=True, weights_init='variance_scaling', bias_init='zeros', regularizer='L2', weight_decay=0.0001, trainable=True, restore=True, reuse=False, scope=None, name='LongResidualBottleneck3D')

A 3D-residual bottleneck block extended from the residual bottleneck described in MSRA’s Deep Residual Network paper. Full pre-activation architecture is used here. (This implementation is an extension of tflearn.layers.conv.residual_bottleneck)

Parameters:
  • incoming – 5-D Tensor [batch, numFrames, height, width, in_channels].
  • nb_blocks – 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
  • bottleneck_size – int`. The number of convolutional filter of the bottleneck convolutional layer.
  • out_channelsint. The number of convolutional filters of the layers surrounding the bottleneck layer.
  • keep_time_dim – ‘boolean’. Tells us if we will keep temporal dimension in this layer.
  • print_Layer – ‘boolean’. Tells us if we will printin this layer.
  • type – ‘char’. Tells us which downsampling strategy we will use.
  • downsamplebool. If True, apply downsampling using ‘downsample_strides’ for strides.
  • downsample_strides – int`. The strides to use when downsampling.
  • activationstr (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: ‘linear’.
  • batch_normbool. If True, apply batch normalization.
  • biasbool. If True, a bias is used.
  • weights_initstr (name) or Tensor. Weights initialization. (see tflearn.initializations) Default: ‘uniform_scaling’.
  • bias_initstr (name) or tf.Tensor. Bias initialization. (see tflearn.initializations) Default: ‘zeros’.
  • regularizerstr (name) or Tensor. Add a regularizer to this layer weights (see tflearn.regularizers). Default: None.
  • weight_decayfloat. Regularizer decay parameter. Default: 0.001.
  • trainablebool. If True, weights will be trainable.
  • restorebool. If True, this layer weights will be restored when loading a model.
  • reusebool. If True and ‘scope’ is provided, this layer variables will be reused (shared).
  • scopestr. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
  • name – A name for this layer (optional). Default: ‘DeepBottleneck’.
Returns:

5-D Tensor [batch, numFrames, new height, new width, nb_filter].

TemporalStream.ResnetLayers.residual_block3D(incoming, nb_blocks, out_channels, downsample=False, keep_time_dim=False, print_Layer=False, type='B', downsample_strides=2, activation='relu', batch_norm=True, bias=True, weights_init='variance_scaling', bias_init='zeros', regularizer='L2', weight_decay=0.0001, trainable=True, restore=True, reuse=False, scope=None, name='ResidualBlock3D')

A 3D-residual block extended from the residual block described in MSRA’s Deep Residual Network paper. Full pre-activation architecture is used here. (This implementation is an extension of tflearn.layers.conv.residual_block)

Parameters:
  • incoming – 5-D Tensor [batch, numFrames, height, width, in_channels].
  • nb_blocks – 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
  • bottleneck_size – int`. The number of convolutional filter of the bottleneck convolutional layer.
  • out_channelsint. The number of convolutional filters of the layers surrounding the bottleneck layer.
  • keep_time_dim – ‘boolean’. Tells us if we will keep temporal dimension in this layer.
  • print_Layer – ‘boolean’. Tells us if we will printin this layer.
  • type – ‘char’. Tells us which downsampling strategy we will use.
  • downsamplebool. If True, apply downsampling using ‘downsample_strides’ for strides.
  • downsample_strides – int`. The strides to use when downsampling.
  • activationstr (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: ‘linear’.
  • batch_normbool. If True, apply batch normalization.
  • biasbool. If True, a bias is used.
  • weights_initstr (name) or Tensor. Weights initialization. (see tflearn.initializations) Default: ‘uniform_scaling’.
  • bias_initstr (name) or tf.Tensor. Bias initialization. (see tflearn.initializations) Default: ‘zeros’.
  • regularizerstr (name) or Tensor. Add a regularizer to this layer weights (see tflearn.regularizers). Default: None.
  • weight_decayfloat. Regularizer decay parameter. Default: 0.001.
  • trainablebool. If True, weights will be trainable.
  • restorebool. If True, this layer weights will be restored when loading a model.
  • reusebool. If True and ‘scope’ is provided, this layer variables will be reused (shared).
  • scopestr. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
  • name – A name for this layer (optional). Default: ‘DeepBottleneck’.
Returns:

5-D Tensor [batch, numFrames, new height, new width, nb_filter].

TemporalStream.ResnetLayers.residual_bottleneck3D(incoming, nb_blocks, bottleneck_size, out_channels, keep_time_dim=False, print_Layer=False, type='B', downsample=False, downsample_strides=2, activation='relu', batch_norm=True, bias=True, weights_init='variance_scaling', bias_init='zeros', regularizer='L2', weight_decay=0.0001, trainable=True, restore=True, reuse=False, scope=None, name='ResidualBottleneck3D')

A 3D-residual bottleneck block extended from the residual bottleneck described in MSRA’s Deep Residual Network paper. Full pre-activation architecture is used here. (This implementation is an extension of tflearn.layers.conv.residual_bottleneck)

Parameters:
  • incoming – 5-D Tensor [batch, numFrames, height, width, in_channels].
  • nb_blocks – 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
  • bottleneck_size – int`. The number of convolutional filter of the bottleneck convolutional layer.
  • out_channelsint. The number of convolutional filters of the layers surrounding the bottleneck layer.
  • keep_time_dim – ‘boolean’. Tells us if we will keep temporal dimension in this layer.
  • print_Layer – ‘boolean’. Tells us if we will printin this layer.
  • type – ‘char’. Tells us which downsampling strategy we will use.
  • downsamplebool. If True, apply downsampling using ‘downsample_strides’ for strides.
  • downsample_strides – int`. The strides to use when downsampling.
  • activationstr (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: ‘linear’.
  • batch_normbool. If True, apply batch normalization.
  • biasbool. If True, a bias is used.
  • weights_initstr (name) or Tensor. Weights initialization. (see tflearn.initializations) Default: ‘uniform_scaling’.
  • bias_initstr (name) or tf.Tensor. Bias initialization. (see tflearn.initializations) Default: ‘zeros’.
  • regularizerstr (name) or Tensor. Add a regularizer to this layer weights (see tflearn.regularizers). Default: None.
  • weight_decayfloat. Regularizer decay parameter. Default: 0.001.
  • trainablebool. If True, weights will be trainable.
  • restorebool. If True, this layer weights will be restored when loading a model.
  • reusebool. If True and ‘scope’ is provided, this layer variables will be reused (shared).
  • scopestr. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
  • name – A name for this layer (optional). Default: ‘DeepBottleneck’.
Returns:

5-D Tensor [batch, numFrames, new height, new width, nb_filter].

Recurrent

Recurrent.SpatialRNN.trainSpatialRNN()

Trains a Spatial Recurrent Architecture from scratch.

  1. Load dataset
  2. Define Network Architecture
  3. Define loss and optimizer
  4. Start training
  5. Store best performing model
Returns:Nothing
Recurrent.DualStreamRNN.trainDualStreamRNN(testing=False)

Trains a Dual-Stream Architecture from scratch.

  1. Load dataset
  2. Define Network Architecture
  3. Define loss and optimizer
  4. Start training
  5. Store best performing model
Returns:Nothing