Auto Generated Documentation¶
Preprocessing¶
-
imageProcessing.
zeroCenterGrayData
(image)¶ Preprocesses image, using our Specular Removal Preprocessing strategy.
Parameters: image – Matrix containing an image Shape = [224, 224, 3] Returns: Preprocessed image
-
imageProcessing.
zeroCenterOFData
(image, datasetType)¶ - Normalizes and zero centers Optical Flow data
Parameters: - image – pixel matrix (224 x 224 x 3)
- datasetType – the sampling rate of our data
Returns: zero-centered and normalized image, based on sampling rate used
Spatial Stream¶
-
SpatialStream.VGG16_tensorflow.
trainSpatialStream
()¶ Fine tunes an improved VGG-16 from tflearn checkpoint.
- Define learning rate and epsilon
- Define Network Architecture
- Define loss and optimizer
- Load checkpoint weights
- Load dataset
- Start training
- Store best performing model
Returns: Nothing
-
class
SpatialStream.Spatial_File_Handler.
File_Handler
(feed_folder_path, dataType, nonfeed_folder_path, numTimesteps, datasetType)¶ -
get_GRAY_Batch
(batch_size, startPos)¶ Builds a new mini-batch through selecting images from a list of video frame paths.
Each mini-batch consists of an equal amount of Feeding and NonFeeding frames.
Each image is preprocessed using the Specular Removal Preprocesing strategy.
Parameters: - batch_size – size for the mini-batch
- startPos – the start position for the list of video frames
Returns: A preprocessed mini-batch, labels and the new start position
-
read_image_list
(folder_path)¶ Read image paths from the folder path
Parameters: folder_path – String Path to the folder containing the images Returns: list of image paths
-
shuffle_GRAY_data
()¶ Shuffles the lists to make them ready for the next batch
Returns: NOTHING
-
Temporal Stream¶
-
TemporalStream.3DConvMultiStream.
trainMultiStream
()¶ Trains a MultiStream 3D-CNN from scratch.
- Define Network Architecture
- Define loss and optimizer
- Load dataset
- Start training
- Store best performing model
Returns: Nothing
-
TemporalStream.3DConvTwinStream.
trainTwinStream
()¶ Trains a TwinStream 3D-CNN from scratch.
- Define Network Architecture
- Define loss and optimizer
- Load dataset
- Start training
- Store best performing model
Returns: Nothing
-
TemporalStream.3DResnet.
train3DResNet
()¶ Trains a 3D-Residual Neural Network from scratch.
- Define Network Architecture
- Define loss and optimizer
- Load dataset
- Start training
- Store best performing model
Returns: Nothing
-
TemporalStream.ResNetModels3D.
longResNet101
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 101-layer long 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResNetModels3D.
longResNet18
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 18-layer long 3D-Residual Neural Network from scratch, using 3D-Residual Blocks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResNetModels3D.
longResNet34
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 34-layer long 3D-Residual Neural Network from scratch, using 3D-Residual Blocks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResNetModels3D.
longResNet50Bottle
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 50-layer long 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResNetModels3D.
resNet101
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 101-layer 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResNetModels3D.
resNet152
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 152-layer 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResNetModels3D.
resNet18
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 18-layer 3D-Residual Neural Network from scratch, using 3D-Residual Blocks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResNetModels3D.
resNet34
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 34-layer 3D-Residual Neural Network from scratch, using 3D-Residual Blocks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResNetModels3D.
resNet50
(x, keep_time_dim=[False, False, False, False], printLayer=[False, False, False], type='B')¶ Defines a 50-layer 3D-Residual Neural Network from scratch, using 3D-Residual Bottlenecks
Parameters: - x – network input
- keep_time_dim – array of length 4. Used to select for how long we will keep the temporal dimension depth through the network. Each index in the array corresponds to a downsampling layer
- printLayer – array of length 3. Used to select which layers to print. Each index corresponds to a downsampling block
- type – Selects the downsampling type we want to use(‘A’, ‘B’ or ‘C’)
Returns: the network globalAveragePool layer, the name of the network, number of residualBlocks, k is not used anymore
-
TemporalStream.ResnetLayers.
LongResidual_block3D
(incoming, nb_blocks, out_channels, downsample=False, keep_time_dim=False, print_Layer=False, type='B', downsample_strides=2, activation='relu', batch_norm=True, bias=True, weights_init='variance_scaling', bias_init='zeros', regularizer='L2', weight_decay=0.0001, trainable=True, restore=True, reuse=False, scope=None, name='LongResidualBlock3D')¶ A long 3D-residual block extended from the residual block described in MSRA’s Deep Residual Network paper. Full pre-activation architecture is used here. (This implementation is an extension of tflearn.layers.conv.residual_block)
Parameters: - incoming – 5-D Tensor [batch, numFrames, height, width, in_channels].
- nb_blocks – 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
- bottleneck_size – int`. The number of convolutional filter of the bottleneck convolutional layer.
- out_channels – int. The number of convolutional filters of the layers surrounding the bottleneck layer.
- keep_time_dim – ‘boolean’. Tells us if we will keep temporal dimension in this layer.
- print_Layer – ‘boolean’. Tells us if we will printin this layer.
- type – ‘char’. Tells us which downsampling strategy we will use.
- downsample – bool. If True, apply downsampling using ‘downsample_strides’ for strides.
- downsample_strides – int`. The strides to use when downsampling.
- activation – str (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: ‘linear’.
- batch_norm – bool. If True, apply batch normalization.
- bias – bool. If True, a bias is used.
- weights_init – str (name) or Tensor. Weights initialization. (see tflearn.initializations) Default: ‘uniform_scaling’.
- bias_init – str (name) or tf.Tensor. Bias initialization. (see tflearn.initializations) Default: ‘zeros’.
- regularizer – str (name) or Tensor. Add a regularizer to this layer weights (see tflearn.regularizers). Default: None.
- weight_decay – float. Regularizer decay parameter. Default: 0.001.
- trainable – bool. If True, weights will be trainable.
- restore – bool. If True, this layer weights will be restored when loading a model.
- reuse – bool. If True and ‘scope’ is provided, this layer variables will be reused (shared).
- scope – str. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
- name – A name for this layer (optional). Default: ‘DeepBottleneck’.
Returns: 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
-
TemporalStream.ResnetLayers.
global_avg_pool3D
(incoming, name='GlobalAvgPool3D')¶ 3D-Global Average pooling
Parameters: - incoming – 5-D Tensor [batch, numFrames,height, width, in_channels].
- name – A name for this layer (optional). Default: ‘GlobalAvgPool’.
Returns: 2-D Tensor [batch, pooled dim]
-
TemporalStream.ResnetLayers.
long_residual_bottleneck3D
(incoming, nb_blocks, bottleneck_size, out_channels, keep_time_dim=False, print_Layer=False, type='B', downsample=False, downsample_strides=2, activation='relu', batch_norm=True, bias=True, weights_init='variance_scaling', bias_init='zeros', regularizer='L2', weight_decay=0.0001, trainable=True, restore=True, reuse=False, scope=None, name='LongResidualBottleneck3D')¶ A 3D-residual bottleneck block extended from the residual bottleneck described in MSRA’s Deep Residual Network paper. Full pre-activation architecture is used here. (This implementation is an extension of tflearn.layers.conv.residual_bottleneck)
Parameters: - incoming – 5-D Tensor [batch, numFrames, height, width, in_channels].
- nb_blocks – 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
- bottleneck_size – int`. The number of convolutional filter of the bottleneck convolutional layer.
- out_channels – int. The number of convolutional filters of the layers surrounding the bottleneck layer.
- keep_time_dim – ‘boolean’. Tells us if we will keep temporal dimension in this layer.
- print_Layer – ‘boolean’. Tells us if we will printin this layer.
- type – ‘char’. Tells us which downsampling strategy we will use.
- downsample – bool. If True, apply downsampling using ‘downsample_strides’ for strides.
- downsample_strides – int`. The strides to use when downsampling.
- activation – str (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: ‘linear’.
- batch_norm – bool. If True, apply batch normalization.
- bias – bool. If True, a bias is used.
- weights_init – str (name) or Tensor. Weights initialization. (see tflearn.initializations) Default: ‘uniform_scaling’.
- bias_init – str (name) or tf.Tensor. Bias initialization. (see tflearn.initializations) Default: ‘zeros’.
- regularizer – str (name) or Tensor. Add a regularizer to this layer weights (see tflearn.regularizers). Default: None.
- weight_decay – float. Regularizer decay parameter. Default: 0.001.
- trainable – bool. If True, weights will be trainable.
- restore – bool. If True, this layer weights will be restored when loading a model.
- reuse – bool. If True and ‘scope’ is provided, this layer variables will be reused (shared).
- scope – str. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
- name – A name for this layer (optional). Default: ‘DeepBottleneck’.
Returns: 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
-
TemporalStream.ResnetLayers.
residual_block3D
(incoming, nb_blocks, out_channels, downsample=False, keep_time_dim=False, print_Layer=False, type='B', downsample_strides=2, activation='relu', batch_norm=True, bias=True, weights_init='variance_scaling', bias_init='zeros', regularizer='L2', weight_decay=0.0001, trainable=True, restore=True, reuse=False, scope=None, name='ResidualBlock3D')¶ A 3D-residual block extended from the residual block described in MSRA’s Deep Residual Network paper. Full pre-activation architecture is used here. (This implementation is an extension of tflearn.layers.conv.residual_block)
Parameters: - incoming – 5-D Tensor [batch, numFrames, height, width, in_channels].
- nb_blocks – 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
- bottleneck_size – int`. The number of convolutional filter of the bottleneck convolutional layer.
- out_channels – int. The number of convolutional filters of the layers surrounding the bottleneck layer.
- keep_time_dim – ‘boolean’. Tells us if we will keep temporal dimension in this layer.
- print_Layer – ‘boolean’. Tells us if we will printin this layer.
- type – ‘char’. Tells us which downsampling strategy we will use.
- downsample – bool. If True, apply downsampling using ‘downsample_strides’ for strides.
- downsample_strides – int`. The strides to use when downsampling.
- activation – str (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: ‘linear’.
- batch_norm – bool. If True, apply batch normalization.
- bias – bool. If True, a bias is used.
- weights_init – str (name) or Tensor. Weights initialization. (see tflearn.initializations) Default: ‘uniform_scaling’.
- bias_init – str (name) or tf.Tensor. Bias initialization. (see tflearn.initializations) Default: ‘zeros’.
- regularizer – str (name) or Tensor. Add a regularizer to this layer weights (see tflearn.regularizers). Default: None.
- weight_decay – float. Regularizer decay parameter. Default: 0.001.
- trainable – bool. If True, weights will be trainable.
- restore – bool. If True, this layer weights will be restored when loading a model.
- reuse – bool. If True and ‘scope’ is provided, this layer variables will be reused (shared).
- scope – str. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
- name – A name for this layer (optional). Default: ‘DeepBottleneck’.
Returns: 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
-
TemporalStream.ResnetLayers.
residual_bottleneck3D
(incoming, nb_blocks, bottleneck_size, out_channels, keep_time_dim=False, print_Layer=False, type='B', downsample=False, downsample_strides=2, activation='relu', batch_norm=True, bias=True, weights_init='variance_scaling', bias_init='zeros', regularizer='L2', weight_decay=0.0001, trainable=True, restore=True, reuse=False, scope=None, name='ResidualBottleneck3D')¶ A 3D-residual bottleneck block extended from the residual bottleneck described in MSRA’s Deep Residual Network paper. Full pre-activation architecture is used here. (This implementation is an extension of tflearn.layers.conv.residual_bottleneck)
Parameters: - incoming – 5-D Tensor [batch, numFrames, height, width, in_channels].
- nb_blocks – 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
- bottleneck_size – int`. The number of convolutional filter of the bottleneck convolutional layer.
- out_channels – int. The number of convolutional filters of the layers surrounding the bottleneck layer.
- keep_time_dim – ‘boolean’. Tells us if we will keep temporal dimension in this layer.
- print_Layer – ‘boolean’. Tells us if we will printin this layer.
- type – ‘char’. Tells us which downsampling strategy we will use.
- downsample – bool. If True, apply downsampling using ‘downsample_strides’ for strides.
- downsample_strides – int`. The strides to use when downsampling.
- activation – str (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: ‘linear’.
- batch_norm – bool. If True, apply batch normalization.
- bias – bool. If True, a bias is used.
- weights_init – str (name) or Tensor. Weights initialization. (see tflearn.initializations) Default: ‘uniform_scaling’.
- bias_init – str (name) or tf.Tensor. Bias initialization. (see tflearn.initializations) Default: ‘zeros’.
- regularizer – str (name) or Tensor. Add a regularizer to this layer weights (see tflearn.regularizers). Default: None.
- weight_decay – float. Regularizer decay parameter. Default: 0.001.
- trainable – bool. If True, weights will be trainable.
- restore – bool. If True, this layer weights will be restored when loading a model.
- reuse – bool. If True and ‘scope’ is provided, this layer variables will be reused (shared).
- scope – str. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
- name – A name for this layer (optional). Default: ‘DeepBottleneck’.
Returns: 5-D Tensor [batch, numFrames, new height, new width, nb_filter].
Recurrent¶
-
Recurrent.SpatialRNN.
trainSpatialRNN
()¶ Trains a Spatial Recurrent Architecture from scratch.
- Load dataset
- Define Network Architecture
- Define loss and optimizer
- Start training
- Store best performing model
Returns: Nothing
-
Recurrent.DualStreamRNN.
trainDualStreamRNN
(testing=False)¶ Trains a Dual-Stream Architecture from scratch.
- Load dataset
- Define Network Architecture
- Define loss and optimizer
- Start training
- Store best performing model
Returns: Nothing