How To Modify The Seq2seq Cost Function For Padded Vectors?
Tensorflow supports dynamic length sequence by use of the parameter: 'sequence_length' while constructing the RNN layer, wherein the model does not learn the sequence after the seq
Solution 1:
This function already supports calculating costs for dynamic sequence lengths through the use of weights. As long as you ensure the weights are 0 for the "padding targets", the cross entropy will be pushed to 0 for those steps:
log_perp_list.append(crossent * weight)
and the total size will also reflect only the non-padding steps:
total_size = math_ops.add_n(weights)
If you're padding with zeros, one way to derive the weights is as follows:
weights = tf.sign(tf.abs(model.targets))
(Note that you might need to cast this to the same type as your targets)
Post a Comment for "How To Modify The Seq2seq Cost Function For Padded Vectors?"