Unexpected Increase In Validation Error In Mnist Pytorch

I'm a bit new to the whole field and thus decided to work on the MNIST dataset. I pretty much adapted the whole code from https://github.com/pytorch/examples/blob/master/mnist/main

Solution 1:

Long story short: you need to change item = self.X[idx] to item = self.X[idx].copy().

Long story long: T.ToTensor() runs torch.from_numpy, which returns a tensor which aliases the memory of your numpy array dataset.X. And T.Normalize()works inplace, so each time the sample is drawn it has mean subtracted and is divided by std, leading to degradation of your dataset.

Edit: regarding why it works in the original MNIST loader, the rabbit hole is even deeper. The key line in MNIST is that the image is transformed into a PIL.Image instance. The operation claims to only copy in case the buffer is not contiguous (it is in our case), but under the hood it checks whether it's strided instead (which it is), and thus copies it. So by luck, the default torchvision pipeline involves a copy and thus in-place operation of T.Normalize() does not corrupt the in-memory self.data of our MNIST instance.