Great point!! If you want to use the augmentation within the model, during test time you have to specifically freeze the layers. Else you can apply the augmentation before for train and validation set and not to the test set. Here's an example from the TensorFlow site (https://www.tensorflow.org/tutorials/images/data_augmentation).

Best wishes!!