Replica exchange molecular dynamics (REMD) becomes more efficient as the frequency of swap between the temperatures is increased. Recently Plattner et al. [J. Chem. Phys. 135, 134111 (2011)] proposed a method to implement infinite swapping REMD in practice. Here we introduce a natural modification of this method that involves molecular dynamics simulations over a mixture potential. This modification is both simple to implement in practice and provides a better, energy based understanding of how to choose the temperatures in REMD to optimize efficiency. It also has implications for generalizations of REMD in which the swaps involve other parameters than the temperature.