https://en.wikipedia.org/wiki/Magnet_motor#Working_principle

>The force between permanent magnets is conservative as the magnetic field follows a potential, so that there is no work done over a closed cycle.

In other words, the magnetic field is shaped like a valley. You can extract a bit of energy from it by falling into the valley (energy you stored when you climbed out of the valley previously), but if you're simple circling the valley without ever reaching the bottom then you're not extracting energy, and certainly not an infinite amount of energy.