If you tested when you posted you wouldn't see it happening, it literally happens around 12:30pm everyday and returns to normal around 8pm-9pm sometimes later.
I know something isn't right with it doing that, especially at such specific times. I know it's not an equipment fault either since my husband as tested everything and even tested more than 1 modem.
Will run the pingplotter around 12pm today and post results once done. Although this issue isn't just effecting one server, once it happens it even shows the fluctuations on speed test sites, tried multiple sites just to be sure.