Packet loss and latency problems random times during the day

I have had this issue on and off for quite some time.  It seems that random times during the day, latency to the first node beyond my router will spike, sometimes to crazy amounts.  When I am experie...