The more I look, the more it seems port forwarding isn't needed but port triggering of some kind. You don't need the traffic forwarded to a specific PC, but rather open the firewall for all the computers on your network. See here for triggering instructions.
I read up on it and it looks like port triggering would help with connection stability. I'm stuck on the outbound port number though. I can't provide a range of ports like I can with port forwarding. I need one port number and I don't know what to use.
Okay I set up port triggering. I set up a rule for every one of the ports listed except for the range (12000-29999) since I need a single outbound port number to trigger it. That one should not matter though since it just handles voice.
Waiting now to see if I get disconnected. Wish me luck!