Outside of a couple tweaks here and there, the Xbox Series controllers aren't that different from the Xbox one controllers at all so if you have an Xbox one S controller, you don't really need to upgrade.
If you want to game at 4k, the Series X is your best bet and it also has a disc drive (which is important to some people). Otherwise, there's not much difference between the two and you'll enjoy both.