Despite what some might tell you, there is no definitive answer. It ultimately comes down to what you want to play and which console you feel more comfortable with.
There are definite benefits to buying a refurbished Xbox Series X if you’ve previously owned an Xbox console. You’ll get access to all of the games you previously owned and backwards compatibility with some of your old Xbox accessories, as well as the comfort of playing with a familiar controller.
One reason you might consider switching to a PS5 console is the PlayStation’s strong exclusive game line-up, with games like God of War, The Last of Us and the Final Fantasy VII Remake only launching on PlayStation consoles (although some of them are available on PC too).
But overall, it really does depend on your personal preference!