The Xbox still would have come out whether Sega released a new console or not. I feel they would have jumped in at some point. Anyway, I don't think Sega could compete at the time. They had two failed consoles in a row on their hand, and things weren't getting better. I doubt them shipping out a third console would have saved them. If anything it probably could have killed them outright. Three failures wouldn't look good. So I think it was right of Sega to move their work to publishing/developing games instead of making consoles.
If anything, I think they should have put more time into making handhelds. Their handhelds were popular and I am willing to bet if they shifted to a more mobile market, they could have at least stuck somewhat in the hardware market. And hey, they are kinda back with their classic consoles. So maybe that will interest them in making a new console one of these days.