Both are direct orders, so depending on the context they may both be rude. You wouldn't go up to a stranger and tell them either "tais toi" or "ferme la" (or even "taisez vous" ou "fermez la") unless you wanted to aggravate them.
"Tais toi" on its own is not really rude; it's the presumption that you can give someone an order when you can't that's rude. A teacher can certainly tell their students "taisez vous" so that they make less noise, because the teacher has authority over the students, for example. But even in this case "Fermez la" would be rude, as "la fermer" is rude in any case, it's definitely not elevated language. If you'd like an analogy in English, "tais moi" means "stop talking" while "ferme la" means "shut up" (it's not a perfect translation obviously).
If you want a more polite manner of telling someone to stop talking, find a way to tell them in a non-imperative manner and add a "please", such as "Pourrais-tu te taire s'il-te-plaît", or if you use "vous" with the other person, "Pourriez-vous vous taire s'il-vous-plaît".
If you want to be even ruder, a common expression is "Ferme ta gueule", or simply for short "ta gueule" -- "gueule" being the word used for the mouths of most animals, so you're at the same time ordering them to stop talking and implying they're an animal. If you say that to a random stranger, prepare for at least a verbal fight. "Ferme ta bouche" is the same thing but a bit less nasty, about on the same level as "ferme la" I would say.