"Of whom are you speaking?"
sounds indeed formal, if only because "whom" is used.
It attempts to stick to the French sentence "De qui parles-tu ?" which might also be translated to "Of whom do you speak?" but the word for word translation would be the broken "Of whom speak you?" There is then no point to try too hard to match English grammar and French one.
De qui parles-tu ?
is a correct literary form to ask this question. It is unlikely to be heard in a conversation because the subject-verb inversion has almost disappeared in spoken French. However, it might still be used by native French speakers in formal contexts, in the few cases where the tutoiement1 would also be used.
If the person who the question is directed to is not someone you would tutoyer, the still literary/formal form would be:
De qui parlez-vous ?
While quite rare in real life French, both of the previous questions are the most commonly found in written material.
In usual conversation, the subject-verb inversion is dropped so the question would be:
De qui tu parles ? or De qui vous parlez ?
De qui tu parles is also common in literature, just slightly less than the previous one.
A last simplification is to build the sentence as a regular subject-verb-complement one leading to:
Tu parles de qui ? or Vous parlez de qui ?
This last form is certainly the most common in relaxed colloquial conversation as it is the more natural to native French. It is slowly making it to books but still lagging behind the more formal variants. Here is a Google NGram graph of the various forms, with quoi used instead of qui as the former is more common.
So to answer to your question, French does have various grammatical constructions that make a sentence formal or colloquial, but they do not particularly match English ones. The examples you found are formal in both languages but that's more by accident than anything else.
You are right considering "Qui parles-tu de ?" as incorrect. The reason it is wrong is that the preposition "de" needs to be followed by something in French, unlike "of".
1 While the second person commonly used in formal French is the plural, even when speaking to a single person (vouvoiement), the tutoiement is not by itself turning a sentence to a standard or colloquial register. There are a few situations where the speech is formal but the tutoiement is nevertheless used because that's the way you address to a person. That person might be a child or someone belonging to your family, company, or club. That might even be God, a rare case where tu has survived in English ("Plus près de toi, mon Dieu" for "Nearer, My God, to Thee" or as Minty evoked in a comment "Tu ne tueras point" / "Thou shalt not kill").