The French grammar book I'm reading has the following sentence:
Mes amis dont les enfants sont si mignons viennent en vacances avec nous.
My friends whose kids are so cute are coming with us on vacation.
What if I wanted to instead translate: "My friends, whose kids I hate, are coming with us on vacation"1. Would it be "Mes amis dont les enfants que je détèste viennent en vacances avec nous"?
1. (Presumably I want those bratty kids to be left behind!)
Not quite. It's important to note that the embedded clause is a full sentence:
Mes amis dont les enfants sont si mignons viennent en vacances avec nous.
Les enfants sont si mignons. The kids are so cute.
This will also be true of any other clause you want to embed.
So to decide what it should be, just construct the normal sentence for "I hate the kids" and insert it.
I hate the kids. Je déteste les enfants.
→ Mes amis dont je déteste les enfants viennent en vacances avec nous.
Your sentence should read:
Mes amis dont je déteste les enfants...
Unlike in English, the order in French within a "dont" clause is always:
subject --> verb --> direct object (with article)
In English, it's equivalent to "of whom" and sounds rather stilted to my native ears:
My friends of whom I hate the kids...
Another example is:
L'homme dont je connais la sœur...
The man whose sister I know...
Your attempt is not entirely ungrammatical, but it's a different construction in which dont is followed by a nominal group that makes this particular sentence semantically doubtful (“My friends, among whom are the kids I hate, are coming with us”). But this isn't what you had in mind anyway.
In fact dont doesn't always attribute the subject of the relative clause to its antecedent; it can also apply to the object. Unlike in English, the word order in the relative clause isn't affected. So, the sentence you're looking for goes as follows:
Mes amis, dont je déteste les enfants, viennent en vacances avec nous.
