I'm having a little trouble finding resources on word order here.
For instance, turn this into an inverted question:
Il n'aurait pas se réveillé.
In fact I don't even think I have the declarative right. Maybe I meant this:
Il ne se serait pas réveillé.
In this case, answers in English are appreciated for max clarity.
My best guess:
Ne se serait-il pas réveillé?
Pronominal verbs use exclusively être as an auxiliary verb for composed tenses. Therefore, independently of the word order, your first phrasing can't be correct.
Your second guess was right. To augment your level of confidence in it, you might want to proceed step by step. To avoid potential confusion due to homophones “est” and “ai”, best is to also consider the 1st person.
Start with present indicative for the auxiliary verb:
Je me suis réveillé.
Il s'est réveillé.
Add negation:
Je ne me suis pas réveillé.
Il ne s'est pas réveillé.
Turn this into question form, with subject–verb inversion:
Ne me suis-je pas réveillé ?
Ne s'est-il pas réveillé ?
Switch to conditional:
Ne me serais-je pas réveillé ?
Ne se serait-il pas réveillé ?
You may of course consider different paths. Yet, your final guess was right.
Your guess is correct if you mean "Would he not have woken up?"
Il n'aurait pas se réveillé
is indeed incorrect.
Il ne se serait pas réveillé
is much better and the interrogative form is indeed also correct:
Ne se serait-il pas réveillé ?
If you want mean: "He wouldn't have woken up."
"Pronominal verbs" have to be conjugated with the auxiliary verb "être" and not "avoir".
Thus, you are correct with :
Il ne se serait pas réveillé.
Ne se serait-il pas réveillé?
For more information about conjugating past participle with pronominal verbs here is French article from the CCDMD (Subject which is indirectly related to the question).
