I think that the phrase of the question's title:
Je doute qu'il étudiât
is erroneous.
The tense of the subjunctive (subjonctif), in a que-clause, is usually controlled by that of the main verb introducing it. The normal sequence is
Primary sequence
Verb introducing Subjunctive
{Present, imperative, future, perfect with have, future perfect} =>
Subjunctive
Present or Perfect
Historic sequence
Verb introducing Subjunctive
{Imperfect, past historic, perfect without have, pluperfect, conditional (perfect)} =>
Subjunctive
Imperfect or Pluperfect
So if the tense in the main phrase is in present it should be, depending on the time sequence, either present subjunctive or perfect.
Je crains qu'il ne vienne = I am afraid that he comes/is coming/will
come/will be coming/may come/may be coming.
Je crains qu'il ne soit venu = I'm afraid he has come/has been
coming/will have come, may have come.
Thus,
I doubt he was studying.
is translated by
Je doute qu'il ait étudié.
If the verb in the main sentence is, e.g., imperfect,
Je doutais qu'il étudiât
then
I doubted that he was studying.
But, Imperfect Subjunctive is used only for literary purposes.
In modern French it would rather be
Je doutais qu'il étudie.
Similarly, instead of the Pluperfect Subjunctive
Je craignais qu'il ne fût venu = I was afraid that he had come
the following structure is normally used
Je craignais qu'il ne soit venu.
Based on my own experience with colloquial French, one usually encounters the Present and Perfect Subjunctive of the French language. Even in academic world, at least in my field: mechanics, mathematics, physics and the like, I barely encountered the so-called literary tenses of the Subjunctive if not at all. Of course in law studies, for instance, may be the situation is different. Nevertheless, it is good idea to know also the usage of the other two forms in order to fully appreciate classic French literature.
Le roi tenait à ce que son autorité fût respectée (St-Exupéry, Le
petit prince).