There's no rule, but there are patterns. For example, words with the same syllables tend to have the same pronunciation.
école, coupole, créole, agricole, vole are all /ɔ/
austral, hausse, cause, exhausser, fausse, gaussien are all /o/
binôme, biome, lymphome, dôme, diplome, atome are all /o/
comme, homme, consomme, dégomme, nomme, pomme, gomme are all /ɔ/
Of course there are exceptions, but if you're unsure you can try that and most of the times you won't be too far off.
However, you should pay attention to where the syllable is in the word, it may be important! With -omm- for example, it's an open "o" only when it's the last syllable. If not, it's a closed "o".
commère, hommage, nommer, consommer are all /o/
voler, abolir, absolu, cajoler, décoler, hologramme, oligarchie are all /o/
It's actually also a common pattern for a "o" syllable to be closed when it's inside the word and open when it's at the end.