In many accents of French, the letter "o" has two pronunciations that can be written in IPA as [o] and [ɔ]. Neither vowel is the same as the vowel in the French word la, which is transcribed in IPA as [a].
Neither vowel is the same as English "toe" either. The English word "toe" tends to be pronounced as a diphthong or(or gliding vowel) that ends with a sound like a "w" or short "oo" sound, while French vowels in most accents are monophthongs i.e. they do not change from start to finish and don't have a final glide. But I think that English "toe" is a better approximation for both vowels than English "pah".
I would say that making a distinction between [o] and [ɔ] is not as important for a learner as making a distinction between [ɔ] and [a], because the distinction between [o] and [ɔ] is not consistent across all accents of French: different accents use one or the other sound in the same position. Most accents use [ɔ] in words spelled with "o" followed by -mme, -nne or -tte, such as homme, bonne, grotte. Some accents in the south of France have no contrast between the sounds: in such an accent, the use of [o] vs [ɔ] is based entirely on the surrounding sounds. Even in accents where [o] and [ɔ] can contrast, the use of [o] vs [ɔ] is often predictable based on the context.
One context where the distinction may be neutralized is in word-final position (at the end of a word with no following consonant sound). Accents from both the north and south of France often only use [o] and never [ɔ] in word-final position. In an accent like that, Po could only be pronounced [po]. But in an accent that allows [ɔ] at the end of a word (for example, I guessin Belgian French or Swiss French), Po couldmight be pronounced [pɔ].
The distinction between [o] and [ɔ] (and likewise [e] and [ɛ], and [ø] and [œ]) in non-final syllables tends to be much less clear than the distinction in final syllables, so I don't know whether you should even bother trying to remember which words have [ɔ] vs. [o] in non-final syllables. As a non-native speaker, I remember being taught to use [ɔ] in general for "o" in non-final syllables, but I don't know how much reason there is to follow that convention. An answer to the previous question Do I have to learn /o/ or /ɔ/ separately? made the reverse recommendation (user21018's answer there says non-final open syllables tend to have [o]).