Native French speaker here.
In my experience, I have never intentionally made a difference between the pronunciation of "ces" and "ses" myself, I've never heard there was a difference, and I've never been able to tell a difference when other people spoke.
So, purely phonetically speaking, those two words are (again, in my experience) exactly identical.
To answer your question "So how do we know if we hear a ces or ses in this case?" more precisely, here are my two cents :
I do think (as other have mentioned) that "ses" is more or less the only choice, but you can't know only by ear, and context of the discussion wouldn't help. The disambiguation context here is purely cultural (imo) here. What I mean by that is that even though the sentence using "ces" is grammatically correct, no native speaker in their right mind would ever say it out loud. If they wanted to convey the meaning of "ces" (namely "those"), they would probably say "ces grands-parents là" (word for word, this gives "those grandparents there"). This habit is so common to allow for diambiguation that it feels like using "ces" is a mistake. Not a grammatical mistake, but an instance where the desired meaning will NEVER be properly conveyed in a discussion between native French speaker (from France. I don't know if it still holds for other populations).