Mon gâteau, ton gâteau, son gâteau, etc,
whether one refers to a man or a woman, because gâteau is a masculine noun.
The accord of the possessif adjective is made with the object in French and not the owner (contrary to the English rule).
As further examples victime, vedette, personne, sentinelle are feminine nouns whether one refers to a man or a woman. So, we would use:
Ma vedette, sa victime, etc,
no matter if we talk about a man or a woman.
Attention with some particular cases (related to your question) which may cause confusion.
Mon cher ami - Ma chère amie (My dear friend).
Mon chéri - Ma chérie (nearly equivalently to My darling).
**EDIT* (thanks @LaurentS)
In this context, mon amour is probably the best example.