I don't think there are grammatical issues with such sentences. Avec mon frère is almost equivalent to moi et mon frère so it is the way this sentence is generally understood. Of course the context can possibly lead to the alternative meaning you expect in English.
When the phrase is put in apposition like this:
Avec mon frère, on joue souvent à ce jeu.
there is even less ambiguity.
While the pronoun on is quite flexible in French, this usage is also common with nous:
Avec ma femme, nous montons dans un taxi.
Using on (or nous) also makes clear both are equally acting.
Je m'amuse bien avec mon frère. I might be the only one having fun.
On s'amuse bien avec mon frère. Both of us are having fun.
Finally, beware that many native French people will tell you using on instead of nous is familiar, lazy or even incorrect while the fact is on has essentially replaced nous (when nous is a first person plural subject) in all language levels of modern spoken French outside very formal ones.