Try Linguee when you want to translate technical phrases. It's a database of bilingual texts in various language pairs, including French and English.
There is one hit for hanging punctuation: “ponctuation hors justification”, found in a Quark, Inc. brochure. I don't have an official reference to offer, but several manufacturers of typographical software (Quark, Adobe) have used the term, as have books about CSS, so I would guess that it is indeed the standard translation.
The phrase “ponctuation hors justification” gets a decent number of hits on Google, unlike “ponctuation dans la marge” (which has a single relevant hit), “ponctuation pendante” (which isn't much better, in spite of its obviousness as a literal translation), or “ponctuation suspendue” (most of the hits are unrelated).
The question was asked in passing on the typographie mailing list, with no conclusive answer.