« Pain amer » seems to be a common enough expression that movies and books are given this title, and it is the subject of this wordplay of Camus' in La peste:
La souffrance des enfants était notre pain amer, mais sans ce pain, notre âme périrait de sa faim spirituelle.
Yet it seems not to be common enough to appear under a separate heading in dictionaries, even Expressio.
In a Google Ngrams search, the first batch of uses (1700-1777) seem to be literal (e.g. « le seigle fait du pain amer »), and the next batch (1778–1857) seem to treat it as an expression already (e.g. « Manger un pain amer détrempé de ses larmes »).
What is the origin? It feels vaguely biblical. But many biblical expressions have calques in English since both are calqued from the original Hebrew or Greek idiom (cf. feet of clay / pieds d'argile), whereas I can't think of a ready-made equivalent for « pain d'amer » in English.