You can use many common nouns as pet names. In that case, you'll naturally use them with determinants corresponding to the grammatical gender of that noun, regardless of the sex of your beloved interlocutor. Among the classical pet names, quite a few are regularly used for both men and women : amour (m. at least in the singular, love), ange (m, angel), bébé (m., baby), cœur (m, heart), trésor (m., treasure)...
As I write this list, I realise that I do not know of a feminine pet name often used for men...
Of course, less classical pet names exist and can borrow from every grammatical category and every language. (Well, maybe not every grammatical category: I've never heard « quatorze » or « duquel » used as pet names, as lovely as that would be...)
A literary reference is inevitable.