The r is never completely lost. When unpronounced, it remains in the sense it changes the pronunciation of the last "e".
A couple of words in your first list are verbs while all the ones in the second are nouns. This is no surprise as an ending in -er in infinitive verbs and (almost all) infinitives used as nouns (e.g. un déjeuner) is always pronounced [-e]. In that case the rule is then very simple.
Otherwise, -er might be pronounced [e], [ɛʁ] or [œʁ] and there is always an etymological/historical reason explaining why either pronunciation is used.
First, the [-e] ending also prevails for non verbs, around 75% are pronounced that way.
The remaining words in [-ɛʁ] are mostly foreign words, essentially from English.
The few ones not coming from English still ending in [-ɛʁ] are:
alter, amer, cancer, cathéter, cher, cuiller, der, enfer, éther, fer, fier, hier, hiver, hyper, imper, inter, mer, pater, sphincter, ver and vétiver.
Finally, there is a large set of foreign words, also mostly from English, with and ending pronunciation usually in [-œʁ] but sometimes not fixed and varying between [-œʁ] and [-ɛʁ]. For example dealer, fürher, hamburger, scooter, sniper, supporter, tanker, trader…
Note that when these foreign words are used as verbs, the standard French ending [-e] is used, e.g. : un dealer gives [dilœʁ] but "to deal" gives [dile], same for dumper, hacker, manager…