Is it fine to call an unknown lady on phone as 'madame', like 'Bonjour madame, je voudrais...'
or it is considered too archaic and should be omitted?
Both answers from @jlliagre and @ptit Xav highlight that it is common, courteous, and respectful to use madame or monsieur in a greeting in French. I agree.
You ask if it works the same way as ma'am in American English: not quite. It depends on how the speakers (including you) use ma'am. The use of ma'am is complicated by social factors, especially regional usage and cultural usage. In some variants it is just as common and respectful as using madame in French. Omitting it could be considered rude depending on the context, notably in the American South and in the military.
However, in your example of placing an order or reaching someone you don't know over the phone, I wouldn't open with "hello ma'am." I would assume a scripted translation if I heard that; it is not an idiomatic expression to me in AmE despite how common the term ma'am is in my dialect. One would likely use it in "yes/no ma'am" and "thank you ma'am" in the same conversation as a polite form of address or common courtesy. In French (as with the corresponding terms of address in many languages), I would use both "bonjour madame" and "merci madame." "?Hello ma'am" not so much. It does not work the same.
For other variants of American English the use of ma'am or sir is rare, unnecessary, bordering on old fashioned and antiquated, and it can even perceived as rude. Peers might find it too formal in the workplace. This public radio interview, "Please don't call me ma'am," shows one issue with how ma'am can be perceived to imply a certain age or marital status or detract from a greater honorific. There is a parallel in French and other languages with madame and mademoiselle. See Comment s'adresser à une femme dont l'état civil est inconnu ?