"Madam" or "Ma'am" are polite words to use when you don't know a woman's name, but need a way to refer to her like a name. They are terms of address. It is fairly formal and not very common, as in situations when you talk to someone who you don't know, you don't often need to address them. The typical example is a shop worker speaking to a customer.
Would madam like cream in her tea? (This uses third person in a very humble way, it is not normal to speak like this)
It is also used (usually as ma'am) in schools, by schoolchildren to address teachers. (Not all schools use it. It is part of the culture of the school). It is also sometimes used in situations of strict hierarchy. A police constable might refer to her Inspector as "Ma'am" because the constable is a junior officer. The same is true in the armed services. In these contexts, it is nearly always reduced to "ma'am".
Now, one particular context in which you might need to address a woman, but not know her name is a brothel. The woman who organises the prostitutes doesn't give her name (after all, brothels are illegal) and so she is addressed as "madam" and as a common noun, "a madam" can mean a woman who runs a brothel. This doesn't make it offensive. You can say "Thank you, madam" without implying that she is like a brothel keeper.
It is also used ironically. A small girl who acts like she is superior to others is ironically called a "little madam". This also doesn't reference prostitution but is irony.
Advice: Try to avoid "sir" and "madam" as much as possible. This is difficult culturally because in some cultures it is over-familiar to use someone's name. This is not the case in English. If you know someone's name you should normally use it. Even in contexts in which you would not use a name in your language.
Ms Jackson, please come in. Your car is ready for you.
Hello Kathy. Is it okay for me to tidy your office now?
If there is no official hierarchy, and you don't know a person's name you don't need to use Madam or ma'am. You don't need the words in brackets.
Here is my passport [ma'am].
Excuse me [madam], I think I'm lost. Can you tell me how to go to the station?
If you are part the army or police, or meeting the Queen, then you will know when you need to use ma'am.