There are several places where we might ask someone to sit down, such as when we extend that invitation to a friend in our home, or to a business associate in a conference room.
Of these two (which are heard rather often):
is one of them more appropriate than the other to ask people whom you do not know too well to sit and have a talk? Would one of them be more appropriate than the other in a more formal setting?
In this context, I would say “please have a seat”. You're inviting the person to sit down, but not formally request that they sit down, merely offering the possibility.
“Please take a seat” is also possible, but slightly less deferential. It tends to be used in contexts where the person really should sit, for example by a doctor talking to a patient.
“Please sit down” and “please be seated” are more directive. For example, an air hostess might tell a passenger on a plane to “please be seated”, because the passenger really must seat while the aircraft is in motion. However, tone is key: it is possible to say “Please, sit down” (or perhaps that should be “Please! Sit down!”), with a pause after “please” and the right intonation, to convey that you are happy to see the person and cordially inviting them to sit down.
In general, "have a seat" is more polite; "please sit down" is more of an order. A teacher might tell her students, "please sit down". A host would be more likely to say to a guest, "have a seat".
As others have noted, tone and context is also important.
In a context where you are not ordering the other person around, requests like “Won't you have a seat?”, “Would you like a chair?”, “Will you sit down?”, and similar are more likely to be used than commands like “Have a seat” or “Please sit down”. (Note, “Please be seated” is a slightly less peremptory form of the latter.)
What you should say is quite context dependent; without a description of the particular context of interest, a specific answer isn't possible. Also, as StoneyB commented, your tone and how you behave as you talk to the other party are quite important. Most phrases can be delivered as politely or as rudely as the speaker chooses.
