"let's get down to business." does not indicate any particular friendliness. By expanding the contraction, as:
Now, let us get down to business.
there is a greater degree of formality, which might suggest lack of friendliness. I can see a lawyer saying this to a group of parties to which s/he is opposed, say. Even greater formality might be achieved with:
- Now, gentlemen, let us get down to business.
- Now, people, let us get down to business. (if the group is not all male)
Now, down to business.
strikes me as careless speech, on the edge of begin wrong, but these are the kinds of omissions that happen in actual informal speech. But I don't think it indicates any less (or greater) friendliness than the form with "let's".