If I'm talking to a group of people and I want the phone number of each person, how would I phrase that? Is "I want all of your phone numbers" correct?
Your wording isn't incorrect, and would most likely be understood the way you mean it, but it is slightly awkward. It could mean you want all the numbers that each person has (landline, cell, etc), whereas I'm guessing you probably want one number from each person. In that case, you could ask for "a phone number from each of you".
In addition, "I want" has the feeling of a command, which may be what you want, and in a very informal setting is probably fine, regardless. If you want to word it a little more politely, you might use "I would like a phone number from each of you." Alternatively, you could sidestep the reference to yourself by using a polite command, such as "Please write down a phone number at which you can be reached." "You" here would be understood from context to refer to each member of the group individually and not the group as a whole.