In the sentence "He became captain of the team", 'captain' (noun) is the subject complement of 'became' and 'of the team' (a PP) is the object complement of 'captain'. Since complements function either as nouns or adjectives, is the PP acting as a noun or an adjective in the given sentence? Also, why can't we say that the PP modifies 'captain' here (For instance, 'captain' of what? --'captain' OF THE TEAM)?
He became [captain of the team].
The subject complement of "became" is the whole NP "captain of the team", in which the PP "of the team" is complement of "captain".
Complements don't "function either as nouns or adjectives", as you put it -- it's the other way round. Complement is a function that may be realised by an NP, AdJP clause etc.
The PP "of the team" is a complement, not a modifier, because the choice of preposition is specified by "captain", i.e. "of".