The terminology for possessives is highly variable, so there is no single correct answer. Still, I'll give it a shot.
In "Whose hat is this?", whose is the interrogative possessive determiner. "Determiner" in that it's analogous to some in "some hat", every in "every hat", and so on; "possessive" in that it indicates a possessive, analogously to my in "my hat" (which is also a possessive determiner); and "interrogative" in that it expresses a question, analogously to who in "Who is this person?" All three of these facts affect its grammar.
In "Whose is this?", I think the situation is much the same, except that hat (or thing) is left implied; just as we can say "She ate some" instead of "She ate some cake" if the cake is obvious from context, we can say "Whose is this?" instead of "Whose hat is this?" if everyone can see that it's a hat. According to some analyses, this usage is still a determiner; according to other analyses, a determiner changes into a pronoun when it does this.
My dialect does not have *"Whose is this hat?" — I'm not sure if there are other dialects that do have it, or if it's strictly a non-native-speaker-ism — but if it did exist, then I suppose whose would be an interrogative possessive adjective or pronoun, analogously to mine in "This hat is mine."