Taste denotes preference—what you like, what sort of dress or book or cuisine you find pleasing.
Indian cooking is more to my taste than Thai.
Symonds had a taste for the exquisite in art and literature.
Her taste is impeccable: she abhors anything showy or vulgar.
Style denotes the manner in which something is expressed:
Milton's style, for all its Latinity, is violent and excessive, intended to bludgeon the reader into submission.
Chanel brought a new style to the runway, one based on youthful ease and athleticism.
Thus, when you say of a dress that “It’s not my taste”, you mean, literally, that it doesn’t suit your preference, you don’t like it; and when you say “It’s not my style” you mean, literally, that it does not present you as you wish to be seen.
But in conversation one is rarely so precise with language. And in this context what you probably mean is a combination of both ideas: “The style of this dress is not to my taste.” So in practice, the two expressions mean pretty much the same thing.
Similarly, when you are congratulating another woman you probably do not mean to restrict your admiration to either her taste in clothes or her appearance. You mean both, and either expression ought to serve. However; the expression “I like your style” usually means something different: unless you are speaking to a fashion designer, it expresses admiration not merely for the manner of dressing but for the manner of behavior—“I like the way you carry yourself, the attitude towards the world which your actions convey.” So you’re probably better off saying “I like your taste”.
Or you could simply say “That’s a really cute dress!”