Yes, "effective" would be correct, but the sentence needs some alterations, perhaps:
A study [reference] shows that an item-item approach is less effective in preventing a shilling attack than a user-user approach.
The word "User" should not be capitalised unless you are using it as a proper noun, which does not appear to be the case here.
I added the indefinite articles "an" and "a" to make it read more naturally. However, if you are referring to one particular item-item approach and one particular user-user approach then "the" would be the correct definite article to use instead:
A study [reference] shows that the item-item approach is less effective in preventing a shilling attack than the user-user approach.
It is probably a good idea to explicitly state that the intention is to prevent shilling attacks.
Finally, if you mention a study, you should give a reference to it so that readers can confirm its conclusion for themselves.