The quick answer is, some people might agree with you, but others will disagree or be confused if you say things that way.
Vegetarians and meat
"Vegetarian" is used by different people in different ways. A 2003 study of a 1994-1998 survey found that about 64% of the 334 Americans who called themselves "vegetarians" in that survey were eating some meat.
The understanding of the authors of that study was that
Although there is no universally accepted definition for the term vegetarian ... the term is nonetheless used to describe a dietary pattern characterized by an emphasis on plant foods and the avoidance of flesh foods (ie, meat, poultry, and fish).
So there are some people (like me) who consider "vegetarian" to mean never eating any meat, and others who feel one can be a "vegetarian" even without following a strictly vegetarian diet.
Meat, poultry, fish
The word "meat", as a class of foods, can also be ambiguous. Some people (like me) consider it to mean all animal flesh: red meat, poultry, seafood, everything. Others consider it to not include fish, and others consider it to not include poultry (chicken, turkey, etc.).
I suggest, when you speak or write, to try to avoid being ambiguous. Don't just say "meat". In the study I quoted above, the authors often wrote "meat, poultry, and fish", so that they could be sure that everyone would understand they meant all types of meat. If you want to talk about meat that's not fish, you could say something like:
I eat fish, but I don't eat other meats.
Conclusion
If someone never eats meat, poultry, or fish, and say they're a vegetarian, everyone will understand them perfectly.
However, if you simply say that you're vegetarian, without explaining the details, it's ambiguous, because people will have different ideas of what you mean. And many will argue that "vegetarian" is not the right word for your diet.