I want to say "size of vectors" but instead of using "of" I want to use something like "vector size" but in the plural form what should I say? Is "vector sizes" right form?
This changes depending on what exactly you are referring to. If you are referring to the multiple dimensions of a single vector, then vector's size would be appropriate since the size is a singular term for the values of each of the dimensions of the vector.
If instead you are referring to the sizes of multiple vectors then the vectors' sizes could be used to refer to that. However, "vector sizes" would also be acceptable as a replacement in this case because it would be used to refer to the collective set of vector sizes recorded from an implied set of vectors (this drops the possessive)
"Size" is problematic, since there are more precise words that you should be using.
A vector has (by definition) a direction and a magnitude. The magnitude is the "size" of the vector. In particular situations there may be more precise words to use (eg "length" of a displacement vector, or "speed" of a velocity vector).
Vectors also have a number of dimensions. A vector may be represented by two, three or more numbers, giving two, three or more dimensions.
The attributive form sounds clunky. The "of" form flows much better. I would prefer "magnitudes of vectors". For example.
The magnitudes of vectors a and b are 5 and 4 respectively. The angle between them is 60 degrees. Evaluate a . b.
(answers in the comments)