Should I say "a boy's name" or "a boy name"? and if both are correct, then what is the difference?
A boy's name is the usual way of saying the name of a boy, using the possessive form of the noun boy.
However, nouns are frequently used as adjectives. When they are used this way, they are known as attributive nouns or noun adjuncts (and by other descriptions as well). Typical examples are stable door, shoe lace and eye shadow.
If you were to say that, for example, Robin or Terry or Chris, sounded to you like boy names, you would be using the noun boy attributively. People do this all the time.
Note that while many nouns are used this way in standard expressions, boy name is not idiomatic. Although people would know what you meant, your teacher would probably mark it as incorrect.