How would you say?
- Referee! That was a penalty!
- Referee! That was penalty!
It is because, in this case, "penalty" is considered as an adjective?
This sentence seems the most natural in the context of a soccer match:
Referee! That was a penalty!
"Penalty" is a countable singular noun, and because of this we place an indefinite article before it.
Example of such usage:
"But he definitely tripped Andre up, so the referee decided that was a penalty." (Express, November 2015)
We can also say
Referee! That was the penalty!
Here, we use the definite article the before "penalty", indicating that the speaker has in mind a particular penalty, and the referee knows (or is able to deduce from the context) what particular penalty the speaker is talking about.
The word "penalty" can also be used in a noncount sense:
That time I hit an entire sleeve of balls into the pond and finished the hole with a Slazenger range ball -- was that a violation?
Trust me, that was penalty enough. (source)
Since it's used in a noncount sense here, it takes no article.
Merriam-Webster's entry on penalty makes a mention of the word's non-count sense
[noncount] They allowed him to pay back the money without penalty.
Hence, I guess that the sentence
Referee! That was penalty!
is also passable, but it has a meaning that is different from the first sentence. Trying to construe a situation in which this statement would fit, I imagine a referee who had made a decision that was formally neutral but in essense hampered the team's ability to play. A player spoke these words, challenging the referee's decision by describing it as "penalty".
An example of this "defining use" of the phrase:
I inquired of the Customs department why that extra fifteen per cent was imposed, and they said that that was penalty. (Official Report of Debates, House of Commons, volume 175, 1926)