Englishpage.com describes the use of the past perfect as follows:
The Past Perfect expresses the idea that something occurred before another action in the past. It can also show that something happened before a specific time in the past.
The word "by" introduces the "specific time in the past" before which the event being discussed occurred. So it's quite common to find the past perfect used when some time frame is indicated with "by". For example, I might say something like this:
I was supposed to meet Rayna in the library at 1 pm. However, the bus was late, so I got there at 1:15. By that time, she had left already.
It's easy to see the relation between "before another action" and "before a specific time" in this example. The sequence of the two actions is:
- She left.
- I arrived.
The specific timing is:
- She left.
- 1:15 pm.
In your example sentence, the decline in the number of science students took place before 2008, and so the past perfect is appropriate to use in the sentence. It is not, however, mandatory. You could certainly write:
By 2008, the percentage of students choosing science subjects decreased markedly.
But the past perfect is preferable, as it's the more usual tense to use when discussing some event that happened before a specified time. If you wanted to rewrite the sentence without the past perfect, you could recast it:
In the period leading up to 2008, the percentage of students choosing science subjects decreased markedly.
Or give a specific starting point to the range of dates:
Between 2004 and 2008, the percentage of students choosing science subjects decreased markedly.
But if the rest of the passage goes on to discuss some consequence of the decline, then the past perfect would be best:
By 2008, the percentage of students choosing science subjects had decreased markedly. Consequently, in the fall semester of that year the school reduced the number of full-time lab assistants from five to three.
In this case, since the previous action directly affects the subsequent one, the relationship between them is more clearly indicated by the past perfect than the simple past.