Usually we use the definite article before "first", "second" and so on.
First, second and so on are acting as adjectives here. The article (or other determiner) is thus dependent in part on the actual noun, and as always on the overall meaning.
English is my first language.
That one is obvious.
English is the second language you learnt.
You only learnt one language second (obviously).
There are many advantages to learning a second language.
It doesn't matter what language it is, it could be different for different people, it's just a language.
However, there's an extra wrinkle here. Sometimes, someone can have several second languages.
Sounds contradictory, right? Well, a second language just means a non-native language, at least in some contexts. If someone is bilingual from early childhood due to the way they were raised, both of those native languages can be called first languages. Every language that they are not a native speaker of is a second language. When used in that sense, there is no such thing as a third language - though some use the terms in a sort-of joking way when they have one or more second languages in which they are more or less fluent, to describe those they can just about get by in. The ones they aren't fluent in might be called third languages, and the ones they dabble in as fourth languages. However, when second language is used in this way, there is no standard or universal (or even terribly widespread) understanding of third or fourth languages.1
Now, not every time you see first language or second language will be using it that way, but it's pretty widespread when you're talking about education.
1: By that categorisation, I probably only have a first language and some fourth languages... I had a third language once, but I'm too rusty now.