Yes, that's exactly how you should use it.
"For as long as I can remember" is a literal phrase, and literally means for as long as you can remember. However it may be used either to refer to how far back in time you can remember, or it may refer to the fact you cannot remember an example of a time when it was untrue.
In this particular case, I suspect your confusion comes from the fact that "As long as you can remember" may go back further than the first time you had a relationship with a girl. Don't worry, that doesn't exclude the use of this phrase.
The thing to note, therefore, is that "For as long as I can remember" to a certain extent "For as long as it has been relevant".
Two example uses
For as long as I can remember, I have had trouble with women
The reader will not assume that you had trouble with women as a child, but that you are referring to the earliest adult relationship you can recall. In this context, you are saying that all instances of relationships that you can remember, have been difficult.
For as long as I can remember, I've been able to drive
In this context you are stating that you cannot remember a time when you were unable to drive.
Your confusion seems to be because you're using the phrase in the first sense, which sounds strange if you think of it in the second. Both uses are valid.