The phrase itself is polite enough, but it is often used in situations where the speaker is being impolite by making the comment (such as correcting a stranger on some trivial mistake). As a result some people interpret the phrase as impolite, even though it is not, itself, impolite.
An unstated assumption of a let you know construction is that your friend doesn't know, but perhaps should already know. In this case, it's possible (though not necessarily likely) that your friend would interpret the comment as suggesting that they need a reminder because of some personal failing.
If your goal is to make sure that your friend is aware of the information, and you are uncertain if they are or not, it may be more polite to emphasize the reason you think they may not have the information:
I don't remember if I told you or not, but the lesson will be at the library.