Yes, "When I first moved here" is perfectly correct in this case. It does not mean that you have moved there multiple times, though I can understand the confusion. When you say "When I first moved here" it means you're talking about something that happened very close to the time that you moved to a place. For example, let's say you moved there four years ago. Now, four years later, there are lots of pubs, and weekends are very interesting! But you're telling a friend that when you first moved there, meaning around the time of four years ago, there were not a lot of pubs, and weekends were boring.
In contrast, if you do mean to say that you moved somewhere multiple times and want to talk about the first time that you moved to that place (as opposed to the second or third time you moved back) you would say "The first time that I moved here". For example, you can say:
"The first time that I moved here, houses were very expensive so I rented an apartment. I moved away to another city for a few years after that. The second time I moved here, houses were cheaper, so I bought a house and settled down for good!
This is not the same thing as "When I first moved here", though, as I have illustrated. Hope that makes more sense!