Is my understanding of the phrase a lot less now too in below context correct?
I go to stadium a lot less now too.
means
Now also, I go to stadium very rarely.
I go to the stadium a lot less now (too).
"too" would not be used with this sentence by itself. There would need to be a previous phrase, for example:
I go to the city a lot less now, and I go to the stadium a lot less now too.
Now if we substitute the meaning you asked about:
I go to the city a lot less now. Now also, I go to the stadium very rarely.
Basically it is OK, but "a lot less" means "a lot less than before". It could have been 200 days/year before and now it is 50 days/year.
However, "very rarely" is "hardly at all", and should be a very small number in any case.