Context, dear boy. What is the context?
The first"only" meaning can hold if the context is (among others):
- what were you doing when you were supposed to do something else
- what [bad] things you just did
- what do we do
For example:
"I just said that he was a failure, not that he was a [insert curse word]."
"You just need to keep up the good work."
The second"recently" meaning can hold if the context is what happened in a recent timeframe:
"We just lost the football game!"
"They just stole my car!"