Fruit is often treated as non-count. So
He picked fruit from the tree.
It doesn't say how many, not what type of fruit, but we would probably suppose there were many pieces of fruit, perhaps of different types.
If you need to specifically say "one" then
He picked a piece of fruit from the tree.
This is good grammar, but slightly odd, since you would normally just say the type of fruit (since if he picked one you would know what type it was).
He picked an apple from the tree.
Don't say "picked up" That means "collected from the ground". Use "branch" only if that is relevant:
He picked an apple from the highest branch of the tree.
Your first sentence could be better phrased so things are described in order:
As he passed by the garden, the man picked an apple from a tree that stood there. He ate it as he walked on.