What is the grammatically correct form for this sentence?
I would try(again) if I think it works.
Depends on what idea you want to convey.
If I thought it worked, I would try again.
Which, in a way, means that you won't try again because you don't think it works. For instance, in a discussion with a friend:
Why are you not trying again dude ?
Trust me, If I thought it worked, I would try again.
(Note: in an oral discussion, you could even emphasize would in order to better convey the idea that "in an hypothetical world where I thought it would be possible, I would indeed try again).
I will try again if I think it works.
Takes place in the future. If you happen to think it could work in the near future, then you'll definitely try it again.
For more read on the subject, see: Sequence of tenses