"Short" is an adjective, while "Shortly" is an adverb. Grammatically speaking, the proper thing to do is to use "short" to describe nouns/things, and "shortly" to describe verbs/actions.
In the first situation, It is a long story, but I'll try to put it as short as possible., you are describing the noun, namely, you're saying that you'll make the story short.
In the second situation, It is a long story, but I'll try to put it as shortly as possible., you are describing the action, namely, you're talking about how long it will take you to talk about the story.
In the final situation, "It is a long story, but I'll try to put it in short.", you are again describing the noun, namely, how the story will be put by you.
All of these three ways are grammatically acceptable, and their meanings are often conflated in everyday conversation.