It is generally allowable to use all of these variations in construction, but they sound less natural, and this difference causes the listener to wonder why the obvious construction wasn't used. I think you understand that, but I'm confirming.
Plan is a very complex word, as it can refer to a state of mind, an action - the action of thinking ahead - or the product of that thinking. Combinations of these uses and different tenses can convey any of a richly diverse set of ideas, so you should stick to the rules to keep your meaning clear.
I won't attempt to list a bunch of examples, but if a speaker needs to convey particular subtle differences (or in certain cases, very significant ones), he would use one or more of these variations, stressing different words or syllables, and still sound perfectly natural. But the needed placement of all of these stresses can be extremely complex. Until you achieve a high degree of mastery, it is good advice to use the 'to-infinitive'; this is why people will say the other ways are just wrong. Early on, it's better for you to show that you know the rule, rather than show that you know it can be broken.