When you say that something is "out of line," it means, generally speaking, that you think something is wrong with it. It doesn't really mean anything more specific than that (unless you're using the phrase to literally mean that something is not lined up in the physical sense).
The phrase is most often (in my experience) used to describe bad behavior - you might say "that's out of line" to somebody to mean that the person has behaved badly or done something offensive.
A price being "out of line" isn't a particularly common use of the phrase, but it would suggest that something is wrong with the price. Without further context, it's impossible to say whether it means that the price is too high or too low. If a known cheapskate were to open up a menu at a restaurant and exclaim, "these prices are way out of line," that would suggest to me that he thinks they are too high. A business consultant hired by a restaurant, on the other hand, may well use the phrase to mean that the prices are too low and that the restaurant should be charging more.
Disclaimer: my knowledge is American English only.