The statement is indeed ambiguous, but understandable. In the absence of other context, to ditch X for Y would overwhelmingly mean, as you surmise, to abandon or discard X and substitute it with Y, in informal registers.
We ditched class for opening day.
She couldn't believe her own sister had ditched her for a boy.
It's well past time to ditch Adobe Flash for HTMl5.
This isn't the only way to interpret for with ditch, however—it is one of the most flexible prepositions in the language. For example, the object of for may represent a purpose or cause for the act of ditching—
I watched Steve blow his brains out because you went and ditched him for your cause! (The Last Hunter, 1980)
It's also very common for it to refer to a scope or duration for the ditching—
She said okay, and then he ditched her for 12 years. (The Women of Doctor Who, 2012)
Fans noticed Mel B ditched her eye patch for the concert return. (from Noise11.com, 2019)
(above film/TV references via english-corpora.org)
In your example, for is used roughly like concerning or in respect to, referring to the intended purpose of the original meeting. It is a compressed way of saying
We were supposed to meet for breakfast, but you ditched me.
You ditched me and our plans for breakfast.