what does "on a note of" mean in this sentence:
Syria talks end on a note of uncertainty, U.S. frustration with Russia
It's just a trivial metaphoric usage deriving from musicology. Classical music in particular often ends on a certain note/chord (which repeatedly either occurred, or was "suggested" but never actually played until the final "resolution" of the piece).
The metaphoric usage often occurs as [a speech, etc.] ending on a high/low note, which is usually a positive/negative reference to either the content or style of delivery (nothing to do with tonal frequency).
A closely-related usage is "The US set the tone of the talks by starting with a robust denunciation of the Russian position".
NOAD has a dictionary entry for high note, saying:
high note (n.) a successful point in an event or period of time : he wants to end his managerial career on a high note.
In addition to high notes, there can be low notes, bad notes, ominous notes, and notes of uncertainty. Under its entry for note, NOAD says:
note (n.) a particular quality or tone that reflects or expresses a mood or attitude : there was a note of scorn in her voice | the decade ended on an optimistic note.
So the headline you are quoting indicates that the mood at the end of the talks was one of uncertainty – everyone is unsure of whether or not any good will come out of the talks.
This phrasing can be used in a variety of contexts; it seems to be often used to describe the overall mood at the beginning or end of some event.