The question sentence
Can you tell me what this photograph is of?
is correct, and it represents natural spoken English. You might say it when you are looking at a photograph and you have some curiosity (or doubt) about what the photo depicts.
Do you know what their photographs are of?
is correct, and it represents natural spoken English.
One example of a sentence of this type would be this: two friends, call them Elaine and Jerry, have been invited by a couple to view their photos. Jerry and Elaine really don't want to go, and one of them asks
Do we even know what their photographs are of? Because if they are of their baby, there is no way I am going. I've seen their baby twice already and that is more than enough for me.
The reason the questions may sound weird have nothing to do with their ending in a preposition. Indeed, the Oxford Dictionary Online has a short article about Ending sentences with prepositions and explains how natural it is to do so.
The reason the sentences (both question and non-question ones) sound weird is because of is/are of. We are used to these when they are spread apart. For example, in such such sentences as Little boys are made of... and in the question form What are little boys made of?
It is when a sentence or phrase ends in is/are of (side-by-side) that it sounds weird. Because there are not many expressions that require this combination. But your examples regarding photographs is one time it is used. It sounds weird to you because you aren't used to it.
I pointed out in a comment that
What this photograph is of is of my brother.
This sounds even weirder, because it contains two instances of is/are of in a row. (Note that the sentence does not end with a preposition.)
My photograph is of a skyscraper.
My research paper is about a skyscraper.
Does the is about sentence strike you as weird? If your answer is no, it's because you have encountered is about a lot more times than you have is of.
Last, the two questions:
What is your research paper about?
What skyscraper is your research paper about?
are also perfectly natural question constructions.