This is a difficult one to answer because "cartoon" and the related adjectives you are asking about can be used in different ways.
A cartoon can refer to either:
- a single-frame, humorous drawing
- a short multi-frame strip, like the kind found in newspapers (longer strips are usually called 'comics' or 'comic-books')
- an animated tv or film feature
Traditionally, cartoons depict people or animals in humorously exaggerated ways - for example, people often have large eyes or disproportionate bodies; animals are often anthropomorphised. However, some artwork in comic books and animations eschews this style and attempts to be more realistic and/or feature anatomically correct characters.
Adjectives like "cartoony", or "cartoonish" tend to be used to describe the exaggerated style of drawing - that is, to describe a drawing or depiction which is not realistic. Even a cartoon could be described as "cartoony" if it uses that particular style.
So the answer to your question depends on what you are trying to say. If you are describing a piece of art that uses photos instead of drawings - perhaps a composite image - then arguably this is "a cartoon". You should perhaps say that it "uses photos rather than drawings".
If instead you are trying to describe a live-action film feature and state that it is not an animation, you should perhaps say that it "is a live-action film, not a cartoon (or an animation)".
If I have not covered what you are trying to describe please add detail to your question or in comments and I will try and address it.