Yes, it exists as a term. Yes, it would be understood in your examples a self-study course and "the effectiveness of self-study". It would not be the most natural term to use in either case, however. Well, at least in British English.
The act of studying something on your own with no supervision or guidance is referred to as teaching oneself, and someone who learned something in this way would be called self-taught. There's also the more esoteric term derived from ancient Greek, autodidacticism, meaning self-teaching, and one who does this or did this is an autodidact, but don't use those unless you're being very technical and have some reason to use such obscure language.
The act of studying alone as part of a course that is supervised or taught is usually independent study. Lots of university courses make clear their expectations for the amount and sort of independent study that students will need to undertake.
A course where you are taught without personal contact, through teaching materials provided by some organisation and possibly with some tutor support through telephone, email etc, has a number of terms available. It might be a distance learning course, or a correspondence course. Such courses usually have some degree of assessment and you get told whether you passed or not.
Self-study is a term that means what you intend, but it isn't actually as widely used, at least in British English, as other terms. Indeed, some would have to guess what it meant when they came across it, and might take it to mean study of oneself. So it's not wrong, but it might not be the best choice of terminology.