There are thumb indexes, also called a cut-in indexes (see also the images at Daumenregister). In contrast to this, there are also not-cut-in-indexes (that is, they are only printed, and leave a mark on the edge of the book).
The article is only available in German, but translating to English that entry says:
Handmarkes are usually mounted at the book on the front section, step-like printed, order words, letters or numbers. Usually they are coloured and help to find desired points, especially in reference works.
The Handmarke is not to be confused with the thumb index, but is often also called thumb index (English chapter thumbs).
The (Google) English translation "Hand stamp" for German "Handmarke" does not seem right for me and several other on-line dictionaries (as well as my printed ones) do not know "Handmarke" at all. And Handmarke is not thumb index.
What should I call this type of register in proper (British) English?