Off here is being used in adjective sense 2a from www.m-w.com:
2a : started on the way
You could consider it to be short for "moving off of the starting line of a race track" (because the race has begun).
In fact, the most idiomatic expression for this usage generally IS at a race track, when the race starts; you'll often hear the announcer say "Aaaaand they're off!"
So when you're off to a slow start, it means you have begun something, but are doing poorly in comparison to either your opponents or to your expectations. (Conversely, you might "get off to a good start or a fast start" if you are doing better than expected at the beginning of your effort.)