There's not much difference. The first has a simple literal meaning.
The second says that he is "sitting" so we suppose that he is not doing much. He isn't standing at the bars protesting his innocence or being interviewed by the police. He isn't meeting his lawyer to plan his defence. He is just sitting there.
It isn't a particularly common phrase. There is a cliché "He is languishing in jail". To "languish" means "to fail to make progress". So this means that he is stuck in jail, can't get out, can't do anything. Your phrase has something of the same implication:
I've been sitting in jail for a week now, and I think the system has forgotten about me. The police haven't even interviewed me since Tuesday. I haven't seen my attorney since Monday. Does any one care? How much longer will I be sitting here.
So "sitting" has a rhetorical effect, but it doesn't change the basic meaning.