What you call a "filler" is in fact a discourse marker. And it is true, in spoken language, actually is very commonly used in this way:
Actually is often used in speaking as a discourse marker. We use it to indicate a new topic of conversation or a change or contrast in what is being talked about. We also use actually to give more detail about a topic. We do not use it to refer to time:
A: I suppose you’re going away this weekend?
B: Actually, I am going to stay at home. I’ve got a lot of work
to do on the computer.
[a customer (A) in a large bookshop is asking about books about
A: Could you tell me where your books on Austria are kept?
B: What kind of books?
A: Well, actually I’m looking for a book on skiing in Austria.
B: Er, yes, they’re in that corner over there. (Cambridge)
Just note that discourse markers are also used in formal language, too:
Discourse markers are words and phrases used in speaking and writing to 'signpost' discourse. Discourse markers do this by showing turns, joining ideas together, showing attitude, and generally controlling communication. Some people regard discourse markers as a feature of spoken language only.
Example Words like 'actually', 'so', 'OK', 'right?' and 'anyway' all
function as discourse markers as they help the speaker to manage the
conversation and mark when it changes.
Discourse markers are an important feature of both formal and informal
native speaker language. The skilful use of discourse markers often
indicates a higher level of fluency and an ability to produce and
understand authentic language. (BBCTeachingEnglish)