I have been thinking about this very question for a very long time. As the answer directly applies to me. I am fluent in 3 languages. And I was not born in an English speaking family. I am also a Canadian. Having lived here for 20 years (out of my 36) my main language of communication is English. While I am fluent in two other languages, one of which is the one my parents speak, the language I think in, write in (I am a writer/poet), read in, and consider my own, is English. I do have an accent, which does not make me less of an English speaker than anyone who doesn't. I consider myself a Native English speaker. As this is my first language - the one I can communicate most proficiently in, and my primary means of communication. That is, the language I can and do express myself best in. And even though I know and am fluent in two other languages, I not as proficient in them in reading, writing or verbal expression as I am in English. Even though one of them is the language I have spoken from childhood. This particular language, I can only speak in, and not to well at best, I cannot write or read in it properly at any time. And even when I do speak it, it is littered with english words and expressions.
Now, most people think that a Native Speaker of any language is someone who is "born speaking" it. Or whose parents speak it at home all the time. Conversely, if someone has learned the language later in life, even as a teen or a young adult, it is necessarily a second language, and thus not Native.
In reality, if you look at people who were first exposed to, say, English, as teens or young adults, most of them are as proficient in it if not more so than a lot of the so called "Native Speakers" who were born to it in the first place. One of the reasons why, is assimilation into the culture and society they are a part of. In other words, they want to be part of the culture they are living in, and have a healthy fear of being the 'outsider' who stands out and who society frowns on. Another, is that young people internalize language and the culture it is such an integral part of, very quickly. This is not so, however for most older adults.
Yet another reason is that the media they use and are steeped in every day, such as the internet, social media, music, books, movies, television etc... are in English, and the more proficient they are in the language and culture of English, the more they can get and give in and through these different media in their day to day lives. Including online friends and groups, real life friends off the internet, better understanding of other young people around them and the culture they are all part of.
So my answer to the question asked above is this: You state in your question that you have to use the English language daily, for at least 5 hours out of every day "I have to use English daily for at least 5 hours a day in my normal life." in your words, but you do not say in what language you think, and communicate best? And as you mentioned, culture is also important as part of language, as language itself is part of culture. How good is your understanding of the culture, which the English language is part of? I'll give you an example: In my own life, I know and understand the cultural values that English as a language is steeped in, I do not only speak the language, but also know and understand the different cultures that surround it (British, Canadian, American etc..). What this means is that I can relate to the mores and expressions in these cultures and people's daily experiences in them. In short, being a Native Speaker of any language is more than just being perfectly knowledgeable and proficient in it's grammar and punctuation. And having to use it a certain number of hours per day every day. It's also more than simply being born into a family who speaks that language, since, as you said, some Anglo Indian people, are "born" speaking English, but may not be versed in the accompanying culture - or as you have put it: "Anglo Indian families (and some purely Indian families too) who are born and raised here in India, speak English from birth and their first language is English and English is their primary means of communication although I cannot admit they fully abide by or understand all English cultural values like a British or an American.". Which means that if, for example they were to come to England, America, Canada etc... They would have to adopt to the new culture, which they are not familiar with, even though they may and probably are proficient in english.
So having said that: I think, the answer to this complicated question is more than just the dictionary definition of 'Native Speaker', which is what I have tried to explain above. I think that you and the other people you described are not Native Speakers of English. In order to be that, the understanding and internalization of cultural values is necessary as well. So, as a last example, if someone used an expression or a more while speaking to you in English, that people in say, England or America or Canada etc... would understand and know for what it is, or means, in the culture, and be able to respond to appropriately, and you only understand the English, but not the deeper cultural meaning, you are not a Native Speaker. Cheers