Anime can be and often is escapism. Though that doesn't mean there isn't anything else to it.
Escapism isn't such a bad thing anyways, as long as you don't take it too far.
It's bad to completely flee from 'handling the world', but it's probably a good thing for your mental health to take breaks every now and then. (Of course this depends on your personality and situation in life.)
The thing to avoid is when you spend a lot of time doing something and then afterwards just feel empty.
This shouldn't happen with anime as long as you're passionate about what you're watching and aren't just watching out of habit.
I feel this happens more when people watch random youtube videos and stuff like that.
If you don't believe in life after death then in some sense living is an act of escapism :)
I don't believe anime OFTEN is escapism. Sure, it can be but often is? No. First let's look up what escapism is.
escapism
/ɪˈskeɪpɪz(ə)m,ɛˈskeɪpɪz(ə)m/
noun
the tendency to seek distraction and relief from unpleasant realities, especially by seeking entertainment or engaging in fantasy.
IF you're referring to yourself and only to yourself, saying that you often watch anime just to seek distraction and relief from unpleasant realities then you maybe correct but saying that many people are like you is INCORRECT.
I believe that most people watch anime because they enjoy it and not just to hide from their realities.
Back to the main question. I think people assumes that because of your background. If they know that you have a lot of problem and your relationships with your family or people in general are not good and you spend most of your time watching anime then they might think that anime is a form of escapism for you. If you sorted out those problems and have good relationships with people (assuming that you don't) then I don't think people would refer to you watching anime as escapism rather they might think it's just your hobby and something that you enjoy.