Is it really all that bad to like my brother's best friend?

Ok so I've liked this guy since I was in 6th grade and now I'm a senior. We have always been around each other growing up so it was inevitably that I would fall for him eventually. So now that I'm 17 and he's turning 20 I'm wondering if it's ok. Not only because of the fact that he's my brother's best friend but also because he has a 2 year old son. I mean to me it isn't a deal breaker. He's an amazing father and i admire that in him. But I feel as though that makes me seem more immature, although he is a child at heart. And well I'm just not sure of myself. We mutually started flirting when I entered 9th grade but I don't know if he means it. I know I myself mean it, I mean come on i get nervous around him, I shake, I can't speak, my body just shuts down around him. So please help me with this dilemma. Thank you in advance ^_^