It's not just you. The book Dog Whistle Politics actually talks about this! As some other folks have noted a similar thing happened with how the Irish were brought into whiteness.
I kinda feel like there's an intentional effort to push a wedge between the Black and Asian community but that's just me.