The anime is an excelent adaptation all things considered, you're fine! the VN is absolutely the one that leans more towards the dating sim fantasy vibes anyway.
I would argue S;G0 (the anime sequel) even fixes some issues I had with the VN counter part.