It doesn't matter to me, because he happens to be a great musician who composes great songs.
Part of the power of Dylan's music is how he plumbs the depths of human experience and expresses it in often memorable ways, lyrically and musically.
This song is on Planet Waves, the same LP that features Forever Young (both versions). Hazel is really simple. But even here, Dylan expresses what may be universal yearnings, in this case the yearnings of every man contemplating the one he thinks may be the woman of his life or of every woman considering the one she suspects is the man of her life.
Dylan doesn't idealize the physical here, but the woman herself. He starts:
Hazel, dirty blonde hairI think it's true that when we look for mates, we look for people who won't shame us in public or among the other's friends or ours, among our families or theirs.
I wouldn't be ashamed to be seen with you anywhere
We look for someone we know that we would never shame because of, not just our love for them, but also our respect for them.
The other lines that particularly strike me, though I love the lyrics of the entire song, come in the bridge:
Oh no, I don’t need any reminders
To know how much I really care
But it’s just making me blinder and blinder
Because I’m up on a hill and still you’re not there