It's like watching a very complex movie, where the more you watch it, the more you catch about it.  Except generally 100x better in most cases. 

I've always felt that music has two extremes...  On one end, you have the sugar coated poppy stuff that takes no brain power to understand and enjoy (if you like that kind of thing).  On the other end you have music that takes a long time to understand and many years to truly appreciate.  The Flaming Lips are a very good example of the best of the latter.  It takes time to really soak in and once it does, it's like a throwing a light switch.  You just get it.

Of course, evolving tastes are always going to play a role as well.  Same with moods.  There have been bands that I heard in high school and hated that now I truly love.  There are also many bands that I loved in high school that now I can't stand. 

Music does not exist in a vacuum.  Everything about our lives at the moment of hearing flavor a song, making it slightly different for each person.  I love that about music, and it is also one of it's greatest shortcomings.