I actually wrote a little blurb on this precise idea a year ago:
Music is the most abstract form of artistic expression, and thus, the most ambiguous. Visual media -- film, painting, sculpure, photography etc. -- usually ultilize some sort of directly representational expressive technique. That is, visual expression corresponds to the "real world" without being mediated through language. As a result, visual media require less effort on the part of the audience to decode.
Literature and music, on the other hand, have no direct correspondence to the real world. They must be mediated by language, requiring at least one extra layer of symbolic expression that must be decoded before an audience can apprehend the artistic content (much less comprehend it). Because a large measure of collective consensus exists regarding meaning within formal languages, literary works operate within a framework that is relatively well understood and largely codified. Even if some writers occasionally "play" with the value and meaning of words, English remains English , Spanish, Spanish, Urdu, Urdu, regardless of the work or author.
Music exists in a far less symbolically formalized environment. Ideally, each composer and artist develops his or her own unique musical language or dialect. As a result, music is the most purely structuralist form of expression; ideas are encoded within a unique language or idiom that can only be interpreted through the relationships between expressive structures internal to the work itself. In a sense, music is a language (really, languages) that consists entirely of neologisms. While this means music can "speak" with far more spiritual precision than other media, it also buries content within many layers of ambiguity, and thus requires maximum effort on the part of would-be interpreters.