Various thrift stores and local record stores tend to have a lot of used classical CDs for anywhere between $1 to $6 -- A great bargain, but it's so underpriced for what you're getting. Go rescue some Beethoven!
New classical CDs and records go for relatively low prices, usually well below $10.
It helps to do research on conductors too, there's so many different ones to choose from.