I always prime, but the better painters around here will do specific colours depending on the miniatures, as one of its jobs is to make the paint you apply after suit the model better, e.g., some people may prime Orcs brown, Gandalf and Saruman white, etc...
I personally just use Army Painter black spray paint, sometimes takes a couple of go's to get an even coverage, and its slightly lighter than Chaos Black, but it does me fine, I really don't have time to had prime stuff anyways.
I think one of the main benfits is supposedly that it helps the paint to stick to the model better, which is also why your suppose to bathe plastics in soapy water before painting, especially onto the bare grey, as some liquid left over form the moulding process makes it hard for the paint to stick.
But if you'er fine without primer then go ahead, you'll certainly save yourself some money in the long run, but I'm sure other people will reccomend it for better results.