In a 500 point game you would easily be able to match a Haradrim force in numbers (48 warriors for about 360 pts). 1 point difference doesn't turn into 2:1 fights unless your army is built/played poorly.
The one point difference is because Rohirrim have shields. They also have the advantage of throwing weapons allowing you to move into range - kill a bunch of them, and then move out of charging range. Or, you can just sit there shielding (and throwing spears) while your cavalry mops the floor with them...
You might want to check the tactics forum for "Rohan" threads
.