I found this:

http://www.ehow.com/how_2156678_sharpen-knives-old-belt.html

So ideally the belt should have a smooth side (grain) and rough side and you would start with the rough and "polish" on the grain side.