Good sextants are as acurate as you state. The errors are introduced with the observation and time measurement. There are systematic errors like refraction and variation of the suns radius and random errors in the observation as you can´t hold the sextant really steady. You may not be able to adjust it with the precision of the calibration. Most literature states that you should expect a position error of about 5 nm even if you have a good sextant and apply all corrections.