I know for me that caffeine does pretty much nothing while if I have a LOT of sugar I can really tell. I think this is due to fast metabolism and processing the sugar rapidly also possibly why caffeine has no affect it gets processed away fast?
I know the caffeine thing runs in my family on my mothers side too.