I usually carry a EMS fleece vest, but it doesn't pack small. It's Polartec 200, I believe. Or, I have an REI shirt... not sure the name of it. It's 60% poly 30% nylon, 10% lycra... it fits and looks almost like a wetsuit when on (it's pretty snug). I've found that over a wicking shirt, I can stay warm down into the 40s if actively moving. It looks like almost identical to this:
http://www.rei.com/product/747571