A good nights sleep is worth a few grams. If a dedicated pillow is what it takes to sleep well then it is worth carrying.
I typically stuff some spare clothing in the stuff sack that holds my sleeping bag. Works great.
So do I.
Or even better: I select one item of clothing as my pillow for tonight and then stuff more stuff spare clothing into that.