Too many rocks explode when heated. I doubt that concrete or bricks would, but I'm stuck on where I'd come up with them. (Frankly, rocks would be a problem, too.) And I'd still be concerned about concrete and bricks popping out of my pan.
I'd guess the other way. Bricks and concrete are more porous than most stone, so they would be more likely to absorb water and thus crack or explode. Just a guess.
Probably a much safer (and more effective) route is the old heated water bottle in your sleeping bag trick.
-john