Jace,
Yeah, La Jolla doesn't get too cold. But, you're from there, so you're not that acclimated to the cold. I grew up in the OC, so I remember that the 50-60 degree temps of winter are "cold." Throw in the fog that La Jolla gets, and you'll be pretty chilly. A hat and vest (or jacket) would be a good idea, as might a poncho, in case it drizzles (yeah, for the one day annually where SoCal gets rain.)