Sweep the magnet across the metal wire in the SAME direction 10 or so times. Make a mental note of the direction you are sweeping the magnet. This is the end of the wire that will point NORTH.
I think whether that end points north depends on whether you are stroking it with the north end of the speaker magnet. Which is hard to know, and even if you do know it's hard to remember whether it should be the north end or the south end. In practice, compasses improvised like this need to be calibrated. (Which may be doable, since you only need to know north very approximately to tell which end of the wire is pointing that way; and worth doing, as the compass may be useful when your calibration reference stops being available.)
There's probably a bigger, stronger magnet in car speakers than in a phone.