There is no absolute limit. When the sun is near the horizon it is far less intense, this makes starting a fire with a given size lens or mirror more difficult.
By concentrating solar energy from a larger area, by means of a larger lens or mirror, then a fire may be lit even in very unfavourable conditions such as with the sun near the horizon or partially obstructed by light haze.

By use of an A4 sized Fresnel lens I have easily started a fire with the sun close to the horizon.

However as others post, a fire is most often wanted under conditions quite unsuited for solar fire lighting. IMHO a large concave mirror or a large Fresnel lens are well keeping for use in any TEOTWAWKI situation, but are less useful for a short term emergency.
Matches are soon expended, though carrying enough for a month or two is easy, a ferro rod and striker will light hundreds, maybe thousands of fires, but is eventually used up.
A suitable lens or mirror will start fires INDEFINATLY with nothing whatsoever being consumed or expended, that could be a lifesaver in any true long term emergency, but is less applicable to backpacking etc.