I agree with this statement for experienced drivers, but for situations like the OP, who just got their learner's permit, I think it's appropriate to keep the speeds down, even in spite of other factors.

New drivers will always be confronting new situations and they often hesitate in deciding how to react. If traffic is flying along at 50mph in a 35mph zone, the new driver may be comfortable going with the flow of traffic and could do it all day. Some may argue that it's safer for this driver to be going 50 than 35. However, if something novel suddenly develops, the new driver may need every millisecond to either make up their mind on how to react, or else to ask the person they are driving with (the OP only has a learner's permit, after all) what to do.

After we've been driving for a while, we almost instinctually know what to expect in various situations and adjust our driving accordingly. But a new driver doesn't have the advantage of experience and may be paralyzed trying to guess what might happen or how to react properly, in addition to the sensory overload that afflicts new drivers at first.

When the new driver takes their driving test, speed really does matter, too. Go over the limit and we may flunk, regardless of whether it's safe to drive that fast. We all adjust to whatever speed we're accustomed to driving. If we have a habit of driving faster than the posted limit, then the driver has one more thing to consciously worry about on the driving test. I moved from a city with relatively fast top speed limits to one with limits much slower. I had a hard time keeping my speed down at first and had to constantly check my speed, but now, I drive without even looking at my speedo and I'm naturally driving at the slower speeds expected in my new city.

New drivers also tend to get into more car accidents. Regardless of why it happens, since the energy of a crash goes up by the square of the speed, any extra speed is magnified in impact energies. Keeping your speed down increases the survivability of any accident, which would be prudent for those at higher risk of getting into an accident.