One possibillity if the small-plane pilot is commercially licensed, is to ask if he/she is "carded" by the OAS. That is the USDept of Interior/Dept of Agriculture "Office of Aircraft Services". The BLM and USFS require all of their employees to use only OAS-qualified/carded pilots and aircraft. Certified pilots and aircraft will have an OAS-issued "card" or certificate for themselves and their aircraft, with a check-off for each type of flying for which they are qualified.
These OAS-certificates are issued to pilots and aircraft who meet certain minimum qualifications for non-scheduled flights in the terrain and conditions that you are going to encounter: mountains, winter flights, IFR-qualified for night/inclement weather, etc. OAS has flight-qualified people who inspect the pilots' logs, airplane, equipment, etc. It is a fairly lenghty process, but it does insure that the pilots and aircraft are qualified for the conditions that are anticipated in a particular environment, especially for low-level flights.
I suspect that most of the problems we encounter are the ones you describe, where a friend or acquaintence invites us up for a "short, fun flight". DO NOT fly with anyone who has "just gotten their license". DO NOT fly with anyone who is not instrument flight rated if there is ANY possibility of a storm or nightfall during your flight. That's waht killed JFK Jr and his passengers - he tried to fly at night when he was not qualified. I would also suspect that it is not a good idea to fly with politicians or movie stars or anyone who is personally arrogant, since they will not admit that they are getting into something they cannot handle.
If you go up with someone who makes you at all uneasy with their attitude, require them to land immediately and let you out at the nearest airport. If they refuse, threaten to barf on them - always worked for me. At the least, you can ask the pilot how many hours he/she has in mountain/desert/whatever flying (1000 hours would be a good start), are they instrument qualified, what kind of survival equipment do they carry in the plane, etc. You could also ask to go with them when they file the flight plan, and ask them to request "flight following" from the airport. This is a service where they check in with the airport/control every 15 minutes, giving their heading and location. It is a lot easier to search for a plane with a 15-minute radius of search than a 6-hour radius.
Bottom line for me, after flying as a passenger for 2 years in an Army helicopter unit, and flying as a passenger in small planes and choppers for 28 years with the govt, is that I figure I have used up my allotment of flying fortune. I do not take flights in anything but commercial, scheduled aircraft, and I do not particularly enjoy doing that. When I retire and have enough time to drive, that or trains will be my preferred method of travel. I realize that from a statistical standpoint this is an irrational attitude, and that I am likely to offend every general-aviation pilot on this web site, but hey, it's my life and my decision.
Actually, that is my basis for not particularly liking to fly - the lack of control. At least if I go off a cliff in my own truck, I can blame myself on the way down. Don't know if this will help, but it might provoke a few more replies.