I get this question often when it comes time to schedule shoots.
The quick answer is late afternoon when the sun is setting, or very early in the morning, as the sun is coming up. I don't think I have ever had anyone opt for the latter...
Photography is all about light. The late afternoon sunlight creates long shadows from objects (such as trees and buildings) which is desirable when taking outdoor portraits. Another example of soft lighting would be an overcast day.
Below is a perfect example of a late afternoon portrait utilizing open shade. I placed the couple and their puppy in the shade which created nice soft light on their faces. I then added just a little bit of flash to add some pop.
Why can't we take our pics in full sun at noon you ask?
The sun is a very bright, hard light. It creates strong contrast. It can make us sweat and makes us squint. While modern day digital cameras have very good dynamic range, they still cannot capture detail in both the highlights and shadows that the sun produces at midday.
Working with softer light allows me to work with less artificial light (flashes) and it lets me use more creative apertures for portraiture.
If you have flexibility for your outdoor portraits, I highly recommend the latter half of the day. Your pictures are worth it!