The guilty look that dogs give is also called the "apology bow" and has been studied extensively in wolves by animal behaviorists including Marc Bekoff. While it may seem like a trivial action, it is actually involved in multiple fairly sophisticated social behaviors. As so often happens when studying behavior, or anything else in biology for that matter, a close look reveals far more complexity than initially expected.
In wolves, juveniles first begin to display the apology bow as they begin their social integration. Besides nursing, the first social interactions that wolves engage in is play, usually rough-and-tumble wrestling. You've probably seen dogs wrestle and they have inherited this play behavior from their wolf ancestors.