Female selection pressure isn’t the only factor though. The penis is an appendage, and in northern climates would suffer frostbite if too large during very cold winters before clothes and fire were discovered and widely and reliably used (I suspect this is why, when flaccid, smaller penises can almost completely retract inside the body). A large penis is also a liability in fighting, where it makes it easier to strike a glancing blow at the scrotum to which it is connected. In fights for females, hitting one’s opponent in the genitalia would not only have the effect of incapacitating them, but would also make them unable to perform sexually even if they won the fight — quite important for a tribe defending its women against attacks from another tribe, where rapes and/or opportunistic copulations by the victors followed close on the heels of winning the skirmish. Anything to make the target smaller is an evolutionary advantage.
Even so, the human penis has clearly evolved to be larger, probably because of sperm competition pressure where a man who brings a woman to an orgasm is more likely to impregnate her than one who does not. Biologists say that apes in which multiple males compete to impregnate the females (e.g. chimpanzees, bonobos) have much larger penises and testes than apes where one male monopolizes the females (e.g. gorillas). Humans appear to fall toward the multiple males competing end of the spectrum when their genitalia are measured against other apes in proportion to their physical size.