Well, actually that is what could make your penis shrink:
“……….The penis is a rather exceptional vascular bed, with its oxygenation dependent upon erectile activity. In normal men, discrete episodes of penile erection are present at least every 24 h, during eye rapid movement sleep. These episodes are three to five per night and last 20–40 min, allowing a sufficient amount of oxygen to be delivered to the CC cells. The pathological absence of such episodes determines a prolonged flaccidity and hypoxia. The present study may indicate that a decreased penile oxygenation for a prolonged time will in turn induce ET-1 expression and SMC proliferation, possibly perpetuating the vascular damage.”.
ET-1 have a negative effect on erectyle function, so has smooth muscle cell proliferation - at least some kinds of it.