If you mean to return to your pre erectile disfunction size, then you would just have to overcome ED. It does not have to do with PE. I believe it is about your psychology and if it is the case your reproductive system, either way you would have to go to professionals such as doctors and psychologists.
Now, if you are talking about PE, that you did certain routines and exercises, thus leading to gains and asking that if you stop, you’ll lose gains, yes you will lose gains, and depending on the routine you’ve made on your PE journey you’ll lose it all. Routines focused on tunica such as jelqs, pumping, extender, A or V stretches will lose it all or at least half of the gains made on the interval time of “gaining routine”. Ligaments focused routines such as hanging and stretches will also lose its gains, however not all. That is why in all of the exercises mentioned above it is with great importance that after doing the “gaining routine” you do a “cementing routine”, consisted on the same exercises, but fewer minutes or less intensity or less frequency, those are the variables you should manage to form your “cementing routine”.