If your PE is actually damaging your penis because you have some sort of obsessive compulsion to PE beyond your body’s limits, you constantly talk about PE, you spend an unhealthy amount of time on the PE forums, etc., then your friend has a point. In this case you might want to stop PE, re-think your approach to PE, and talk to a doctor before resuming.
But — as I think is most likely — if you are doing PE properly (starting with the newbie routine for a few months, not having any negative physiologic indicators, etc.) then you don’t have some sort of weird, unhealthy obsession — you’re just trying to PE like the rest of us. In this case, your friend should mind his / her own business.
Someone on this forum has a signature that says something like “obsession is what lazy people call dedication.” Maybe that applies here.
edit: found the sig! it reads “Obsession is a word used by the lazy to describe the dedicated.” It’s in aom91’s signature.