Heat may cause permanent testicular damage
Most of what I have read about the effect of heat on the testes refers to damage to sperm. This, of course, is only temporary - the lifecycle of sperm is around 64 days, and new sperm are being created all the time.
However, out of an abundance of caution, I decided to do a bit more research and came across a review of studies regarding the effects of heat on the testes of mammals. I included a few excerpts below:
The effects of heat on the testes of mammals
Anim. Reprod., v.3, n.2, p.81-91, April/June. 2006
Although many authors believe that the effects of heat are fully reversible, there is now evidence of long-term effects, following either locally applied heat or temporary induced cryptorchidism. These long-term effects appear to be different from those seen following irradiation, and may indicate a failure of Sertoli cell function.
Most techniques have been shown to affect principally primary spermatocytes and early spermatids, but there is some evidence for effects on spermatogonia and Sertoli cells. The Leydig cells and androgen secretion do not appear to be directly affected.
However, other recent evidence has been obtained that there may be much more long-term effects of heat than previously thought. Furthermore, the effects of heat may not be confined to cell death in the testis and the consequent fall in sperm numbers in semen, but the sperm produced may be less capable of fertilization and the production of normal embryos.
In other words, locally applied heat may cause long-term damage to cells responsible for creating sperm, but it does not appear to to affect cells responsible for hormone production. Therefore, pointing a heat source directly at your balls may not be a good idea if you want to have kids in the future.
I have attached the fully study.