The elastic potential energy gained by the rubber band is the work done by you in elongating it. Positive work is done on the rubber band, so it gains energy. While work is done by you (i.e. negative work is done on you), so you lose some energy. The total energy is conserved.
In simple words, the energy gained by the rubber band is the energy lost by you while elongating it.
