Acids give off H+ (Hydrogen) ions in water; bases give off OH- (Hydroxide) ions in water. Acids generally taste sour due to the sour H+ ion; bases taste bitter due to the OH- ion; but they may have other tastes depending on the other part of the molecule.
