A kayaker was taken to the hospital after being rescued from frigid waters in the Long Island Sound off the coast of Milford on Monday.