My three month old hasn't wanted his bottle all day?

Normally he has an 8 oz. bottle in the morning (he sleeps all night) but this morning he just swatted at it and when I finally put the nipple in his mouth he got fussy. I don't understand what's going on! He's completely content when I sit him in his bouncer but when I try to offer him the bottle (I have been in 15 min. intervals) he gets pissy. Should I be worried if he's not eating as much as he should?