baby fever?

Ok so I have my husband and I have 3 kids 4,3, and 18 months... over the past few months I have started having slight "baby fever" but after thinking about it always came back to I have a lot on my plate as it is with 3 young ones.... and I just stopped breast feeding my last about 4 months ago (she was nursing to sleep at night) 
I have started to realize I have a little more freedom to do things.... I even recently thought it might be fun in June with my hubby on our anniversary to go to a amusement park which we did on our honeymoon and ride roller coasters together again...
I'm ovulating this week and out of the blue yesterday my hubby said.... I wanna have another baby a boy (we have 3 girls) long story short I now could be pregnant....
I'm not sure how I feel? Is that awful of me? 
Yes I want more kids and I would love to have another baby (or two 😀) but I guess when I had baby fever and I convinced myself there was so much I could do if I wasn't pregnant knowing my hubby wasn't ready and he was! 🤔