What's the deal with thinking you have a penicillin allergy if you don't have one?

I thought I've had side effects to medications, but never an allergy. Is this from like just general people's anxiety to being sick? like taking a medicine while they are sick makes them think it's the medicine itself causing the problem and not the sickness? I don't know how a penicillin allergy would have presented itself in the first place.