This is tough. Being on the totally opposite end of the spectrum, if my SIL and Brother asked me to be in the delivery room, I would probably break down crying.
However, this is about you, not me. If you are truly not comfortable, I would just pull your brother aside and explain it. Tell him that you will be at the hospital, and that you can't wait to meet your new little niece or nephew. He should understand. I bet the nurses will be happy, too, to have the extra space.