Does bonus doubler increase stats by the buff amount, or by the buff minus debuffs?
I believe this. Since buffs and debuffs work by simple addition and subtraction. So if you have +5 attack buff and -4 attack debuff, you will have a total of +1 attack. But you will still have a buff of +5 upon you. Panic on the other hand turns buffs into debuffs, so that works differently.
It increases stats by the buff amount. Debuffs do not affect it at all. Panic on the other hand completely cancels Bonus Doubler.