That's a toughie.
I love the pairing but I'm a very moral person. I'd probably do the same but the only thing that really sours the whole deal is Ivy herself.
I mean the only thing she really wants is his lips. :)
Edit: (Not that I'd complain)

It would be a tough decision for me. Even though I think Poison Ivy is hot, well, read my motto. Once you read my motto, you'll see why it's tough for me to decide. But like bellboyjr said, It's not that I don't like Poison Ivy, I'm just kind of neutral in this situation.