It was because of the doll with buttons. Wybie gives the doll to Coraline, and through it, the Other mother knows everything that happened to her. How did she knew that Coraline was there in the first place? We don´t really know, but maybe Wybie told his grandma about a blue haired girl wearing a yellow jumper on the roze Palace and the Other mother heard about it through another doll in Wybie´s house and made another doll that looked like Coraline. Nevertheless it´s just a theory
Any children who verplaats into the roze Palace are instantly found out about door Beldam. That's why Wybie isn't allowed inside. His Grandmother doesn't want to loose him too
