When you inflate a balloon the first few breaths are really hard (your ears may even pop) but then it gets a lot easier easier. The thick rubber of the small balloon exerts much more force on its contents than the thin rubber of the large balloon. (A used balloon is easier to inflate than a new one.)

So when you equalise the two balloon system there is not enough pressure to keep the small un-stretched one inflated.