You would have to have a large enough balloon.
I'll try and just give the very basic physics involved:
So, first things first, you need to get past the force of gravity on the city. Basic F=ma kinda stuff.
So you need to provide a force that is large enough to equal that due to gravity. You suggest balloons. Ok, so basically:
F(balloon)=-ma , or opposite gravity.
So we're done right? Not quite. There's one last thing that comes into play with this simple model: tension. The balloons need to be tied down with something, so that something will be doing all the heavy lifting basically.
Basically same F=ma stuff.
So to sum up, a floating city isn't very viable, as the equipment used to support it would be able to handle the mass of the city without using impossibly strong "somethings" and balloons.
You could also use this same situation to explain why tying a bunch of balloons to your baby brother isn't going to make them fly away anytime soon. He, like the city, weights too much for the strings on the balloons to handle it.