I think it's because they are like magic for grown-ups. In children's books (and increasingly, grown-ups' books, but that's another story) magic is a way of easily resolving the problems of the physical universe – restoring the world to the way it was when you were little, and could get what you wanted without effort or deferment. In grown-up fantasies about the city, hermetically sealed or free-standing urban entities deal with the real-world problems of cities by pretending they don't exist, or by proposing technological solutions to them that work best in fantasy.
In the real world, the inhabitants of cities need food to be brought in every day, and it needs to be moved to where it's needed despite all those pesky people getting in the way. Sure, there have been times when some of this food has been produced within the confines of the city itself, and it's hard not to get dewy-eyed about dig-for-victory gardens, backyard chickens, and the urban pigs that graced London and New York into the nineteenth century. Others are turned on by urban agriculture as practised in Cuba, or by dreams of giant vertical farms within cities. But agricultural surplus in the surrounding countryside, and consequent surplus of population, is what has historically made cities possible.
Cities also create huge amounts of waste, which includes but is not limited to the organic waste products of their human and animal inhabitants. Disposing of this stuff is, and always has been, one of the biggest problems with which all urban settlements struggle. Again, the surrounding countryside has historically been the solution. Relatively straightforward arrangements have ensured that the shit was taken to where it was useful, with the result that the farms and market gardens in the vicinity of cities had above-average productivity.
As transport became cheaper, the definition of 'surrounding' countryside became wider. At the heart of the Roman empire, the city of Rome imported its grain from North Africa and elsewhere. When the empire collapsed under the weight of its political, economic and energy-equation contradictions, there was a move towards re-localisation. Transport did not become so easy or so cheap again until the nineteenth century; until then cities were usually smaller and more closely linked to their immediate region.
But by the nineteenth century London, as the centre of a global empire, was obtaining its grain from one continent and its meat from another, and thereby supporting a population far greater than its region would have allowed. In terms of economics, the city only made sense as part of a global system; in terms of ecology, it didn't really make sense at all. Despite a flourishing network of market gardens on the fringes of the city, its sheer size and population meant that the night-soil economy was no longer sufficient to move all of the shit, human and animal, away from where it wasn't wanted. The immediate result was the 'great stink', and the longer-term consequence was the Bazalgette sewerage system, which collects human faeces from within our homes, transports them in a gush of potable water and then dumps them, more or less processed, somewhere downstream of us. Moving people round the city presented an analogous problem, and modern methods of transport were only possible because they relied upon energy imported from outside the city's boundaries – first fodder, then coal and coal-fired electricity for trams and metropolitan railways, then petroleum.
Cities are complex technical systems, embedded in complex social systems, with the latter both logically and historically prior. Cities can't exist outside of a web of economical and social relationships, not only within themselves but with the rest of the world. The twentieth century was rich in examples of cities that were politically separated from their surrounding regions – think Hong Kong, Singapore, West Berlin, West Jerusalem, or Nicosia. All of these have survived and sometimes thrived because of the way that they managed to plug themselves into a bigger support-system.
The idea of an arcology, or of a city that can pick itself up and move out of its supporting bio-region to find a better one, is a fantasy of denial – of pretending that with the right technology, it would be possible to do without all those messy social arrangements. Some of these fantasies are influenced by green thinking, and others by its opposite – lots of the literature on 'seasteading' seems to be driven by the attraction of leaving the rest of humanity to go to hell, and the involvement of Milton Friedman's grandson is probably not a coincidence. There are utopian and dystopian versions of this, and some of the dystopias seem to be intended as warnings rather than blueprints. As a thought experiment, and as an influence on some more serious proposals for making cities 'smarter', these visions might be a useful tool.
Ultimately, though, in so far as they are intended seriously, these proposals are a kind of survivalist response to real urban problems – I can't hole up in the woods with a gun because I'd miss my cappuccinos, but my entire city can cut itself off from the world. As in Bob Dylan's “Talking World War Three Blues”, in which everybody sees themselves walking around with no-one else; all the people can't be all right all of the time. Believing that you can make your city sustainable by preparing it to go for a stroll belongs with the fairies at the bottom of the garden.