I think zombies should be able to destroy any wall or obstacle, but only if it makes sense.
Like, they will path to a nearby door or window as it's recognisable as such, but otherwise will pick a random wall to start wailing on it, with a preference for walls already attacked by other zombies (ie, it will have damage and blood on it) or that appear weaker then others. A level 1 or level 2 wall has holes that might give it a line of sight/sound of a player, whereas an unplastered tier 3 wall allows more sound through then a plastered one. And an interior non-loadbearing wall will be way weaker then an loadbearing or exterior wall.
And damage will be tiered as well, so a level 3 will first have to be degraded to a level 2, then a level 1, then a destroyed wall (like you see in burned out houses) which allows zombies to climb through them. Actual destruction only happens if a wall is bracketed by destroyed walls. (Fire will work like that too then, degrading floors and walls, outright collapsing if too much neighbouring tiles are damaged as well.
Some walltypes have different rules. Concrete walls cannot be damaged, but they might be scaled by climbing zombies if enough zombies gather at one section. Chainlink fences will tend to lock zombies in place because they can see the player, but if too many gather at one point they might actually start to damage it by the mass of bodies pushing it, so you want to break line of sight or kill them. (Note that attacking through the fence has a chance of inflicting damage to it.)