Hello everyone! To save the Mondoid being a million miles long, we thought we'd post the details here and link them on the blog.
So we've made some big changes to the rendering engine. While the effects of this change may be somewhat subtle in the short term, it will have dramatic effects to the future of the game, and opens up many many exciting doors that were previously closed to us.
Last Wednesday, we were concerned to discover the game’s memory usage had once again bloated to the point the game would no longer function on many lower end PCs. We had one more quite dramatic card up our sleeve. As is usually the case, necessity is the mother of invention, or rather necessity made us do something rather dramatic that we’ve wanted to do for a long long while, and while the timing of such upheaval couldn’t have been worse, now we’re at the back end of it and everything looks set to work smoothly, we’re immensely happy and excited to be able to share.
To explain, and since the Isometric blog post we wrote way way back in time (with now defunct missing images due to the various server moves over the past few years) we’re going to have a revisit to the subject. The TL;DR version is on the blog here.
THE GENESIS OF ZOMBOID
As many who have been around since the start may know, while Zomboid is a game we’d wanted to do for many years, as an actual project it was born out of myself and Binky realising we couldn’t afford to pay the rent on the flat, and panickedly having to get something out there with a rather pleading blog post and paypal button. This was in a time before Kickstarter was a ‘thing’ in terms of game development, and inspired by what Minecraft showed was possible, we figured it would be our best shot at keeping the landlord at bay.
The decision to make the game isometric came from a few angles. One, we realised that with a topdown 2D perspective would mean making zombies visible at windows would be really difficult. Secondly was a certain obsession with XCOM and the ambition to make something that had a similar look. Thirdly, we found this awesome program http://www.mapeditor.org/, Tiled, that would one day be taken and expanded in exciting ways by the awesome EasyPickins.
In its basic form, Tiled supported the isometric perspective. So we started making a few isometric tiles that we could use to construct maps in Tiled. This is what we ended up with:
As you can see, the tiles have been drawn directly in a faked iso perspective, so when they are assembled, they create the illusion of a 3D. This fit our needs perfectly, and meant we could get a working isometric engine up and running in a matter of days. Within a few more days, we had a basic prototype of character running around being hunted by zombies:
Tiled was key to getting stuff where it was so quickly. Writing an entire map editor and not getting the game up and running ASAP was not really an option.
Of course, alpha-funding being what it is, the momentum is always behind you to get results, and we had no idea where Zomboid would one day end up. Smooth character animation, real-time lighting of the world, and many thousands of tiles. Also in our naivety to Java, OpenGL and engine development in general, as well as the degradative effect of the pressure to make visible strides forward, or of course all the roadblocks that lay ahead of us, we didn’t then realise how much trouble this would cause us later down the road.
2D doesn’t always mean ‘fast’. Our assumption at the time, just like everyone who has ever complained of the FPS of Zomboid with comparisons to Crysis or Left4Dead, that drawing a few low resolution sprites on the screen for walls, floors and zombies, would never lead to issues with performance. However, we didn’t realise at the time quite what a nightmare the isometric perspective is when approached this way.
First of all, if you look at those walls again, you can see quite how wasteful they are in principle.
That’s one set of walls, and there is a lot of repetition there. Also, because of the iso perspective, it means there is a ton of wasted blank space around those walls. While when you’re dealing with one or two set of walls this doesn’t seem like an issue, once you’re dealing with 50 sets, or 100 sets, or 200 sets of walls, suddenly that begins to bloat. Same with the floors.
Second comes the problem with the isometric draw order. Imagine you have a deck of cards, and place them around on the surface of a table. Each card will end up obscuring cards under them, and this is the principle of drawing 2D scenes. You draw the stuff furthest away first, then progressively draw stuff closer and closer to the ‘camera’ until you’ve drawn everything. This will ensure that everything appears correctly, and you don’t see objects that should be obscured by walls.
Draw order in top down 2D games generally isn’t that big a deal. If you look at Dwarf Fortress, for example:
Nothing overlaps, so it makes no difference what order any of those tiles are drawn. The only time it matters is if one tile is on top of another, a character stood on top of an object for example.
For this reason, 2D games can usually sort their drawing into any order they like, and end up with the same scene.
Other perspectives, such as the orthographic viewpoint used in RPG Maker style games, have the requirement that stuff be drawn back to front, but again, like our game, they tend to have rather small requirements when it comes to rendering (assuming of course PC and not a mobile device with very limited rendering speed)
What we had is literally the worst of all worlds. Not only does stuff have to be drawn back to front, but also side to side. And then down to up. We might have a screen in 1920x1080 where the player can see thousands of tiles, and all of them, even the characters on the tiles, have to be drawn in a very strict order, otherwise it all falls apart.
This would be okay, if graphics cards didn’t suck so badly at doing this.
The key issue is that graphics cards deal best with doing ‘a bunch of similar stuff’. They can do ‘a bunch of similar stuff’ very very fast indeed, but as soon as you get into the realm of them doing ‘lots of individual different things’ they perform woefully bad, to the point that (setting aside java, a small development team and limited time) a 2D isometric game like Zomboid can have severe bottlenecks that a much grander and more impressive 3D game would not.
Imagine Far Cry 3, for example, in the midst of a jungle scene. It would be crazy to suggest for a second that Zomboid had more on its plate to deal with than Far Cry 3 graphically, so the assumption is that if Far Cry 3 runs well on your machine, then it’s shockingly poor of Zomboid to have even a slight hiccup.
However, 3D games have a few tricks up their sleeves which give them huge advantages.
So we have a jungle scene, and logic would suggest that, like I have described, you would need to draw everything from furthest away to closest, so that you don’t draw a tiny distant radio tower over a tree right next to the player.
In this case it would involve a very very intensive sort, taking everything in the world and ordering it by distance. On top of that, as I said, graphic cards deal very well with drawing ‘lots of the same kinda stuff’ and very badly at drawing ‘individual different stuff’. So drawing a radio tower, then a bit of hill, then a tree trunk, then a leaf, then a branch, then a fence, then a branch, then a leaf, then a tree trunk, would leave even the most powerful PCs in the world crawling at less than 1FPS.
However this is not necessary due to something called a ‘z-buffer’. A z-buffer, or depth buffer, is like an invisible canvas on which the graphic card automatically draws to as it draws anything in the scene. Imagine it is a greyscale black to white image. Every pixel it draws in the screen, it will also draw a value into the z-buffer where stuff far away is black, and stuff right next to the camera is white. Or vice versa, it’s not really a colour, this is just to ease visualisation.
Then when it draws another thing, it can test the value in the z buffer for each pixel it tries to draw, and if the value in the buffer shows what is there is already closer than the thing you’re drawing, then it simply ignores that pixel. As such, this allows you to draw the scene in pretty much any order you like, and it will magically construct itself in back to front. It’s frightfully clever.
As such, you can leverage the ‘same kinda stuff’ power of the video card, and perhaps draw every single leaf in the scene first, all using the leaf texture and perhaps the leaf 3D geometry. Then you could draw all the tree trunks, then all the branches, then all the ground, then all the radio towers, then all the characters, and then all the birds. As such you can basically send a massive list of things to the graphic card that it can burn through at a magically impressive pace, instead of sending individual things, one by one.. And so you get super fast drawing and much faster FPS.
So therein lay the problem for Zomboid. While technically it is perfectly possible to use z-buffers on 2D games, due to the nature of the engine, the tiles, and everything else this was never an option for us. As such drawing the world is ‘draw a wall, draw a fridge, draw a floor, draw a wall, draw a zombie, draw a wall. And this took a heavy toll on the best case speed we could ever strive for.
Add onto that the smooth lighting. I think everyone will agree the emersion and graphical improvement of the smooth lighting adds a lot to the game, but it massively exacerbated our problem. 3D games do lighting via tinting each vertex of a 3D model to different colours (actually this is not strictly true, lighting has moved on dramatically and is more likely to use shaders and normal maps and things these days, but this is certainly how all 3D games used to do it) however due to our isometric tiles being ‘faked’ by being drawn by the artist directly in the isometric perspective, we never have the corner vertices to do the lighting this way.
Instead, we had to draw a polygon over the top of the unlit tiles, using something called a blend mode (basically changing the way the texture is drawn to do funky effects not unlike photoshop blend modes for example). The state changes required to change blend modes, as well as the fact that these had to be drawn immediately after the wall / floor piece, and then the blend mode being switched BACK to draw the next tile, only served to make the problem worse.
And at the same time, the memory requirements for the graphic cards increased and increased, due to the increasing numbers of tiles, and the smooth character animations. The price of progress, huh?
Then came surprise number 2. It turned out many lower end graphic cards, particularly integrated cards such as our (now nemesis) Intel, either did not have any dedicated memory of their own, or worse did but still for some reason required that they mirrored the contents of that memory within the main memory of the PC. Worse still, that used memory would be within the same process as the game, and on 32bit systems particularly, there is a limit to how much memory each process can use before it bombs out with an out of memory error.
This struck a few months back and took us a while to track down, and was a rather worrying moment when we realised quite how much memory the characters and tiles were using, and how that memory had been masked by most graphic cards taking care of it in their own internal memory chips.
The most obvious answer to remedy this problem would seem to be ‘make them all jpegs’ or something, however the problem is that the actual file format is not at all relevant in this case, since whatever format the texture is stored as on a hard disc, once it gets into opengl memory it’s the very same 32bit RGBA texture. However, there is one compression format that pretty much any graphic card these days support. S3 Texture Compression (S3TC). It’s used very heavily in games, or at least was when we were making PS2/XBox games. DXT textures, while bigger on your hard drive than a PNG or JPEG, can be loaded into openGL in their compressed format, and even drawn directly in their compressed format, so take up much less space in memory (potentially a 16th). However, like MP3s or JPEGs, it is a ‘lossy’ format, which means the quality of the texture is irreparably reduced by the compression.
For something like a 1024x1024 dirt texture, this is annoying but more or less trivial, since the artifacts would barely be noticeable unless you had the textures side by side (though I’m sure most texture artists would strongly disagree). On a small sprite, this is another thing altogether, so we didn’t really think this was an option.
Even so, we attempted to compress all the character sprites with DXT5 and, somehow, remarkably, the results weren’t bad at all. There was a very noticeable drop in their quality, but we totally got away with it and to this day haven’t read one comment pointing this out. Furthermore we shrank the total size of the character textures to about a fifth. That was a good day.
However, due to the frequent complaints about the FPS of the game, and knowing the limitations imposed upon us we’ve described, we’d started to run out of ways to dramatically speed up the game, and with more features being added, more tiles, more things for the game to deal with, the problem was only going to get worse. Profilers (special programs that let you measure how much time is being spent in every part of the code) were yielding less and less ways to speed the game up, beyond caching things.
Caching in terms of optimization basically involves creating extra data in memory to save you repeatedly redoing the same CPU expensive things, and as you might guess, these kind of optimizations may speed the game up, but also take it’s toll on the memory. The two things are entwined, and once you’ve sped up all the inefficient code you can, its hard to make any savings without bloating the memory requirements. An eternal tug of war ensues.
In the meantime with West Point under construction, crawling zombies, and various other things then memory starts creeping back up. Of course it creeps up to pretty safe levels on most machines, but then in the frantic climb up Steam mountain it’s easy to miss that malicious demon sneaking back in. So we got to last Wednesday, when it became apparent that the test team members with problem GPUs were having problems with the game again.
So we whip out the S3TC again, this time our target is the tile sprites. We push them through, and have a look at the file size. Hurrah! We’ve done it again, DXT to the rescue. The Shadows are gone and will not return for another 1000 years.
Then we run the game, run around and wonder what all those blotches are around the world, and the colour drains from our faces. To make sure, we load the DDS files and look at them.
Oh no, what are those? They aren’t supposed to be there!
Looking through more and more files and it only gets worse, and we realise what a colossal fluke it was getting away with compressing the character sprites. Massive blocks of colour, horrible lighting artifacts. In a game where we had just reintroduced zoom back into the game, they were just awful to the point of being unusable.
So there we are, in a Newcastle centre coffee shop, at a mass mocha/latte quaffing session of the Indie Stone crisis management panel: Lemmy and Binky rolling off ideas on new ways to lower the memory usage of the game.
“We could always….?”
“You know what. Is this the time to do it?”
“My God, you can’t possibly mean… but we’re dialing up for Steam, now is not the time.”
“Now is exactly the time.”
And so we made some rather dramatic steps, or rather we decided to explore the possibility of doing something we’ve wanted to do for about a year and a half but were too scared to entertain the possibility.
Ladies and Gentlemen, say goodbye to the walls of yesteryear.
...and say hello to the walls of the future:
So we went back to our prospective PCs, and Binky cracked out C# and tackled the projection of the iso tiles into 3D (which frankly hurt my brain to even think about) as well as developing a tool that would automatically convert every floor and wall into flat textures. Meanwhile I set about adding support into the PZ engine to render 3D polygons in an iso perspective, and creating a system for ‘prefabs’ to allow us to make walls, windows and doors with any of those textures, and for those to be automatically created from the isometric tile data.
Yes, in a rather remarkably sudden move, Project Zomboid is now, TECHNICALLY, in 3D. Madness!
BUT WAIT, THIS DOESN’T MEAN WHAT YOU THINK IT DOES.
It’s worth putting that as a header by itself. You are NOT going to get a first person view, and you are NOT going to get a rotatable world. This can’t be expressed strongly enough.
We still have hundreds of fridges, sofas, cabinets, trees, characters all in bonafide 2D with fake isometric drawn directly into the tiles. This isn’t going to change (though who knows for the next game in the PZ engine, or perhaps post-multiplayer, years down the line, we’ll get some hair brained ambition to do it, so it’s always a tantalising possibility where prior was a literal impossibility)
However, in very real terms, the PZ you will see from the next version going forward, is doing all the same things a 3D game does. The only difference is it’ll have a cast-iron fixed isometric orthographic viewport of the 3D world, so all the furniture, trees, tall grass and characters, still in 2D, can be drawn into the world as pixel perfect camera facing billboards and match the perspective perfectly.
SO WHAT DOES THIS MEAN?
It means many things.
- Primarily, it means that if you compare the two different sets of walls above in terms of space, a single 1024x1024 texture will now be able to hold every single wall, floor, wall overlay and floor overlay in the entire game onto one single texture, with room to spare. For each single texture, we could perhaps get another 200 different wall designs, Instead of, perhaps 5-10. This is obviously huge in term of memory saving.
- Just as primarily, it also means that we can finally use a z-buffer in our game. We therefore are no longer confined to the isometric draw order, the single most wasteful and restrictive thing we’ve had to deal with since day 1. We could draw every wall visible on-screen, all at once. Then we could draw all the zombie hairs, then all the clothes, then all the furniture. In any order we like. The potential difference this will make to our FPS is, hopefully, the most significant optimization we could make to the game, and dwarfs everything we’ve been able to do before. We suddenly have every tool in the 3D toolbox available to all those games doing a lot more intensive stuff than we are. We’re free, liberated finally, and it feels gooooooood.
It should be noted however, that these FPS savings may not come straight away. It will be gradual. We still have a version to get out, and Steam to get ourselves on, and so our first implementation of this will unlikely get the full benefit, and will likely offset the FPS increase with sufficient slowdowns in the immature first version of the system. Don’t expect huge things straight away. However, now finally there is a whole pile of things we could do that could have a dramatic effect, instead of us having to claw back little bits here and there with one giant immovable bottleneck.
We would have done this sooner, but please understand that this was a bold move that we only dared tackle now of all times because we felt we had no alternative, and we could have gotten a few days into it and discovered another month of work ahead to get it finished. If that was the case we’d have had to find a plan B, and we’re immensely glad that won’t be necessary, since it’s difficult to imagine what that plan B could have possibly been. However, the last hurdle in the ‘is this even possible?’ category is behind us, and while it may delay the release a bit further (we’re talking days, or a week, or a couple if we’re dramatically unlucky, and not weeks or months)
So if memory savings, and FPS aren’t exciting to you, here is an insight into those ‘new doors opened’ beyond that. Things that were not possible but now are. Please note some, or all of these, may not appear for months or even years from now, and some may be trivial to add in the short term, and this is just an indication of what is now on the table.
- Cutaway walls. Yes. Sims style cutaway walls becomes trivial, and is a certain addition in the near future. As a bonus this means no more reliance on the stencil buffer, which means lower end users may be able to play the screen with smooth lighting, shaders, split screen and zoom, where they were once unable to.
- Terrain heightmap and freeform texturing. Imagine running through the wilderness over hills and into ditches, falling off cliffs, digging the ground out for construction, natural cave systems. We’re no longer shackled to the isometric flatness and 90 degree roads. The potential for making the wilderness an infinitely more exciting place to be is very exciting.
- Proper lighting - Imagine shining a flashlight around a room and seeing the light on the wall, the filament of the bulb, seeing the tall shadow of a zombies cast through a backlit doorway.
- Fog / Particles - The iso walls always had the potential to make particles clip through the walls in an ugly way without the zbuffer. Now we could use 3D fogging on the map, as well as look into more atmospheric particle effects, think of ground hugging fog drifting about the place.