Stuttering Whilst Driving - Cause

SilentLight · April 3, 2024

Whilst I know the indiestone devs are already aware that there is FPS stuttering whilst driving in Project Zomboid, commentary suggests no-one is entirely sure of the cause, and based on my previous experience in programming and observations during sandbox mode driving, I think I've got a good idea what the cause is, based on when it happens and who it happens to. I will explain my reasoning - and solution - below, but the summary is: the stuttering is caused by the game attempting to load too many objects all at once in every 'frame update' whilst driving.

What It Can't Be

We can rule out insufficient computer resources, because even players with powerful gaming computers report this phenomena. The game generally has a low tris count (a metric used to determine graphics demand and performance), and the CPU does not sound like it is 'firing up' or 'melting down' during the stuttering. Driving does not stutter even for low-end computer during certain situations (I.E. mostly empty main roads). This points us towards a suboptimal algorithm or piece of code that takes the game a long time to complete (so it isn't CPU intensive, but time intensive).

What It Most Likely Is (Based On Tests)

With sandbox mode and all zombies disabled, driving generally does not stutter until you hit Louisville. The fact stuttering is (somewhat) reduced based on the absence of the zombies gives us an important clue; it suggests when the game is trying to update or 'load in' zombies - even ones visually the player cannot see - there is a drop in FPS thoroughput performance. This tells us, as the driver is rapidly traversing through areas, the game is attempting to rapidly spawn, then despawn, large quantities of zombies, which means the thoroughput time performance is mostly spending on loading (I.E. it is a roundtrip memory performance or similar issue). This explains the lack of overall CPU demand.

The fact FPS massively drops when one first enters Louisville whilst driving (and is generally poor even after first entering), causing stuttering even without zombies, gives the first major breakthrough on the underlying cause. One thing Louisville is chock full of is buildings, and whilst one could suspect maybe the game is attempting to spawn zombies, I don't think that is where the major loading lag comes into play.

My suspicion is as the game loads in the buildings, it also attempts to load in every single object and item inside the building. In smaller buildings like huts and two storey houses, this brute force approach isn't a problem, but in Louisville where you have multistorey apartment blocks and large industrial complexes, the impact is a force multiplier (so a 10 floor building has ten times the loading time of a single floor building). My additional suspicion is, on top of loading the physical objects, the game also attempts to set things like the interior items loot tables and other things the player would not be able to access whilst driving.

We can reinforce this by the fact walking does not produce the stuttering in the same area; if it was graphical, walking would produce the same stuttering effects. If it is loading, however, walking significantly slows down the rate of which objects become visible, giving the system time to 'catch up' or even 'keep pace'. I bet if you turned a walker into a runner who moves at the same speed of a car, you would likely induce the same stuttering effect (it is not related to driving).

The Solution (For Objects)

This points us to an elegant solution. The game should not attempt to load indoor items (ever!) whilst the player is driving, and should keep the interiors 'blacked out' (for aesthetic purposes). The game should only attempt to load the interiors as the player exits the vehicle. In multiplayer, the loading should be client-centric (so the server can see all objects, but the clients only load interior objects when they leave a vehicle).

Now, this might work for say, a mostly hidden building with walls, but what is the solution for say, a grocery store with giant windows, or a recently burned out interior that changes every run? Or a custom player base? The game needs only spoof the interior. It can pre-render a single image "screenshot" of the interior, which it only uses when the player is driving (it does not load in any objects besides exterior, it just visually spoofs the entire interior as a screenshot layered behind the front-Z exterior). The moment the player exits the vehicle, it will update the screenshot, storing it back to memory and render the interior properly with all objects.

The Solution (For Zombies)

As for zombies, the game should not be allowed to even attempt to spawn zombies in non-visible areas (E.G. trees, buildings) whilst the player is driving, especially at speed. Zombies on upper floors of buildings beyond the first, also should not spawn whilst driving (as the player cannot reasonably get there whilst driving). It can have 'placeholder zombies' (which are just coordinate points rather than a full model load), which spawn full object zombies + pathing algorithm the moment the player's vehicle either comes to a halt or drops to a particular speed (E.G. below 15 MPH). Note, only non visible zombies are excluded here, so the zombies that still spawn to block roads etc are fair game. They may have to be reduced to stop stuttering but that number would need to be based on unit testing (which I don't have access to).

Alternatively, as a kludge, you could create a spawn probability algorithm, where the faster the vehicle is travelling, the lower the overall probability of spawning a zombie is in that given frame (and thus, the associated object load with it). This would likely create the effect most players fear and desire during driving - that zombies will swarm their vehicle the moment it comes to a sudden crashing halt. It would also reduce spawn rates and thus reduce stuttering whilst driving.

A patch could be retroactively implemented by modders for the current build by making any sort of 'household object' (tables, chairs, sinks, cupboards, beds, etc) simply not load for clients whilst driving. It might make interiors look "janky" visually, but it would likely result in a much smoother driving experience if implemented correctly.

I am not familiar with the game's code, for the record, this is just based on my observations during testing.

Modin · April 3, 2024

Vehicles can go in interiors (through double doors), and some interiors simply do not have walls/doors and/or are _meant_ to have vehicles go there (e.g garages, or parkings from modded maps).

As for the proposed zombie solution, not spawning zombies above level 0 is a bad idea if it's going to prevent them from reacting to noise (especially the car's noise).

Another thought: if 500+ (yes some interiors have that many zombies above level 0) zombies ALL display at once upon exiting a vehicle, or going below 15mph, aren't you just moving the problem somewhere else? The game will stutter every time the player exits a vehicle in a busy area, and hovering around 15mph would create catastrophic stuttering.

Edited April 3, 2024 by Modin

krazmuze · April 3, 2024

I think your analysis is off. The 2.5d depth buffering is for drastically improving the performance of what gets drawn on top of other things so it can be handled by the GPU, since it is currently a 2d game that means it is doing that layering now using the CPU. So when you are driving where there is lots of building there is lots of layering going on - because it is an open world game where in/out zeds can hear/see in/out so that hordes have the potential to form based on how loud your driving is.

So you want to stop the possibility of doing drivebuy looking thru the windows to see what loot/zed might be in there and using your vehicle noise to draw the zeds out? You are basically changing it from an open world game to an instance building game where the outside and inside are different worlds - but the existance of the sledgehammer and burnt walls shows even that in/out distinction does not exist. The loot is already abstracted into how full or empty the shelves using clutter levels dependent on the room it is in, but you do not actually know what is in shelves until you open them but very useful for peeking into stores to see if looted by zeds, pcs or upcoming npcs. So already your premise that buildings are containers of visible loot hitting performance is wrong, because shelves are already instanced containers with visual abstractions it only needs to load how full it is and not what is in the shelf. Reallly annoying when you store dry kitchen goods in the living room, and it shows up as books though - but it serves purpose of marking shelves as already looted.

The instanced building approach is not new, Bethesda has used it for decades because they love to have building with lots of loot both useable and collectable with much of it visible on the shelves without abstraction. Starfield was widely panned because every door is a loading screen, I guess people forgot over the past decade that Fallout and Skyrim was the same way. Otherwise if it was truly open world they would have to cut down on the loot to save fps, and it is no longer a Bethesda game if they did that!

The problem with your idea for PZ is that zeds are attracted to the sounds of your car - so if they are in the building they should all get out and horde up to follow you down the highway - depending on how loud your driving skills and car is. Line of site is only used to compute a path but before that a general direction to head is based on sound. Your car even has a horn to force them to mob up as a distracting strategy to pull all the zeds to another area. If you remove that then you might as well replace the vehicles with a fast travel system, as the result would be the same - safe roads are skippable boring roads.

Keep in mind that walls are not just used for buildings, tall walls are also used for security fences. So if walls closed off inside from outside algorithm is done I could wall a city block near the highway and completely stop the horde from getting on the roads - which means the horde can now push down tall walls is a vestigal new feature because they will not aggro on your noisy car in the first place?

An idea would be to change the audio alerted zombie pathing when driving to use a single vector for the group (flocking) rather than individual vectors which you would still use for computing a line of sight path . This should work since zeds cluster into small crews when idle. That would not be useable on foot as pulling one zed from a group is an important strategy, but maybe for cars would work. But that is only worth looking at after the 2.5D GPU optimization for layering is in, as it is already known that will solve much of the CPU layering performance issue.

Edited April 3, 2024 by krazmuze

SilentLight · April 3, 2024

You appear to be confusing a gameplay mechanic fix, required to prevent stuttering whilst driving, with a gameplay change.

The suggestion isn't to do with gameplay mechanics, and to do with performance. It does not relate to 'CPU layering' or 'GPU layering', and if I might politely interject, I believe you thoroughly misunderstand my analysis. However, given a misunderstanding is possible, I will clarify.

Firstly, memory loading issues have little to nothing to do with GPU performance or even CPU performance. The Z-layers (visuals) don't matter as much as the loading of the objects into memory. In order for a program to load data into memory, it must either A) pull that data in from the harddrive and store it in the program's allocated memory cache, or B) generate new and/or unique information to store in the cache.

So, let us take the object cupboard, and this is just my example for memory loading. The CPU goes off into the game files, pulls up the cupboard coordinates, the cupboard sprite, it then pulls up a data table lookup for the (randomised) loot in the cupboard, it then generates a list of the loot, adds it to the cupboard, then loads the sprite into GPU memory, renders it, and then adds it to the buffered queue to update the position of the cupboard sprite every frame. Whilst this single, individual task, isn't taxing for a CPU, what you have is a very long MEMORY roundtrip - it is loading all of the data relating to the cupboard.

Now each building's floor has say, 4 cupboards (we're excluding the other types for now). Let us say the computer does the operation 4 times. Again, seems small, 4 cupboards is no biggie for the CPU but would take time to load in memory. But what happens if you have an apartment block with 10 floors load into memory? Well now you have 10 times 4 cupboards running the operation each time plus loot tables. Say there's 10 items of loot per cupboard. There's 4 times 10 times 10 operations occurring (items are objects too, and they need loading as well!).

A place like Louisville has hundreds of buildings. Let us round down and say there's 100 buildings you drive past at high speed. That's 100 (buildings) times 4 (cupboards) times 10 (floors) times 10 (items). That's 40,000 objects loaded in. Generating and loading this data takes time, by which time you are loading even more buildings, which doesn't give the earlier loaded data a chance to 'catch up'. Whilst doing this, I presume objects are also being unloaded, so lets say, 40,000 loaded in, and 40,000 being unloaded.

So the solution is to stop loading objects whilst the vehicle is in motion, however, as you point out, players want to "look inside" buildings, hence my suggestion for a snapshot/screenshot of the interior.

My remark about the Z-layer interior as a suggestion is what you would call a rendering trick. A placeholder, if you will. The game does not have to load 4 cupboards, a sink, oven etc as objects to give you the impression the interior has those features. A spoof snapshot image that shows those things as a single image will suffice. The player only cares about physical interaction with the objects once they actually leave the vehicle. Looking != interaction.

My remarks about zombie behaviours is an attempt to reduce the object loading burden of the zombies as well (cupboards are just one set of objects; obviously, you can count many more, such as chairs, fridges, sinks, baths, toilets, mirrors, boxes, crates, etc).

Hopefully this explains my rationale. Stuttering gameplay is not normal, and many game engines practice occlusion, both for graphics, and in this case, physics (I.E. they do not render objects that are 'out of sight' or 'inaccessible').

SilentLight · April 3, 2024

Retroactively, I realise there's an even more basic analogy.

Imagine you have an SQLite database that takes a long time to complete an operation because it stores 1GB of data and the query has to complete a look-up operation searching end-to-end, and say it takes an extreme 30 seconds for a query to completely search the database.

You would say it is slow, but it is neither a CPU nor a GPU problem. It is a memory problem. You would want to run that query the least amount of times possible.

krazmuze · April 3, 2024

It only needs to load the contents list of shelves and cupboards when you actually open the container, it certainly would be suboptimal if they had a master list of every item in the game saying what container it goes in. Much better to have a database hierarchy with a list of containers each pointing to a a list of their items in that container rather than searching a master list for items referencing that they are in this container.

The only thing it needs to load dependent on contents is open shelf sprites as they have capacity used abstractions. They do not actually show you what is on the shelf though, while cupboards have no capacity variants. That capacity number can be stored in the top level database and then used to select the capacity sprite, but better to just store which sprite variant is being use since it needs to know what sprites to load anyways. There is no difference between a container sprite and a wall sprite - just need depth info to draw them in order - a container sprite is not more of a load than a wall sprite. When you click on them only then does it use the type of sprite to either load the container list to open the container, or give the bash wall menu option if you have the sledgehammer equipped.

It is only those things in the floor "container" that need to get displayed as part of the open world, but you should expect bad performance if you are going to make a floor into ripped sheets storage and not use containers for things, as world design rarely has much loot left on the floor outside of special story scenes for performance reasons. Also manual decoration with placed objects show up in the floor "container", but that is also used sparingly outside of the player doing this in their base. (Use inventory to loot the floor container for faster looting as it skips the grab animation) Bethesda makes houses instanced containers because they know that players love dropping loot all over the floor in every house and decorate every place they visit - the upside is players like that in RPGs, the downside is no peeking in the windows.

The devs have already said the reason for bad performance in the cities is the 2d CPU depth layering which is why they are changing to 2.5d GPU processing. They are going to need to do that for the big city skyscrapers they want to do. Obviously the faster you drive the more sprite layers you have to load/display per second with more object sprite layers in city as well as likely using the widest zoom which is again more sprite. (it used to be zoom varied with screen rez making 4k worse - but that is fixed).

Instanced containers for houses is most certainly is a gameplay mechanic change because instancing houses breaks open world zed sight/sound pathing - so faking a window image does not help that, and fake window view also means that you cannot see how many zed are shambling inside the store using line of sight nor can they see you. Your fake window solution also ignores the sound vectoring which is a huge part of the game. Zombies/players peeking/hearing in/out windows is a huge part of a zombie game so you would not want instanced housing.

Edited April 3, 2024 by krazmuze

SilentLight · April 4, 2024

15 hours ago, krazmuze said:

it certainly would be suboptimal if they had a master list of every item in the game saying what container it goes in.

I can't comment how their load performance works in terms of memory because I haven't seen the code directly, however I am going by the combination of my years of experience as a programmer and years of experience in playing video games (I've previously built systems that process something like millions of records per hour). Even if items are not loaded in (a x10 reduction), you're still looking at 4,000 load and 4,000 unload events. It would sink the thoroughput of any system.

Per the first post, we can narrow it down to a loading event because the stuttering does not occur on the open, empty highways in a zombieless sandbox game. Consider for a moment there are tons of trees on these highways, and they do not appear to cause the stuttering, despite being a type of object. I strongly suspect, however, the trees make full or partial use of occlusion (occlusion isn't just purely a graphical non-rendering trick: it can also tell the game what not to load into memory).

For some reason, household objects, buildings, and even I suspect zombies, do not appear to make full use of occlusion (in the game options under display, there is an option, enabled by default, that it will update zombies out of view 'less often'; however this implies their full object is still fully loaded), which means, even if you cannot see into the other floors of an apartment building, all the objects inside that building are still loaded anyway. My suspicion it is internal building objects being loaded and unloaded as you rapidly drive through high-density areas that causes the stuttering.

It is less about database management (my SQLite analogy was purely an example to show you how performance can be slow even if a GPU isn't involved), and more about object loading management. In a video game like Jedi Knight 3, the GTK editor would have something like 'octotrees', which gave the rendering engine hints about what not to load into memory.

A loaded object isn't purely just about visuals (even many basic intel CPUs can handle many thousands of tris), but also about physics calculations, and roundtrip in memory. If we assume 4,000 loaded objects are all performing collision checks for the player's character, that's 4,000 collision checks every frame (there's also the User Interface collision checks, as every object is interactable).

If we were to overhaul how Project Zomboid works - I presume, in my probably wrong opinion - on the underneath, then my fully fledged suggestion (even though devs have made clear they're not keen on ideas that require major overhauls of how coding fundamentally works, hence my lame 'snapshot' suggestion, a kludge):

Introduce a full occlusion engine, and:
Register objects to tiles (with tiles acting as parents). We would count every isometric grid tile as a tile
Register tiles to floors (an area defined by the wall bounding box, which must exist on all four sides with no gaps)
Register floors to buildings (this might only apply to hardcoded, static buildings)
If a building is not visible, then we skip loading any of the contents
If a building is visible, we check to see if every floor within the building is visible from the side we are viewing it (so if it is barricaded, has curtains closed, has a metal sheet, etc and we can't see in, it is not visible). If the floor is not visible, we do not load any objects.
If a floor is visible, we split the floor into four sections (simple squares; we won't do anything more advanced). If any part of that tile section is visible, load all objects on it, otherwise, do not load the objects.
Introduce a full physics occlusion engine:
If the player is in a vehicle, then the player does not perform a character based collision check with any objects (vehicular, we would still check, but there's a neat fix)
If we are checking for vehicular collisions, then the vehicle's floor number (say us say ground zero is '0') is checked against the floor number of a building. If the floor number is not the same (I.E. the vehicle is not on the same floor as the object), then we do not perform a physics collision check for vehicles, either.
If we are on the same floor, and we're checking for collision, then we first perform a Fast Squareroot operation (using a radial, or to translate: circles) to see if the points are even close enough to each other to bother performing an isometric collision check. (This might seem inefficient, but 40 objects doing a fast squareroot using radials is faster than doing rectangle versus rectangle collision checks).
If they are close enough, then we perform an isometric collision check.

There may be some scenario exceptions to the above occlusion rules, but it is easy enough to make an exception. For example, if a player is trying to interact with zombies outside of the vehicle, whilst still inside the vehicle, then you would perform collision checks when they execute the action (which makes it a one-time cost per interaction, and not something checked on every frame).

Obviously, this entire system might sound 'simple enough', but it might require a radical re-write of the underlying code, and trying to integrate new approaches to pre-existing code can be hell. Hopefully this gives some idea. My main goal was to nudge the developers towards the problem, so they could design their own solution (mine was merely to provoke some ideas).

krazmuze · April 4, 2024

While there are lots of trees, there are lots of the same trees so there is a magnitude less sprites to load than there is in the city where there are lots of many different things wherever you look.. So again this is not proof of your assertion that they are loading items in containers before opening containers. Besides trees themselves are containers that when "opened" yield variable logs, twigs, and branches so even if your assertion that all items are loaded is not mistaken, it would also apply to forests having to load a ton of log lists. Since that does not have bad performance that pretty much disproves your assertion. Instead it indicates that container item lists are procedurally generated on the fly themed to the container type and location and loot settings, the container is empty until you actually open it. It is the entire point of the (un)lucky trait to bias these item lists - so obviously they are dynamically generated and not stored on your HD. It only stores container lists once you have opened them, but if you have loot respawn on even those get time deleted. Floor container population is managed by deleting drop items on timer being the default, prevents multiplayer griefing by mass dropping items exploiting that they are visible in the open world.

Default settings also cluster zeds into urban settings not forests so there are fewer zeds, so far less pathing to compute, combined with the sound vector indicating there is likely no line of sight because of all the trees, so it does not go beyond the sound vector into computing a visual pathing grid. Both cities and trees should implement sound diffusion, IRL it is actually harder to track sounds than in open plains - but that is similar RTX GPU load as light path tracing so obvious why a decade old game does not do it. They plan a forest map once they get the new blacksmithing so you can truly live off the grid - that will mean zeds everywhere - throw in some critters to track and hunt rather than just trap you will never know what that rustle over there is.

Again you are trying to optimize it when if you go read the updates regarding the next major patch, they have already acknowledged the performance issue is due to CPU driven depth layering, and the solution is depth mappng baked in so GPU depth layering can be done. Then occlusion culling and like instance object replication basically becomes easily managed as well if they need to optimize it further, but that is likely more needed with 3D objects that have 3D geometry with lots of poly, edges and vertices to deal with combined with complex texture atlas wrapping. 2D sprites are just a single texture and no geometry. This is what will enable them to do skyscrapers with magnitudes more rooms and floors, they already tried it and mentioned they had to turn down the hordes because they was bigger than highway hordes!

That is rule number one about coding, do not optimize things unless you actually know what the problem is.

Edited April 4, 2024 by krazmuze

April 4, 2024

It really is just a cpu issue, as Krazmuze states. More individual items, more sprites to sort, longer it takes to process the list. Single-core performance of the CPU determines how fast that happens. There's not much else to it and we'll see what 42 brings to the table regarding this, as the point is to cache 8x8 groups instead of single tiles, as I understand it.

You really shouldn't be seeing any issues with streaming data from your hdd. Any spinning rust after 2015 should be "ok;" any SSD shouldn't even flinch. But it depends just how much is going on in that chunk -- how big it is (individual items and carpentry can push em from 28 kb - 1 mb to 40 mb before the buffer gets overwhelmed, iirc) or what mods use LoadGridSquare.

SilentLight · April 4, 2024

2 hours ago, EnigmaGrey said:

It really is just a cpu issue, as Krazmuze states. More individual items, more sprites to sort, longer it takes to process the list. Single-core performance of the CPU determines how fast that happens.

This does not explain why high-end machines with 8-core CPUs and GPUs encounter stuttering whilst driving in Project Zomboid, or why stuttering is absent in forests on low end machines, given games with high definition, in-depth 3D rendering using far more advanced graphics like GTA 5 do not experience such stuttering effects as the low-res Project Zomboid does. That is because it isn't a graphical (or even a CPU) issue that Project Zomboid is experiencing.

It isn't solved by brute forcing with a more powerful CPU. I guarantee if you run unit tests the CPU won't even be hitting 100% (which is the hallmark of a CPU bottleneck). Computational time is linear, CPUs can only process tasks in order, which means code needs to optimise what gets processed. Why load an unviewable, non-interactable sink? Or cupboard? Why increase the roundtrip time?

Returning to my SQLite example, you would not use brute force checking every entry, like you're suggesting, using 'more CPU power', you would implement an array of optimisations, such as a sorted list with a binary tree search (you can find any item in a sorted list within 7 operations).

Addendum:

Quote

You really shouldn't be seeing any issues with streaming data from your hdd.

The HDD example is show how you can have roundtrip bottlenecks in memory that don't relate to how fast the CPU is. I suspect what you're imagining is something like 'no file would take 1 second to retrieve'. Try opening a 30GB binary file in, say, Notepad (or Mousepad in Linux), and watch the program struggle to load. It isn't due to a lack of CPU the file struggles to load. Think of the many, many, many caching operations that are occurring between disk, RAM and program (plus page cache and others). It's a type of memory delay, an extreme version, but one not related to the CPU.

Edited April 4, 2024 by SilentLight

SilentLight · April 4, 2024

2 hours ago, krazmuze said:

Besides trees themselves are containers that when "opened" yield variable logs, twigs, and branches

Trees aren't containers. That's a state-event change after they've been cut (removed).

Quote

Default settings also cluster zeds into urban settings not forests so there are fewer zeds,

In sandbox you can enable distribution to be even (with insane population) so the number of zombies can be equal, which is how I know you can cause stuttering on the normally stutter-free highways simply by having a large quantity of zombies. You can also emulate the same scenarios in sandbox described above if you think my observations are somehow incorrect.

Quote

there are lots of the same trees

Many of the trees have different sprites (pine, oak, birch, etc), and differing states of growth (scrub, young, fully grown). Trees also cause occlusion; therefore, trees can hide other trees - which as I said, I presume they're not being rendered. Waist height cupboard do not occlude other waist height cupboards.

Quote

performance issue is due to CPU driven depth layering

Why would it only occur in a city? Per the other remark...

Quote

as the point is to cache 8x8 groups instead of single tiles

That would be a type of (2D!) octotree (for the lack of a better term). Which is what I suggested earlier.

Quote

That is rule number one about coding, do not optimize things unless you actually know what the problem is.

Actually, the rule regarding optimisation is 'don't optimise prematurely'. If a game is however lagging or stuttering, that is the time to optimise.

I feel like this thread has been altered from 'here's my observational analysis on issues relating to too many objects being loaded in, using these real world tests', in which the high quantity of objects isn't being denied, and has morphed into a sort of 'lets forcefully argue it is definitely a CPU problem'. It feels like I'm being argued to with repetition and hairsplits but without explanation.

If the goal is the convince me it is a CPU related problem, then I'll flip the script so this can continue to be a constructive coding dialogue: what is the objective evidence it is solely a CPU issue, and how does it explain my noted test case observations? Specifically:

Why do both low and high end CPUs experience the same stuttering effects, if CPU makes a difference?
Why can a CPU manage to load without stutting whilst running on foot, but not driving?
If it's only CPU, why does it not stutter when driving on highways, but it does stutter in the presence of zombie hordes on said highways whilst driving? Even if one can run in the presence of said zombie hordes and generally not experience stuttering (until hitting prison-density insane levels)?
If it is based on the CPU struggling to render the total number of sprites on screen, why do cities lag, but forests don't?
If occlusion isn't a solution to performance, why is 'perform fewer updates' an option on non-visible zombies?
What CPU stress tests have been undertaken to demonstrate it is a CPU bottleneck? And what were the results of those tests? If the tests were not conducted, what is the basis for declaring the CPU is the bottleneck? (other than 'because someone says so')
If it's purely CPU, why can a low-end machine from 2010 run Quake 3 without stuttering, but a newer game with (no offence, purely observational) classic Sims-esque graphics running on a newer machine have stuttering? [Sidenote: Other games that have used Q3 engine include the Jedi Knight series, and the original Call of Duty 4]
If it's purely CPU/GPU, why does GTA 5 not have stuttering with far more detailed graphics (even when compared to the same machine architecture and stats), but Project Zomboid does?
What optimisations, if any, have already been adopted for object loading? What occlusion is presently in-use that minimises the overheads on computer resources?
What is the specified/average tris count during a drive in Louisville? And is this higher/lower than the tris count in, say, an empty highway, or Rosewood? [No zombies]
What is the average CPU utilisation re: Louisville/Empty Highway/Rosewood? [No Zombies]
What is the average threading/core usage in those areas?
If you don't have tris/CPU utilisation/core usage statistics, what metrics/benchmarking statistics are you using to determine CPU load?

And before you flip the metric/stats questions on me, remember, I'm not a developer, don't have access to the raw code, and thus lack the ability to set up any sort of meaningful, precision level unit tests (which is standard fare for most developers and pointless me trying to implement a redundant set of features the PZ devs likely already have). I'm asking in order to educate myself how you came to the conclusion it was the CPU.

krazmuze · April 4, 2024

I have come to no conclusions that CPU is doing the 2D drawing, I simply pay attention to dev updates. The devs said there is a performance problem because the 2D sprite depth ordering has to be managed by the CPU, and that the solution is to pay the art cost of baking depth layers into maps and objects to make them 2.5D, thus the GPU has the information it needs to do the sprite depth ordering. This is not conjecture on my part and I do not need to be educating you, do that yourself by watching and reading their updates. They even deep a deep dive showing the depth map view as well as demonstrating skyscrapers.

Disregard that cities have more zeds by default which is a CPU simulation so it will obviously interfere with CPU doing 2D drawing, but even if you kill all the zeds or spread the zeds out - cities have more sprite objects of different types with lots of different buildings with more floors and rooms, and obviously the drawing update rate increases the faster your speed. So obviously it would be dumb to add skyscrapers making that even worse, which is why it makes sense to be doing the 2.5D GPU optimization.

Using the GPU is why all those 3D games you mention are far better performance despite being more complex 3D. 2D games use the CPU unless they convert their art to be 2.5D so the GPU has something to work with. This game has a lot of system simulation so it obviously already has a heavy CPU load.

I will leave this thread so you can keep telling the devs they do not know how to profile and optimize their code, despite their past updates where they said they figured out the performance problem and the solution.

April 4, 2024

A city has tens of thousands of tiles instead of hundreds; thousands of isoobjects (like interior decoration and objects) and up to eight layers to sort through. Trees and grass on one level are much simpler.

Think of it like unpacking a Pickup. You have 36 boxes. The only thing you can change is how fast you move them, not how many you move at each time. (The forest.)

Now you try to unload a box truck. You have 180 boxes to move.

Now imagine that box truck might be 8 trucks high and has half a dozen boxes nested inside each box that need to be sorted first. (A city.)

What's going to happen? The job's the same -- move a box. But it takes much longer to complete because there's so many. You can only move them one at a time -- faster when fitter, but still only one at a time.

Lot of programming boils down to this sort of thing -- how long it takes to interact with and sort a group of things; and unfortunately you do eventually hit a wall where you just can't go faster without significant consequences (if at all). ( That's also why having an arbitrary goal like using "100%" of your gpu or cpu is meaningless in practice, because sometimes things can only happen one way to get your end result.)

(also you have to do it in reverse because the game is 2d and isometric, hence issues with occlusion.)

b42 is in theory the fix -- take the cpu out of the equation so it can do other things (depth buffer on the gpu); reduce the number of boxes to sort by putting them all in one big box, etc. But there are trade offs to this -- we can't use that depth buffer for other things anymore, it'll need more ram, or the game will have to reprocess that megabox if something changes.

btw, I'm self taught and have only a cursory knowledge of this stuff myself, picking up what I could while I worked here for the past 12 years and I am not the people that wrote it -- tested the results and pointed out things, though. So I'm not going to write an exam on it, frankly. No idea why you'd see the same results on significantly different hardware -- I certainly don't. Iirc in testing, hiding the interiors was simply the difference between 60 fps and 40 fps driving through West Point, on a middling system, so we did it. Easy trade, since most people never noticed.

(3D fpses have far less to sort compared to 2d isometric games. Dramatically less. Old isometric games tended to be much simpler and cheat an awful lot to pull it off -- see how many floors you can build in the sims for example. GTA is also full 3D, so isn't applicable.)

(and you're not wrong -- zombies are a significant cost themselves. Nothing is free.)

(CPU sorts the tiles then passes the list to the gpu; the cpu isn't drawing the tiles unless you're using some sort of software renderer to make up for missing features in your gpu.)

(Cores don't matter -- not everything can be multithreaded. You'll be limited by how fast the main thread or the rendering thread runs which will ideally be on different cores, but thats up to your os. And note GHz is fairly meaningless these days; generally cpu manufacturers use other tricks to get more operations per cycle, so you can't compare a 2.4 ghz system from 2024 to 2011 on ghz alone.)

SilentLight · April 5, 2024

22 hours ago, EnigmaGrey said:

I'm self taught and have only a cursory knowledge of this stuff myself, picking up what I could while I worked here for the past 12 years and I am not the people that wrote it

I can appreciate that, I think as programmers we're always in a cycle of learning. My own experience is a mixture of self-taught, University and work (albeit the latter not games). I wanted to reference a video I had seen a while back which taught how tris below a certain resolution size (rather than count!) causes rendering to use more resources. Something about how a 2x2 tris can chew out resources. Cannot find the video for the life of me.

Quote

generally cpu manufacturers use other tricks to get more operations per cycle, so you can't compare a 2.4 ghz system from 2024 to 2011 on ghz alone

Would it surprise you to find Jedi Knight Outcast could run on a PC with a 220MHz CPU? It was also available on the GameCube, which ran 486MHz. So the idea of an older game running smoother than a newer game on a newer system would strongly suggest it isn't strictly about lifting power. I don't even recall my old system specs, but I doubt the RAM was much more than 512MB either back then. If you ever grab a copy of Star Trek Bridge Commander (extremely old game) spawn in something like 40 ships.

Quote

3D fpses have far less to sort compared to 2d isometric games

What sort of sorting do you refer to?

If we're talking large quantities of units all at once on one screen using a similar viewing angle (Isometric), then Age of Empires II was around during Jedi Knight: Outcast's time, and also could run on a 220MHz (if you want a 3D isometric game, Empire Earth is the next closest, but 220MHz wasn't sufficient - you needed something like 456MHz or greater). AoEII could have easily over 200 units a team, and I believe, 8 players, plus an ungodly number of buildings. Empire Earth had something like 300 units a team and 8 players. If the unit cap doesn't impress you, then Red Alert 2 with unlimited units for all sides which included 8 player online play might. All of them are isometric games, and involve bulk unit navigation.

During my Uni days, I wrote an Isometric game with a lot of units which lagged to absolute hell. However, it wasn't due to the graphics, as every unit was literally a sprite of a circle. It was because every unit had individualised pathfinding (and it was built in not the not particularly efficient uni's copy of GameMaker). The CPU trying to calculate custom pathing for every single 'ball' destroyed the FPS.

The correct solution (for that niche game) was a 'pathing master' (similar to Valve's Left4Dead wayfinding director), which calculated a limited set of routes, and then basically assigned balls (heh!) to each one. I think more advanced algorithms use grouped pathfinders (so you have a central node for a group, and the group then moves as a semi-disorganised collective), which I think is similar to what Project Zomboid has, although I must confess I am not familiar with the code.

I've got to say, for pathfinding zombies, Left4Dead's system still amazes me (the zombies can pathfind a route to the character by using the same movements they did - essentially, players leave a "trail"; so even if you climb up an object they're still able to reach you). How that game does not lag when hordes spawn is still something I have not figured.

Quote

CPU sorts the tiles then passes the list to the gpu

This might be a bit old school and you might have newer terminology (feel free to get technical, I'll go look it up), but can I ask what sort of rendering algorithm do you use? Is it reverse painter's algorithm?

April 6, 2024

Sorting in terms of occlusion; the frustum of s 3D fps/3rd person over shoulder will contain far fewer things than an overhead camera giving 360 degrees of vision.

AoE capped max units for the entire map. It was a very small number compared to PZ; we'd exceed the cap pretty much immediately on spawn if we were to do that. Red Alert 2 letting its fps chug if too many units were present wasn't a great approach. I just don't get the point of this -- yeah, its cool (I lived through it :p), but we can't roll back the clock and write the game like it's 1999 anymore (though the first versions probably count as they could be run on hardware from 2001 or so. It wasn't until 2018 that PZ required shaders. If Intel GMAs weren't so common, we probably would've advanced a lot faster -- but it couldn't handle the drawing the 3D models with em).

Similar thing for Jedi: Academy, but we made a very generalized game spanning 3 platforms simultaneously in Java, not one custom built and optimized for specific hardware, as it's ports likely were.

Rewriting the game in C and assembler isn't an option for us. Neither is greatly scaling it down, unfortunately.

Iirc, caching is already done for PZ's pathfinding, as is reducing individual zombies into large hordes when unloaded to reduce costs. Modified A* to handle the possibility of 4 walls on a single tile? But again, not my job to know the intricate details of it. Same deal for whatever rendering algorithm is used.

You're always welcome to poke around with IntelliJ IDEA's decompiler if you want. It'll probably provide clearer answers.

SilentLight · April 9, 2024

On 4/6/2024 at 11:44 AM, EnigmaGrey said:

Sorting in terms of occlusion; the frustum of s 3D fps/3rd person over shoulder will contain far fewer things than an overhead camera giving 360 degrees of vision.

Would not the range also be a factor? A 3D FPS can look off into the event horizon (in older games, requiring distance culling techniques, and in newer games, sprite and poly management in the form of low to high resolution models based on distance), an isometric top down the camera generally points to the floor, and has a fixed rendering area.

I have a question. So, I know if one holds right-mouse they can 'move' the viewing angle around and into the distance - in regards to the zombie viewing update tick option, where zombies that can't be seen are updated less often - is the visibility determined based on the viewing occlusion angle of the character (which to me it seems to be?), the viewing occlusion of the viewing window itself (I.E. what the player can actually see), or both, or something else?

Quote

AoE capped max units for the entire map. It was a very small number compared to PZ; we'd exceed the cap pretty much immediately on spawn if we were to do that.

Units were capped, but buildings weren't (I say this as someone who used to spam castles progressively to the destination). You could also bypass this cap with monk conversions of enemy units. In-fact, the only way you could tank the game's performance was if you used the cheat code for the sports car, which itself fired lots of individual (!!!) projectiles. About a thousand of them on-screen all firing at once would tank the FPS - if it was viewed - but to be fair, the viewing area could only really fit about a thousand of them anyway.

Quote

Red Alert 2 letting its fps chug if too many units were present wasn't a great approach.

I never found RA2's upper bound for triggering FPS lag. The closest you could get was to pack an area full of infantry, and then nuke them. You'd see maybe a split second of FPS lag (as it created an animated death for every unit simultaneously due to the nuke blast), but at that point I lost count of how many units there were.

Bottom end estimate was probably a thousand, but could have been 3 to 5k. If you have an older copy, pick a large multiplayer map, and literally play solo (I think it was a bug that allowed you to initiate a map with no enemies). In the newer copies, pick a large map but a very dumb AI, then capture (and sell) all except their construction yard so they don't resign. Then consume all the resources. Or if you're able to make a custom map, make one with a ton of ore and trap the enemy team on an isolated island. Pick any of the Soviet factions, mass produce infantry, then self-target a nuke launch at your own units.

I think the trick RA2 used to keep FPS down despite such high quantity of units is via the use of referencing. That is to say, it loads one default template unit instance, and all other units are references to that default template, with the exception of state changes, animation timing differences, sprite image direction, XP and health. Which admittedly doesn't sound like a reference but it makes sense in my head.

Quote

but we can't roll back the clock and write the game like it's 1999 anymore

I agree there are newer and better techniques (such as the low-to-high poly models and tris-efficient spritework management), my point was there are older games that follow complex variables that don't experience stuttering. The stuttering to me, whilst it visually impacts FPS (the symptom), does not strike me as a graphical cause. An easy way to emulate what I'm thinking of, would be to:

Create a basic game loop (simple background)
Cap FPS to 60
Have a square move right 1 pixel every step of every frame (we'll skip multiplying by delta time)
On every frame loop, have the game do some massive, non-graphical task (we can emulate this with a randomised sleep command between 0.5 to 4 seconds)

What you should experience is the square 'stutters' across the screen. Even though it's a square. With the world's most basic background.

Quote

Similar thing for Jedi: Academy, but we made a very generalized game spanning 3 platforms simultaneously in Java, not one custom built and optimized for specific hardware, as it's ports likely were.

Jedi Academy was sort of just thrown at various machines (in-fact, it was never optimised for the GameCube, and didn't perform as well as PC). Ravensoft ultimately made the game code public for anybody to use (and at the time it had a very strong modding community). I can appreciate the difficulties with Java, however - I had to deal with it's ugly, unheard of brother Rhino, which is this horrible mis-mash of both Java and Javascript, an "engine" that essentially creates the worst of both languages. I had to use it to integrate 3 completely different clinical systems of varying ages (some as old as 30 years!) across what could be charitably described as a random assortment of every OS and file format imaginable. And it had to be stable. And process millions of records. What kind of programming language still has a stack of timedate bugs in 2024? Java does.

I cannot imagine the graphical rendering within Java is much fun. Last time I touched that was for a (now old fashioned) mobile game for Uni coursework. Which was supposed to be a Pacman variant with smart ghosts that did not require A* pathfinding (so you could have an infinitely regenerating set of levels). The algorithm did a little too well and the ghosts would corner Pacman on all sides and kill him within about 30 seconds. Then, somehow, the ghosts realised walls didn't matter to them and during several games found all four of them just converging on my position like some sort of nightmare. When the lecturer looked over my shoulder he drily commented 'well, at least the ghosts are realistic'.

For those curious, the non-A* pathfinding algorithm for my old "Pacman" ghosts goes:

Get player direction relative to self
Between all four directions, pick a direction that is closest to direction to player.
If there are only two directions, continue in your current direction (if no current direction, pick one towards player)
If you encounter a wall and no new directions, turn back
If you encounter another ghost, turn back
When at a new junction, you cannot pick the direction you've come from.
If the player gets a power pill, invert the player direction (still following same rules, just heading away from player)

Quote

Rewriting the game in C and assembler isn't an option for us. Neither is greatly scaling it down, unfortunately.

I would never advise this anyway. C is an absolute nightmare with third party library imports (at least use C++ my man and make use of SFML!) and assembly is for CPUs and micro-controllers.

If I was going to propose a rewrite (which I wouldn't, not without a proper examination of the code), I would suggest either a dedicated game engine tool to streamline development (I personally emotionally dislike Unity, but I can't deny many games are built in it successfully), or, because I'm bias, Python using the pygame library, given Python is multi-OS, pygame does a lot of the initial legwork, plus Python can import so many third party libraries.

Quote

You're always welcome to poke around with IntelliJ IDEA's decompiler if you want. It'll probably provide clearer answers.

Thank you. I intend to poke around PZ's code once I free up my current work schedule. I only wanted to write my hypothesis for the stuttering so developers who know it inside and out have a headstart.

Do you know if there is some way I can make a duplicate copy of PZ, such I could still boot up the duplicate? That way, if I experiment with the code and it goes awry, it's not going to ruin my active installation, and I can run some before and after tests.

I've got a few objectives I'd like to achieve, mostly involving looking for 'easy wins' in terms of optimisation, extensibility, and digging deeper into some unusual interaction mechanics (which I think might be bugs but I only have speculation for now).

April 9, 2024

Yes, distance is a factor. But remember that 3D game engines have been built to handle this via decreases in detail or even basic fog to optimize it. We don't benefit from any of that/have to roll our own stuff because the game is 2D and isometric. Everything is drawn to a larger-than-the-screen texture (twice the size to facilitate zoom but this amounts to like .. a 5 tile border). That's how things like holding ctrl and dragging the mouse to the side of the screen work. This is also why 4K is such a boondoggle because now you're working with an 8-12K texture instead of "4k." That's a lot of pixels to push for any GPU on top of waiting for a theoretical 50 x 50 x 8 tiles and everything else on the pile (overlays, fov, lighting, objects, moving objects .etc) to be sorted. This shouldn't matter in b42.

You can optimize this to a limited extent to gain a bit of performance (maybe draw the tiles in a circle so that there's 1/16-1/8th as many) but it's not really worth it. (Or you reduce the size of that texture and just let people deal with the black borders if they move too quickly or zoom out. That's a no-go.)

In AoE, buildings were simple 2D sprites, weren't they (a composite of a base image and a depth image to fake 3D effects?)? I can't imagine it was costly at all (especially given the lack of variety). Older games tend to use tricks like this or baking everything into the ground plane (like C&C, Baldur's Gate), whereas we have separate Z levels (think the early 90s X-COM. In fact, X-Com Apocalypse is probably a good analog for the challenges that PZ faces in its current, public form). Most 2D isometric games don't do this even today.

Rhino: If you have the option to use JDK 17 or above and can use the ZGC pauseless garbage collector then I'd do it. But if Rhino requires reflection to facilitate scripting, you're going to have a similar bad time to us with Khalua2. You need to avoid small allocations in Java to avoid it bogging down. Things like foreach can have unexpected costs. Profile constantly. I don't see the point in dick measuring languages -- you can bog down anything or blow your foot off. Use a third party lib for datetime if it affects you (Apache always has good libs)? I mean, my heart belongs to straight C89, but all that matters is you get your work done with what you have.

Graphical rendering isn't really done in Java. LWJGL provides bindings for GLFW which provides a wrapper for various DLLs on the system of your choice, such as OpenGL.DLL or Wing.dll. It's no big deal beyond the theoretical overhead of JNI.

C++ has the same problem with third party libs being crud. That's just universal to C and C++. Honestly it's why I rarely use either anymore. People tend to resist writing self-documenting code and the preprocoessor + pointers seem to really encourage this, imo. Not a bad thing, but it means you can start a project with a really nice library, hit a wall, go check the source and just have a complete "wtf" moment. With languages like Java or C#, it's usually clearer what the intent was.

We'd spend years jamming PZ into an engine that was never built to handle something like PZ. It's custom or bust, really. The only change i"d personally like here is using a proper C build of Lua instead of a Java implementation, as "it'd at least help" with performance. Note that Khalua2 requires reflection, which means an utter crap ton of trash, which means bogging down the rest of the game for the sake of mods / GUI. It's a fair trade, but it's just another hole to fill. (Note this is why we have FPS settings for the GUI in options and cap it at 30 FPS -- or every second frame).

I'm not going to touch on duplicating the game, sorry. I'm not comfortable with that.

SilentLight · April 9, 2024

Following up on the invite to take a look at the decompiled code, having done a cursory glance, I must say I find the design, shall we say... surprising? It would also explain why you thought my explanations were weird.

So, in typical object-oriented design, one would expect there to be either classes or subclasses of a specialised object (usually, the rule of thumb is, any time a class specialises or deviates, it becomes a new subclass). For example, my comment about a 'Cupboard', I was clearly expecting some sort of 'Cupboard class'. Perhaps something like WorldObject->CupboardObject. Or 'WorldObject->SinkObject' and so-on.

Project Zomboid has thrown me because it seems to try to dump almost everything (with a few exceptions) into a class called "IsoThumpable". Whilst I get the object's intention is to standardise breakable and sound behaviours, the code within suggests it is also trying to be multiple classes at once, rather than a generic, standardised container that can then be subclassed into other classes.

For example, door, doorframe, window, floor, stairs, containers, paintable walls, walls, and more are all being set from within IsoThumpable's load function:

this.isDoor = var4.hasFlags(512);
this.isDoorFrame = var4.hasFlags(1024);
this.isCorner = var4.hasFlags(2048);
this.isStairs = var4.hasFlags(4096);
this.isContainer = var4.hasFlags(8192);
this.isFloor = var4.hasFlags(16384);
this.canBarricade = var4.hasFlags(32768);
this.canPassThrough = var4.hasFlags(65536);
this.dismantable = var4.hasFlags(131072);
this.canBePlastered = var4.hasFlags(262144);
this.paintable = var4.hasFlags(524288);

What tips us off subclassing ought to be used is several of these traits are mutually exclusive (unless Project Zomboid has some sort of even deeper mechanics I don't yet understand). For example, an object can't logically be a door, stairs, floor and door frame at the same time (and if it somehow can, that is extremely counterintuitive and violates policy of least surprise). Is there really a use-case where isDoor and isStairs are both true? Is there a use-case where a stairs can be plastered? Is there ever a door where door frame can be false?

[Less important: the flag numbers are a type of magic number and ought to be declared under a Java enum (see the planet example here).]

Now you could argue a tile could contain all, but we would expect them to be held as individual classes (so a list container that holds an IsoDoor, IsoFloor, IsoStairs, etc). An IsoThumpable that subclasses into IsoDoor (we would naturally assume all IsoDoors have door frames, so an IsoDoor would inherit from an IsoDoorFrame), IsoFloor, and IsoStairs. We then don't need to perform an 'isFloor' check because the class container itself infers that information already.

Subclassing would also allow you to give hints to both the physics engine and rendering simply on the basis of the class type. For example, an IsoDoor has a probability of being transparent (read: open) and thus would need to be checked for graphical rendering behind the object. An IsoDoorFrame is always presumed to be open (on the presumption it is missing a door) and thus always transparent. But an IsoFloor (assuming all IsoFloors are non-transparent) could never show anything behind it, therefore the rendering tool can automatically infer not to visually render anything under the IsoFloor.

Yes, you could check "isFloor", but then what does the renderer do if both isDoor and isFloor are both true? At that point you will end up with a lot of convoluted if-statements trying to determine weird case scenarios that should never logically exist.

I suspect this code layout would make it difficult to optimise visual rendering if there is an all-singing, all-dancing singular object. And my suggestions aren't entirely absurd, I notice for example, IsoRadio is a subclass.

In terms of making file loading faster, I would propose the following example change:

//This isn't a working example, purely draft code to give you an idea

this.TraitFlags = var4.getFlags(); 

//...//

public boolean getTrait(NameOfEnum PassEnumForTraitHere):
    return this.hasFlags(PassEnumForTraitHere);

If I misunderstand anything about Project Zomboid's code, please bear with me there is quite a lot here to process and it usually takes me about 2 weeks to 'get up to speed' on a novel project.

April 9, 2024

IsoThumpables are objects that can be thumped; it's inherited by IsoObjects which may be things like IsoDoor -- but every IsoDoor is ultimately an IsoThumpable. It's not that convoluted.

You're looking at byte flags for loading isoThumpables from a save file. *Shrugs* I don't see how replacing a couple if statements with a full on trait system would make a difference in loading files that are so tiny. It's premature to blame your problems on load speed, frankly.

OOP is often needlessly complex/verbose to the point it's difficult to conceptualize and expand upon. It's increasingly fallen out of favour for game development, giving way to entity-component systems. It's really what we should have done back in the day. Regardless, there'll be a lot of jank that developed out of necessity or while transitioning from C# to Java from back in the day.

Welcome to the infinitely interleaving alpha-beta-release-maintenance cycle that is Agile.

SilentLight · April 15, 2024

On 4/9/2024 at 6:03 PM, EnigmaGrey said:

Welcome to the infinitely interleaving alpha-beta-release-maintenance cycle that is Agile.

I can appreciate that! I also recognise everyone has their preferred style of coding design and layout. Obviously, my observations were cursory, given it'd take me a while to get familiar with the code, especially with work.

But I come back with gifts! I recognise airing what might be perceived as critiques isn't the best way to endear myself to a community. I managed to find the videos I wanted to mention, including that one with the weird tris optimisation I mentioned earlier.

I'm not sure how one embeds videos on this forum, so I'll just copy the titles and hyperlink with underline and hope you'll take a look.

When Optimisations Work, But for the Wrong Reasons - This mentions the tris stuff I was having a hard time conveying. The visuals are so much easier to understand.

How Quake Failed their way to Success - Deceptively titled, but this refers to the 'binary tree' rendering Quake engine stuff I mentioned earlier. It doesn't cover the 'octotrees' (I don't think they appeared until either Quake 2 or 3). The proper term I should have used was 'Binary Space Partition' (BSP), but my brain doesn't retain jargon. Note, it isn't purely for graphics, it also includes physics, such as object loading.

How Games Have Worked for 30 Years to Do Less Work - This covers some more general stuff relating to the above two, and mentions the thing I was trying to say earlier hamfistedly (I meant frustrum culling and occlusion, but I oversimplified my comments to occlusion).

On reflection, I think I partially misunderstood your earlier comment about '3D iso' not having the same optimisation profile as a 3D game. In my mind I was thinking 'but surely they're both 3D?', however I recognise trying to adopt a 3D fustrum culling with freewheeling angle is a bit overkill, because you can make better optimisations by always assuming only specific texture faces are visible (for example, you can never see from the other side, or any other angle, for that matter).

BSP also presumes a sort of, free moving 3D space, and I think having a tree node for every ISO tile in existence would be bloated and overkill, so inbetween work I'm trying to design a sort of... optimised variant, I guess? But I don't know how compatible it would be with your code. If I can think of a viable design concept I'll try to draft something up in Python, although my schedule is frustratingly packed.

I also have another gift - generated audio for Project Zomboid TV broadcasts:

https://theindiestone.com/forums/index.php?/topic/71009-peace-offering-generated-audio/

BettyPiper · December 16, 2024

On 4/3/2024 at 9:52 AM, SilentLight said:

Whilst I know the indiestone devs are already aware that there is FPS stuttering whilst driving in Project Zomboid, commentary suggests no-one is entirely sure of the cause, and based on my previous experience in programming and observations during sandbox mode driving, I think I've got a good idea what the cause is, based on when it happens and who it happens to. I will explain my reasoning - and solution - below, but the summary is: the stuttering is caused by the game attempting to load too many objects all at once in every 'frame update' whilst driving.

What It Can't Be

We can rule out insufficient computer resources, because even players with powerful gaming computers report this phenomena. The game generally has a low tris count (a metric used to determine graphics demand and performance), and the CPU does not sound like it is 'firing up' or 'melting down' during the stuttering. Driving does not stutter even for low-end computer during certain situations (I.E. mostly empty main roads). This points us towards a suboptimal algorithm or piece of code that takes the game a long time to complete (so it isn't CPU intensive, but time intensive).

What It Most Likely Is (Based On Tests)

With sandbox mode and all zombies disabled, driving generally does not stutter until you hit Louisville. The fact stuttering is (somewhat) reduced based on the absence of the zombies gives us an important clue; it suggests when the game is trying to update or 'load in' zombies - even ones visually the player cannot see - there is a drop in FPS thoroughput performance. This tells us, as the driver is rapidly traversing through areas, the game is attempting to rapidly spawn, then despawn, large quantities of zombies, which means the thoroughput time performance is mostly spending on loading (I.E. it is a roundtrip memory performance or similar issue). This explains the lack of overall CPU demand.

The fact FPS massively drops when one first enters Louisville whilst driving (and is generally poor even after first entering), causing stuttering even without zombies, gives the first major breakthrough on the underlying cause. One thing Louisville is chock full of is buildings, and whilst one could suspect maybe the game is attempting to spawn zombies, I don't think that is where the major loading lag comes into play.

My suspicion is as the game loads in the buildings, it also attempts to load in every single object and item inside the building. In smaller buildings like huts and two storey houses, this brute force approach isn't a problem, but in Louisville where you have multistorey apartment blocks and large industrial complexes, the impact is a force multiplier (so a 10 floor building has ten times the loading time of a single floor building). My additional suspicion is, on top of loading the physical objects, the game also attempts to set things like the interior items loot tables and other things the player would not be able to access whilst driving.

We can reinforce this by the fact walking does not produce the stuttering in the same area; if it was graphical, walking would produce the same stuttering effects. If it is loading, however, walking significantly slows down the rate of which objects become visible, giving the system time to 'catch up' or even 'keep pace'. I bet if you turned a walker into a runner who moves at the same speed of a car, you would likely induce the same stuttering effect (it is not related to driving).

The Solution (For Objects)

This points us to an elegant solution. The game should not attempt to load indoor items (ever!) whilst the player is driving, and should keep the interiors 'blacked out' (for aesthetic purposes). The game should only attempt to load the interiors as the player exits the vehicle. In multiplayer, the loading should be client-centric (so the server can see all objects, but the clients only load interior objects when they leave a vehicle).

Now, this might work for say, a mostly hidden building with walls, but what is the solution for say, a grocery store with giant windows, or a recently burned out interior that changes every run? Or a custom player base? The game needs only spoof the interior. It can pre-render a single image "screenshot" of the interior, which it only uses when the player is driving (it does not load in any objects besides exterior, it just visually spoofs the entire interior as a screenshot layered behind the front-Z exterior). The moment the player exits the vehicle, it will update the screenshot, storing it back to memory and render the interior properly with all objects.

The Solution (For Zombies)

As for zombies, the game should not be allowed to even attempt to spawn zombies in non-visible areas (E.G. trees, buildings) whilst the player is driving, especially at speed. Zombies on upper floors of buildings beyond the first, also should not spawn whilst driving (as the player cannot reasonably get there whilst driving). It can have 'placeholder zombies' (which are just coordinate points rather than a full model load), which spawn full object zombies + pathing algorithm the moment the player's vehicle either comes to a halt or drops to a particular speed (E.G. below 15 MPH). Note, only non visible zombies are excluded here, so the zombies that still spawn to block roads etc are fair game. They may have to be reduced to stop stuttering but that number would need to be based on unit testing (which I don't have access to).

Alternatively, as a kludge, you could create a spawn probability algorithm, where the faster the vehicle is travelling, the lower the overall probability of spawning a zombie is in that given frame (and thus, the associated object load with it). This would likely create the effect most players fear and desire during driving - that zombies will swarm their vehicle the moment it comes to a sudden crashing halt. It would also reduce spawn rates and thus reduce stuttering whilst driving.

A patch could be retroactively implemented by modders for the current build by making any sort of 'household object' (tables, chairs, sinks, cupboards, beds, etc) simply not load for clients whilst driving. It might make interiors look "janky" visually, but it would likely result in a much smoother driving experience if implemented correctly.

For instance, when it comes to real-world car registrations, using tools like a license plate lookup https://www.faxvin.com/license-plate-lookup/dmv can simplify accessing vehicle history without bogging down processes, much like optimizing game algorithms to streamline loading times.

I am not familiar with the game's code, for the record, this is just based on my observations during testing.

The idea that the stuttering stems from loading objects and zombies in bulk during driving seems logical, especially considering how the game behaves differently at slower speeds. Your proposed solutions, like blacking out interiors while driving and using pre-rendered placeholders or screenshots, sound like they could effectively reduce the strain on the system.

Edited December 16, 2024 by BettyPiper

BettyPiper · March 21

On 4/3/2024 at 9:52 AM, SilentLight said:

Whilst I know the indiestone devs are already aware that there is FPS stuttering whilst driving in Project Zomboid, commentary suggests no-one is entirely sure of the cause, and based on my previous experience in programming and observations during sandbox mode driving, I think I've got a good idea what the cause is, based on when it happens and who it happens to. I will explain my reasoning - and solution - below, but the summary is: the stuttering is caused by the game attempting to load too many objects all at once in every 'frame update' whilst driving.

What It Can't Be

We can rule out insufficient computer resources, because even players with powerful gaming computers report this phenomena. The game generally has a low tris count (a metric used to determine graphics demand and performance), and the CPU does not sound like it is 'firing up' or 'melting down' during the stuttering. Driving does not stutter even for low-end computer during certain situations (I.E. mostly empty main roads). This points us towards a suboptimal algorithm or piece of code that takes the game a long time to complete (so it isn't CPU intensive, but time intensive).

What It Most Likely Is (Based On Tests)

With sandbox mode and all zombies disabled, driving generally does not stutter until you hit Louisville. The fact stuttering is (somewhat) reduced based on the absence of the zombies gives us an important clue; it suggests when the game is trying to update or 'load in' zombies - even ones visually the player cannot see - there is a drop in FPS thoroughput performance. This tells us, as the driver is rapidly traversing through areas, the game is attempting to rapidly spawn, then despawn, large quantities of zombies, which means the thoroughput time performance is mostly spending on loading (I.E. it is a roundtrip memory performance or similar issue). This explains the lack of overall CPU demand.

The fact FPS massively drops when one first enters Louisville whilst driving (and is generally poor even after first entering), causing stuttering even without zombies, gives the first major breakthrough on the underlying cause. One thing Louisville is chock full of is buildings, and whilst one could suspect maybe the game is attempting to spawn zombies, I don't think that is where the major loading lag comes into play.

My suspicion is as the game loads in the buildings, it also attempts to load in every single object and item inside the building. In smaller buildings like huts and two storey houses, this brute force approach isn't a problem, but in Louisville where you have multistorey apartment blocks and large industrial complexes, the impact is a force multiplier (so a 10 floor building has ten times the loading time of a single floor building). My additional suspicion is, on top of loading the physical objects, the game also attempts to set things like the interior items loot tables and other things the player would not be able to access whilst driving.

We can reinforce this by the fact walking does not produce the stuttering in the same area; if it was graphical, walking would produce the same stuttering effects. If it is loading, however, walking significantly slows down the rate of which objects become visible, giving the system time to 'catch up' or even 'keep pace'. I bet if you turned a walker into a runner who moves at the same speed of a car, you would likely induce the same stuttering effect (it is not related to driving).

The Solution (For Objects)

This points us to an elegant solution. The game should not attempt to load indoor items (ever!) whilst the player is driving, and should keep the interiors 'blacked out' (for aesthetic purposes). The game should only attempt to load the interiors as the player exits the vehicle. In multiplayer, the loading should be client-centric (so the server can see all objects, but the clients only load interior objects when they leave a vehicle).

Now, this might work for say, a mostly hidden building with walls, but what is the solution for say, a grocery store with giant windows, or a recently burned out interior that changes every run? Or a custom player base? The game needs only spoof the interior. It can pre-render a single image "screenshot" of the interior, which it only uses when the player is driving (it does not load in any objects besides exterior, it just visually spoofs the entire interior as a screenshot layered behind the front-Z exterior). The moment the player exits the vehicle, it will update the screenshot, storing it back to memory and render the interior properly with all objects.

The Solution (For Zombies)

As for zombies, the game should not be allowed to even attempt to spawn zombies in non-visible areas (E.G. trees, buildings) whilst the player is driving, especially at speed. Zombies on upper floors of buildings beyond the first, also should not spawn whilst driving (as the player cannot reasonably get there whilst driving). It can have 'placeholder zombies' (which are just coordinate points rather than a full model load), which spawn full object zombies + pathing algorithm the moment the player's vehicle either comes to a halt or drops to a particular speed (E.G. below 15 MPH). Note, only non visible zombies are excluded here, so the zombies that still spawn to block roads etc are fair game. They may have to be reduced to stop stuttering but that number would need to be based on unit testing (which I don't have access to).

Alternatively, as a kludge, you could create a spawn probability algorithm, where the faster the vehicle is travelling, the lower the overall probability of spawning a zombie is in that given frame (and thus, the associated object load with it). This would likely create the effect most players fear and desire during driving - that zombies will swarm their vehicle the moment it comes to a sudden crashing halt. It would also reduce spawn rates and thus reduce stuttering whilst driving.

A patch could be retroactively implemented by modders for the current build by making any sort of 'household object' (tables, chairs, sinks, cupboards, beds, etc) simply not load for clients whilst driving. It might make interiors look "janky" visually, but it would likely result in a much smoother driving experience if implemented correctly.

I am not familiar with the game's code, for the record, this is just based on my observations during testing.

When preparing for a road trip, it’s essential to ensure your vehicle is in top condition to avoid any unexpected surprises. A thorough check-up of your car’s engine, tires, and brakes should be the first step. But one often overlooked aspect is verifying the vehicle’s history, especially if it’s a used car or if you’re renting one for the trip. It's important to make sure the car you're driving hasn't been reported as stolen or has any hidden issues. You can do this by running a quick VIN check, which provides a comprehensive history of the vehicle. This step can save you from potentially serious problems, like being stopped for driving a stolen car. A reliable VIN check tool https://www.faxvin.com/vin-check/stolen can give you peace of mind by revealing whether your vehicle is clear of any theft claims. With the right precautions, you can focus on enjoying the journey ahead without worrying about the vehicle's past. Whether it’s a short weekend getaway or a long road trip, making sure your car is safe and legitimate ensures you have a smooth, enjoyable ride.

You’ve done some solid detective work here! The stuttering while driving in Project Zomboid is definitely a tricky issue, but your theory makes a lot of sense.

Edited March 21 by BettyPiper

Sign In

Stuttering Whilst Driving - Cause

Recommended Posts

SilentLight

Modin

krazmuze

SilentLight

SilentLight

krazmuze

SilentLight

krazmuze

Guest

SilentLight

SilentLight

krazmuze

Guest

SilentLight

Guest

SilentLight

Guest

SilentLight

Guest

SilentLight

BettyPiper

BettyPiper

Create an account or sign in to comment

Create an account

Sign in

Browse

Activity