ZGC causes lower performance on Linux (especially modded)

[ulsac]

So I've been wondering why my game is massively underperforming in Linux lately to the point where I have to either enable Proton compatibility (terrible on my DE) or boot in Windows just to play modded since the game is unplayable with enough mods such as with Sophie's collection. After some profiling using perf, I found that a lot of time was spent on ZGC symbols and decided to discard the flag. The results were night and day, removing a lot of the stutterings I had when driving and leaving the game playable again.

 

For reference, here's the json file for my game:

 

{
	"mainClass": "zombie/gameStates/MainScreenState",
	"classpath": [
		".",
		"commons-compress-1.18.jar",
		"istack-commons-runtime.jar",
		"jassimp.jar",
		"javacord-2.0.17-shaded.jar",
		"javax.activation-api.jar",
		"jaxb-api.jar",
		"jaxb-runtime.jar",
		"lwjgl.jar",
		"lwjgl-natives-linux.jar",
		"lwjgl-glfw.jar",
		"lwjgl-glfw-natives-linux.jar",
		"lwjgl-jemalloc.jar",
		"lwjgl-jemalloc-natives-linux.jar",
		"lwjgl-opengl.jar",
		"lwjgl-opengl-natives-linux.jar",
		"lwjgl_util.jar",
		"sqlite-jdbc-3.27.2.1.jar",
		"trove-3.0.3.jar",
		"uncommons-maths-1.2.3.jar"
	],
	"vmArgs": [
		"-Djava.awt.headless=true",
		"-Xmx8g",
		"-Dzomboid.steam=1",
		"-Dzomboid.znetlog=1",
		"-Djava.library.path=linux64/:.",
		"-Djava.security.egd=file:/dev/./urandom",
		"-XX:-OmitStackTraceInFastThrow"
	]
}

 

I've also run some tests just to confirm if I was not exaggerating and plotted my FPS while driving around riverside on a brand new save both modded and not modded, with and without ZGC. The plot is attached. The points are instant FPS extracted from Mangohud using a single game execution, and lines are the average of all values.

[Beard]

I was not able to replicate it on my Linux. What distro are you using? And what version of Java do you have installed on it? And what are your PC specifications?

[ulsac]

I'm running Arch Linux with pretty much everything up-to-date.

 

lucas@lucas ~$ uname -a
Linux lucas 6.8.9-arch1-2 #1 SMP PREEMPT_DYNAMIC Tue, 07 May 2024 21:35:54 +0000 x86_64 GNU/Linux

 

System specs:

CPU: Ryzen 5 5600G

GPU: RTX 3070

RAM: 32 GB DDR4 @ 3333Mhz

 

Java version:

lucas@lucas ~$ java -version
openjdk version "22" 2024-03-19
OpenJDK Runtime Environment (build 22)
OpenJDK 64-Bit Server VM (build 22, mixed mode, sharing)

 

It looks like the game ships with its version of openJDK which is what it seems to be using for all my tests:

lucas@lucas ~/Games/steamapps/common/ProjectZomboid$ projectzomboid/jre64/bin/java -version
openjdk version "17.0.1" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu17.30+15-CA (build 17.0.1+12-LTS)
OpenJDK 64-Bit Server VM Zulu17.30+15-CA (build 17.0.1+12-LTS, mixed mode, sharing)

 

I believe I have tested using my machine's java by running the binary without adding anything to PATH or preloading any libs except for the linux64 folder but I can't for anything make mangohud work with it, it either does not work or breaks some dependencies and the game crashes. Therefore, I can't measure the FPS objectively. But from some simple benchmarking it seems to perform pretty much the same.

[Beard]

Hm, I personally use Arch Linux as well with similar specs and I have pretty much the exact same FPS when using both G1GC or ZGC. From my testing, G1GC pauses were a lot more severe and I could actually feel them happening during gameplay compared to just ZGC.
Are you certain that you only changed the ZGC flag? Have you used 8GB RAM on both the ZGC test and without it?
Additionally, try using the -Xms8G flag as well, just to see if it would be some sort of allocation issue, since this would preallocate the RAM and disable ZGC uncommitting it.

It does seem unusual to me that ZGC with Proton works perfectly fine on your system, but natively it does not, which makes me think there is something unusual with your Linux configuration, I personally use EndeavourOS to install my Arch, since I do not trust myself to properly set it up. Have you done a custom Arch install?

[ulsac]

4 hours ago, Beard said:

Are you certain that you only changed the ZGC flag? Have you used 8GB RAM on both the ZGC test and without it?

Yes, double-checked now.

4 hours ago, Beard said:

Additionally, try using the -Xms8G flag as well, just to see if it would be some sort of allocation issue, since this would preallocate the RAM and disable ZGC uncommitting it.

 

Just tried and results were the same. To be clear, stutters happen even after hour-long sessions which means the system is using all the memory it can.

 

4 hours ago, Beard said:

Have you done a custom Arch install?

I have installed it manually following the wiki if that's what you mean. It's not my first arch install so I don't believe it's anything wrong with my setup and if it was I honestly have no idea where to start debugging. Would honestly be interesting to test in other distros and I'll try this later. If it helps, I see the same results in both wayland and X11. Also done some tests on the native JVM in my machine and it seems somewhat better with ZGC, but the game crashes A LOT. As I only see these issues using ZGC and I have expendable RAM it won't bother me for now, but I'm reporting in case someone else seems the same problem.

I have found an old reddit post that says the game runs better in proton for that user too, so it's something to consider. I'll update with tests on EndeavourOS to see if I see any changes.

[Beard]

There are several other posts that specify that the Native version runs better, which is the same experience on my rigs, there should not be many reasons for Proton to run better, though I would need to look into those in a case by case basis to find the reason for that.
Let me know how the tests with the other distros go, as that might help pinpoint the issue.

Also, I am curious what filesystem you use. I assume just the default EXT4? It should not really cause any problems, but I am asking just in case it is anything nonstandard to add to the list of possibilities.

[ulsac]

So I ran some tests with a fresh install of EndeavourOS, and had somewhat similar results:

 

 

I've only run modded but I can provide more tests on vanilla if you need, but I believe the results will be similar. I've seen a lot more stutters (both ZGC and no ZGC) here but I guess it's expected as it's run on an HDD, while my setup is on an nvme.

 

To answer your question, my system runs on an nvme BTRFS, but the tests on EndeavourOS were done with the game installed on an ext4 partition. It's important to note that i'm running a pretty beefy modpack (400+ mods) so the disparity may be accentuated by that.

[Beard]

From further testing and actually recording the framerates on a graph, it seems like you are correct with the FPS, though not as much like when using as many mods as you:



Seems like there is about 10 FPS difference. Though I have tested a testing build of the game, where the FPS between them is basically identical. ZGC actually seems to have more stable framerate, which I believe is more important than occasional high framerate? (green)




I would conclude that you are right in your findings, though the next update should hopefully improve the ZGC implementation. There may also be a Java update coming (cannot confirm as of now) which should bring some ZGC improvements with it. I know that JDK 21 enables Generational ZGC, which should help in PZs case.
You can play without ZGC for now if it improves your gameplay, we will likely not retroactively change the defaults for now and keep using ZGC.

I have also tested it on Windows, and it seems like there is a difference there as well, but only by about 3 percent, compared to the 14 percent on Linux, so it is fairly unusual that Linux is hit by the ZGC implementation as much on the same hardware.

[ulsac]

6 hours ago, Beard said:

There may also be a Java update coming (cannot confirm as of now) which should bring some ZGC improvements with it. I know that JDK 21 enables Generational ZGC, which should help in PZs case.

I've seen some info on ZGC being significantly better on newer Java versions too, glad to know it may improve things. Wish I could do more tests with a more recent version of Java but, as I said, I was seeing too many crashes to reliably test things. MangoHud was also not working there.

 

6 hours ago, Beard said:

I have also tested it on Windows, and it seems like there is a difference there as well, but only by about 3 percent, compared to the 14 percent on Linux, so it is fairly unusual that Linux is hit by the ZGC implementation as much on the same hardware.

Yeah, way above my paygrade to debug and understand why the impact is not as large on Windows but maybe filling a bug report to the guys from OpenJDK? I don't even know if it's something they'd look up.

 

Anyway, I'll keep playing without ZGC while waiting for the update as I have a lot of expendable RAM. Thanks for looking into this!