non-steam Let clients download required mod files from host

[stuck1a]

Hey there,

 

since I don't like Steam very much, I'd like to distribute the mods I develop for my server without Steam. Of course, I want to avoid that players have to do download the mods by theirself externally, so I thought about adding a script which let the client receive the required mod files/meta from the host itself instead of Steam (or even another server to conserve game servers resources).

I'm just wondering if this is even possible, since it would require a server-side lua script clients doesn't have yet which would break the checksum validation. Also de clients have to request/accept the file transfer somehow of course. So I'm looking for a possibility to execute the  lua script before the checksum validation occurs and send the files through the  excepted stream, the client excepts and there accepts.

Maybe a simpler possiblity would be to use at least one workshop mod which holds the mod downloader script only. This way, no checksum mismatch should occur and there would be an interface on both sides of the line, so sending/accepting the data shouldn't be the problem. It might also increase the hurdle for use on pirate servers  (at least somewhat), because of the requirement to use at least that one workshop-distributed mod.
I'm sure there are some possible solutions by using sh, php, etc, but I'd like to make this operable on hosted gaming servers as well. Usually you have no permission to execute arbritrary scripts there. (I use one of them for PZ myself as this is way cheaper than using a common/private servers for  game hosting)

 

So before I spend hours trying to figure out that this is absolutely impossible, I thought I'd first ask here, what do you think and/or know about possible restrictions.

 

 

Just another spontaneous thought about possible uses
I guess such a downloader script could also allow to rollback workshop mods client-side, if necessary, to prevent the issue where joining is impossible, due to newer mod files than on server-side (which is very common on servers with dozens of workshop mods...) Also servers which have to be restarted every 1-2h to allow clients to join anytime without waiting for the next restart are really annoying. I mean, updating mods once per day or two would be more than enough, I think.

This shouldn't be a problem, because any downgraded (workshop) mod should receive the latest version again from SteamCMD, as soon as  they are required when attempting to join another (or the restarted) server. Just not sure about any integrity checks like reading manifest etc.

 

The longer I think about it, the more other use cases I can imagine...

 

Thanks in advance and best regards,
stuck1a

 

 

Edited August 7, 2022 by stuck1a
typo

[Zeros]

Heya, there's bunch of limitations to writing files using mod script but it's possible in a different way. It might not be accurate since I didn't read all of the decompiled sources.

First thing first, you can certainly pass data from server to clients using OnServerCommand and OnClientCommand. This part needs protocols and buffering.

 

1. You can't write files to any mod folder user didn't subscribe to. So users certainly need to subscribe to steam workshop item in order to be able to read / write files to the mod folder.

2. You can read / write binary files but it's limited to lua directory which I'm not sure if it's the game, the server or the mod's lua directory since I didn't test it yet.

3. You can read / write text files but it's limited to mod's folder.

4. You can load lua scripts inside the mod's folder using custom module loader. This will not the force checksum as long it's not from lua folder. I'm not sure if the game do checksum for folders other than 'lua' inside the media folder.

 

Based on these limitations, you can do something similar but since I'm not experienced with other type of asset loading, I'll say that you can certainly do mod downloader for mods that is updating their lua scripts frequently. This will require modders to create a lua script folder containing folders for old and new versions. Then, the server can just let users load which version the server has.

 

As for other assets, you'll need to ask modders experienced with custom loading of the other type of assets for their opinions.

[stuck1a]

Hi Zeros,

 

thanks for your reply.

In meantime, I've decompiled the Java part of the game and checked the given possibilities. By tweaking the classes, this faeture would be no problems at all, even without the modders having to follow certain rules. Well since this is primarly for my own mods, this wouldn't even bother very much, but I'll avoid recompiling any Java classes, since this will lead to some no-go disadvantages.

 

 

2 hours ago, Zeros said:

You can read / write binary files but it's limited to lua directory which I'm not sure if it's the game, the server or the mod's lua directory since I didn't test it yet

Yea, this should be the server-side mod data storage, located under .cache/Lua/ (for my linux distri).

 

 

2 hours ago, Zeros said:

This will not the force checksum as long it's not from lua folder. I'm not sure if the game do checksum for folders other than 'lua' inside the media folder

 

This is an interesting information, thank you!

I've just checked my latest logs on the fast and it looks like you'r right. But I will test it later, to be sure.

 

 

I will give it a try. By created a workshop based mod for the mod loader together with the fact to prevent checksum errors I think there is a chance to get this working, at least within my requirements.

I will report here, but this could take a while since I'm still primarly busy with mapping the environment of my current server project.

 

 

Best regards,

stuck1a

[Zeros]

No problem, I'm working on something similar too since my mod keeps breaking whenever I make a big change. Just don't know if there's a mod version check when joining a server.

 

Actually, I would like to say thank you since your thread gave me the direction for dealing with broken script update issue.

 

21 minutes ago, stuck1a said:

Yea, this should be the server-side mod data storage, located under .cache/Lua/ (for my linux distri).

Based on my experience with making a module loader, the path used by binary file reader / writer should also work the same way on client-side since there's no server / client validation. So, I did a quick test on window just now and confirmed it. Window user use the same folder for both server and client. Precaution is needed when using the binary file writer.

[stuck1a]

 

1 hour ago, Zeros said:

No problem, I'm working on something similar too since my mod keeps breaking whenever I make a big change. Just don't know if there's a mod version check when joining a server.

Yes, there is. Take a look into the steam config files under ".steam" there you will find some json-like information about your installed workshop objects which also contains a timestamp and fields like "must update" which depends on them. Just can't give you the exact paths since I'm still in office.

 

 

1 hour ago, Zeros said:

No problem, I'm working on something similar too since my mod keeps breaking whenever I make a big change. Just don't know if there's a mod version check when joining a server.

 

I agree, this might work as well. But like you said, it's way error prone, so a lot of testing should be done, escpecially whenever the game is updated.

Good luck with it, please keep me posted.

[Zeros]

I've been busy lately so now I would like to give report my current progress on the mod loader.

 

1. Turning it from SP into MP based mod loader requires me to ensure that player request for server response and wait for it. Once established, only then can they start loading the mods. I haven't tested it on dedicated server yet but for listen server host, you can only get server response few seconds after you're spawned using the 'OnPlayerUpdate' event. This is caused by listen server to start ticking right after the host player start the player update tick. For MP clients, I think it might be fine to use 'OnConnected' event but this will need further testing which I am planning to do tonight.

 

2. Mods using the mod loader feature can only be loaded after player update start ticking or when connected if possible. Custom event is needed for mods that initialize data using those two events.

 

3. There might be a need for waiting lobby to ensure that the connected players has fully loaded the mods.

 

4. I tried sending read-only loading order array through server and client command but it's not working since metatable are not transmitted together with the array. So, there's a need to apply custom metatable after the client receive the data.

 

5. After going through intensive testing, it's possible to make clients join without having version mismatch issue. This require modders cooperation though. They'll need to make multiple workshop pages.

- Server Mod: You don't need to enable the mod, just copy mod script files for server and clients to server folder. Server will let clients to download those client files when they joined the server.

- Client Mod: Contain some other files such as textures, item definitions and etc. These files usually don't get updated that frequent. It's fine to enable it in workshop mod list.

 

6. Need UI for download progress.

 

There you go. Sorry for the rough report though.

 

On 8/10/2022 at 11:31 PM, stuck1a said:

Yes, there is. Take a look into the steam config files under ".steam" there you will find some json-like information about your installed workshop objects which also contains a timestamp and fields like "must update" which depends on them. Just can't give you the exact paths since I'm still in office.

I'll take a look at it later.

Edited August 20, 2022 by Zeros

[stuck1a]

On 8/19/2022 at 9:44 AM, Zeros said:

I've been busy lately so now I would like to give report my current progress on the mod loader.

 

1. Turning it from SP into MP based mod loader requires me to ensure that player request for server response and wait for it. Once established, only then can they start loading the mods. I haven't tested it on dedicated server yet but for listen server host, you can only get server response few seconds after you're spawned using the 'OnPlayerUpdate' event. This is caused by listen server to start ticking right after the host player start the player update tick. For MP clients, I think it might be fine to use 'OnConnected' event but this will need further testing which I am planning to do tonight.

 

2. Mods using the mod loader feature can only be loaded after player update start ticking or when connected if possible. Custom event is needed for mods that initialize data using those two events.

 

3. There might be a need for waiting lobby to ensure that the connected players has fully loaded the mods.

 

4. I tried sending read-only loading order array through server and client command but it's not working since metatable are not transmitted together with the array. So, there's a need to apply custom metatable after the client receive the data.

 

5. After going through intensive testing, it's possible to make clients join without having version mismatch issue. This require modders cooperation though. They'll need to make multiple workshop pages.

- Server Mod: You don't need to enable the mod, just copy mod script files for server and clients to server folder. Server will let clients to download those client files when they joined the server.

- Client Mod: Contain some other files such as textures, item definitions and etc. These files usually don't get updated that frequent. It's fine to enable it in workshop mod list.

 

6. Need UI for download progress.

 

There you go. Sorry for the rough report though.

 

I'll take a look at it later.

 

Hey Zeros,

 

thanks for your report.

This will surely save me a lot of work as well once I'm ready to start my mod loader mod. Just a few thoughts on that:

 

To 1:

Yea, I also think the OnConnected event should be the way to go for it.

 

To 3:

Shouldn't the server be able to handle asynchronous file requests by itself?

There is a queue functionality for joining crowded servers. Maybe you can re-use/derive some functions of it.

Or do you mean some kind of timeout function, which ensures that the joining process stops until all files have been downloaded?

If so, maybe you can hook into the original function for downloading files from the workshop?

 

 

Best regards,

stuck

[Zeros]

1 hour ago, stuck1a said:

thanks for your report.

This will surely save me a lot of work as well once I'm ready to start my mod loader mod. Just a few thoughts on that:

No problem. And, your input is highly appreciated since there might be other solutions to current issues I encountered.

 

1 hour ago, stuck1a said:

 

To 1:

Yea, I also think the OnConnected event should be the way to go for it.

I haven't test it yet since I had to streamline some processes involved with the mod loading. My mods became awfully messy after implementing the trial MP mod loader. This is delaying the test.

 

1 hour ago, stuck1a said:

To 3:

Shouldn't the server be able to handle asynchronous file requests by itself?

There is a queue functionality for joining crowded servers. Maybe you can re-use/derive some functions of it.

Or do you mean some kind of timeout function, which ensures that the joining process stops until all files have been downloaded?

If so, maybe you can hook into the original function for downloading files from the workshop?

I'm using server / client command events to transfer the files. I have no idea if there's other available file transfer method(s) but, the method I'm using only handle file request from clients after the server game logic starts ticking. This creates an issue of players needing a window to download / load mods without ruining the progress. Therefore, yes. I need some sort of timeout function. That's why I considered waiting lobby as an option since I was running out of idea on how to do it. Thanks for the heads up, I will look for the queue functionality later and see if it's possible. Otherwise, I can try radical approach I recently came up with such as overriding the player spawn process through character creation mod and create the window from there.

 

As for the suggestion to hook the workshop download process, it might be possible if there's a way for clients to download the mods without waiting for the server game logic to start ticking. After giving some consideration, there's a need to look at how the server sandbox settings gets transmitted to the clients when connecting. There might be another file transfer method available.

 

Thanks stuck. I almost forgot to check for other file transfer method.

 

Edited August 23, 2022 by Zeros

[Zeros]

I went through the source again. The most legitimate way to transfer files is to use the server and client commands. You can try look at gameserver, gameclient and packettypes for more details. To be honest, I might have skipped some important things while scanning for related functions.

 

If you need better transfer method, an external application might be required for clients to download the mods. Otherwise, try sending suggestion to developers to consider second option for downloading mods which is the game server.

 

Anyway, since there's no other option left.. I can only try override player spawn now to create the timeout window.