What is better mod data format?

[Maris]

I'd like to optimize my mod, to make it faster.

So I wonder how the game deals with networked mod data.

 

What is better: true/false or 1/0?

If the data is transfered in JSON format, 1/0 will be better. Won't it?

[Fenris_Wolf]

Mod Data is transferred over the network (and saved to disk) in binary format, not JSON.

Strings are UTF-8, which is variable in size.  It can range from 1-4 bytes in size per character (A single byte for characters in the ASCII range), plus a additional 2 bytes (short) at the start of the string (the number of total bytes in the string)

 

PZ's kahlua engine treats (and converts) all numbers to Doubles (binary64) thus any number will take 64 bits (8 bytes size).

Bool values (true/false) are saved and transmitted as a single byte value.

 

Edited December 31, 2019 by Fenris_Wolf

[Fenris_Wolf]

Expand on my previous explanation  with more concrete data...
Every key/value pair in the data table consumes extra space:

 

1 byte (key type) + 2 bytes (key string length) + ??? key string data (length varies) + 1 byte (value type) + ??? value data

consider the following examples, keeping these facts in mind:

 

* each key/value pair has 2 bytes padding.

* each key length in the examples below is 7 bytes long with a additional +2 bytes (the size of the string)

* not counting the value, each entry in the table is at least 11 bytes in size.

-- +8 bytes (double) for the number value.... 19 bytes per entry
{
    ["number1"] = 1,
    ["number2"] = 10,
    ["number3"] = 100,
    ["number4"] = 1000,
    ["number5"] = 10000,
    ["number6"] = 100000,
    ["number7"] = 1000000
}
-- total: 133 bytes

Each size in the above example is consistent, and the total size is133 bytes.

Now using string values instead of numbers

 

-- same key and padding size (11 bytes per entry)
-- but value size varies using strings:
{
    ["number1"] = "1", -- +2 bytes (length) +1 bytes (string characters) = 14 bytes total
    ["number2"] = "10".  -- +2 bytes +2 bytes (string characters) = 15 bytes total
    ["number3"] = "100", -- +2 bytes +3 bytes (string characters) = 16 bytes total
    ["number4"] = "1000", -- +2 bytes +4 bytes (string characters) = 17 bytes total
    ["number5"] = "10000", -- +2 bytes +5 bytes (string characters) = 18 bytes total
    ["number6"] = "100000", -- +2 bytes +6 bytes (string characters) = 19 bytes total
    ["number7"] = "1000000" -- +2 bytes +7 bytes = 20 bytes
}
-- total: 119 bytes

Notice only the last entry in the table exceds the "19 bytes per entry" used in the first example. The total size is a bit smaller.

Finally, lets compress all these numbers into a single table entry:

 

-- single entry. same key and padding size (11 bytes).
{
    ["numbers"] = "1,10,100,1000,10000,100000,1000000" -- +2 bytes (length) + 34 bytes (data) = 47 bytes total.
}
-- total: 47 bytes

Quite a drastic drop in size compressing the numeric values into a single string.

 

However the value of such knowledge is debatable. Yes it will save on network transmission size (as well as save on disk size and in memory), but at the cost of CPU time doing conversions.
 

Edited December 31, 2019 by Fenris_Wolf

[Maris]

Thanks for the explanation!

 

What about indexed arrays?

{1,2,3,4,5} --Each number is +9 bytes (8+1). But there is no keys.

And what about nil value?

{ 1, nil, 3, nil, nil }
or
{ key = nil, }

 

[Fenris_Wolf]

26 minutes ago, Maris said:

What about indexed arrays?

 

There are keys...the index number itself. Which is saved as a double (8 byes). So in your first example, each entry is 1 + 8 + 1 + 8 for a total of 18 bytes.

{1,2,3,4,5}
-- is the same as:
{[1] = 1, [2] = 2, [3] = 3, [4] = 4, [5] = 5}

18 bytes seems extremely wasteful given if those were strings:

 ["1"] = "1", -- 1 (padding) + 2 (str len) + 1 (str data) + 1 (padding) + 2 (str len) + 1 (str data) = 8 bytes total 

 

Nil values are never saved, or kept in tables at all.

{ key = nil } -- completely removes key from the table

The same applies to indexed versions.

 

Edited December 31, 2019 by Fenris_Wolf