Seriously, it's really easy! Take a look:
-- Server
local data = file.Read( "huge_data_file.json" )
express.Broadcast( "stored_data", { data } )
-- Client
express.Receive( "stored_data", function( data )
file.Write( "stored_data.json", data[1] )
end )
Compared to doing it yourself...
-- Server
-- This is just an example!
-- It doesn't handle errors or clients joining, and it doesn't support multiple streams
util.AddNetworkString( "myaddon_datachunks" )
local buffer = ""
local function broadcastChunk()
if #buffer == 0 then return end
local chunkSize, isLast = math.min( 63000, #buffer ), false
buffer = string.sub( buffer, chunkSize + 1 )
if #pending <= chunkSize then
buffer, isLast = "", true
end
net.Start( "myaddon_datachunks" )
net.WriteUInt( chunkSize, 16 )
net.WriteData( string.sub( pending, 1, chunkSize ), chunkSize )
net.WriteBool( isLast )
net.Broadcast()
end
function BroadcastFile( filePath )
local fileData = file.Read( filePath, "DATA" )
buffer = util.Compress( fileData )
end
local interval = engine.TickInterval() * 8
timer.Create( "MyAddon_DataSender", interval, 0, broadcastChunk )
BroadcastFile( "huge_data_file.json" )
-- Client
local buffer = ""
net.Receive( "myaddon_datachunks", function()
buffer = buffer .. net.ReadData( net.ReadUInt( 16 ) )
if not net.ReadBool() then return end
local datas = util.Decompress( buffer )
processData( datas )
end )
In this example, huge_data_file.json
could be in excess of 100mb (soon) 25mb post-compression without Express even breaking a sweat.
The client would receive the contents of the file as fast as their internet connection can carry it.
Instead of using Garry's Mod's throttled (<1mb/s!) and already-polluted networking system, Express uses unthrottled HTTP requests to transmit data between the client and server.
Doing it this way comes with a number of practical benefits:
- ๐ฌ These messages don't run on the main thread, meaning it won't block networking/physics/lua
- ๐ช A dramatic increase to maximum message size (~100mb, compared to the
net
library's <64kb limit) - ๐๏ธ Big improvements to speed in many circumstances
- ๐ค It's simple! You don't have to worry about serializing, compressing, and splitting your table up. Just send the table!
Express works by storing the data you send on Cloudflare's Edge servers. Using Cloudflare workers, KV, and D1, Express can cheaply serve millions of requests and store hundreds of gigabytes per month. Cloudflare's Edge servers offer extremely low-latency requests and data access to every corner of the globe.
By default, Express uses gmod.express, the public and free API provided by CFC Servers, but anyone can easily host their own! Check out the Express Service README for more information.
-- Server
-- `data` can be a table of (nearly) any size, and may contain (almost) any values!
-- the recipient will get it exactly like you sent it
local data = ents.GetAll()
express.Broadcast( "all_ents", data )
-- Client
express.Receive( "all_ents", function( data )
print( "Got " .. #data .. " ents!" )
end )
-- Client
local data = ents.GetAll()
express.Send( "all_ents", data )
-- Server
-- Note that .Receive has `ply` before `data` when called from server
express.Receive( "all_ents", function( ply, data )
print( "Got " .. #data .. " ents from " .. ply:Nick() )
end )
-- Server
local meshData = prop:GetPhysicsObject():GetMesh()
local data = { data = data, entIndex = prop:EntIndex() }
-- Will be called after the player successfully downloads the data
local confirmCallback = function( ply )
receivedMesh[ply] = true
end
express.Send( "prop_mesh", data, { ply1, ply2, ply3 }, confirmCallback )
-- Client
express.Receive( "prop_mesh", function( data )
entMeshes[data.entIndex] = data.data
end )
This function is very similar to net.Receive
. It attaches a callback function to a given message name.
string name
- The name of the message. Think of this just like the name given to
net.Receive
- This parameter is case-insensitive, it will be
string.lower
'd
- The name of the message. Think of this just like the name given to
function callback
Set up a serverside receiver for the "balls"
message:
express.Receive( "balls", function( ply, data )
myTable.playpin = data
if not IsValid( ply ) then return end
ply:ChatPrint( "Thanks for the balls!" )
end )
Very much like express.Receive
, except this callback runs before the data
has actually been downloaded from the Express API.
string name
- The name of the message. Think of this just like the name given to
net.Receive
- This parameter is case-insensitive, it will be
string.lower
'd
- The name of the message. Think of this just like the name given to
function callback
- The function to call just before downloading the data.
- On
CLIENT, this callback receives:
string name
: The name of the messagestring id
: The ID of the download (used to retrieve the data from the API)int size
: The size (in bytes) of the databoolean needsProof
: A boolean indicating whether or not the sender has requested proof-of-download
- On
SERVER, this callback receives:
string name
: The name of the messagePlayer ply
: The player that is sending the datastring id
: The ID of the download (used to retrieve the data from the API)int size
: The size (in bytes) of the databoolean needsProof
: A boolean indicating whether or not the sender has requested proof-of-download
boolean
:- Return
false
to halt the transaction. The data will not be downloaded, and the regular receiver callback will not be called.
- Return
Adds a normal message receiver and a pre-download receiver to prevent the server from downloading too much data:
express.Receive( "preferences", function( ply, data )
ply.preferences = data
end )
express.ReceivePreDl( "preferences", function( name, ply, _, size, _ )
local maxSize = maxMessageSizes[name]
if size <= maxSize then return end
print( ply, "tried to send a", size, "byte", name, "message! Rejecting!" )
return false
end )
Removes the callback associated with the given message name. Much like net.Receive( message, nil )
.
string name
- The name of the message. Think of this just like the name given to
net.Receive
- This parameter is case-insensitive, it will be
string.lower
'd
- The name of the message. Think of this just like the name given to
Create a new Receiver when the module is enabled, and remove the receiver when it's disabled
local function enable()
express.Receive( "example", processData )
end
local function disable()
express.ClearReceiver( "example" )
end
The CLIENT version of
express.Send
. Sends an arbitrary table of data to the server, and runs the given callback when the server has downloaded the data.
string name
- The name of the message. Think of this just like the name given to
net.Receive
- This parameter is case-insensitive, it will be
string.lower
'd
- The name of the message. Think of this just like the name given to
table data
- The table to send
- This table can be of any size, in any order, with nearly any data type. The only exception you might care about is
Color
objects not being fully supported (WIP).
function onProof() = nil
- If provided, the server will send a token of proof after downloading the data, which will then call this callback
- This callback takes no parameters
Sends a table of queued actions (perhaps from a UI) and then allows the client to proceed when the server confirms it was received. A timer is created to handle the case the server doesn't respond for some reason.
local queuedActions = {
{ "remove_ban", steamID1 },
{ "add_ban", steamID2, 60 },
{ "change_rank", steamID3, "developer" }
}
myPanel:StartSpinner()
myPanel:SetInteractable( false )
express.Send( "bulk_admin_actions", queuedActions, function()
myPanel:StopSpinner()
myPanel:SetInteractable( true )
timer.Remove( "bulk_actions_timeout" )
end )
timer.Create( "bulk_actions_timeout", 5, 1, function()
myPanel:SendError( "The server didn't respond!" )
myPanel:StopSpinner()
myPanel:SetInteractable( true )
end )
The SERVER version of
express.Send
. Sends an arbitrary table of data to the recipient(s), and runs the given callback when the server has downloaded the data.
string name
- The name of the message. Think of this just like the name given to
net.Receive
- This parameter is case-insensitive, it will be
string.lower
'd
- The name of the message. Think of this just like the name given to
table data
- The table to send
- This table can be of any size, in any order, with nearly any data type. The only exception you might care about is
Color
objects not being fully supported (WIP).
table/Player recipient
- If given a table, it will be treated as a table of valid Players
- If given a single Player, it will send only to that Player
function onProof( Player ply ) = nil
- If provided, the client(s) will send a token of proof after downloading the data, which will then call this callback
- This callback takes one parameter:
Player ply
: The player who provided the proof
Sends a table of all players' current packet loss to a single player. Note that this example does not use the optional onProof
callback.
local loss = {}
for _, ply in ipairs( player.GetAll() ) do
loss[ply] = ply:PacketLoss()
end
express.Send( "current_packet_loss", loss, targetPly )
Operates exactly like express.Send
, except it sends a message to all players.
string name
- The name of the message. Think of this just like the name given to
net.Receive
- This parameter is case-insensitive, it will be
string.lower
'd
- The name of the message. Think of this just like the name given to
table data
- The table to send
- This table can be of any size, in any order, with nearly any data type. The only exception you might care about is
Color
objects not being fully supported (WIP).
function onProof( Player ply ) = nil
- If provided, each player will send a token of proof after downloading the data, which will then call this callback
- This callback takes a single parameter:
Player ply
: The player who provided the proof
Sends the updated RP rules to all players
RP.UpdateRules( newRules )
RP.Rules = newRules
express.Broadcast( "rp_rules", newRules )
end
This hook runs when all Express code has loaded. All express
methods are available. Runs exactly once on both realms.
This is a good time to make your Receivers (express.Receive
).
Creates the Express Receivers when Express is available
-- cl_init.lua
hook.Add( "ExpressLoaded", "MyAddon_SetupExpress", function()
express.Receive( "MyAddon_ObjectData", function( data )
processData( data )
end )
end )
Called when ply
creates a new receiver for message
(and, by extension, is ready for both net
and express
messages)
Once this hook is called, it is guaranteed to be safe to express.Send
to the player.
Player ply
- The player that registered a new Express Receiver
string message
- The name of the message that a Receiver was registered for
- (Note: This will be
string.lower
'd before calling this hook, so expect it to always be lowercase)
Sends an initial dataset to the client as soon as they're ready
-- sv_init.lua
hook.Add( "ExpressPlayerReceiver", "MyAddon_InitData", function( ply, message )
if message ~= "myaddon_initdata" then return end
express.Send( "myaddon_initdata", MyAddon.CurrentData, ply )
end )
-- cl_init.lua
hook.Add( "ExpressLoaded", "MyAddon_SetupExpress", function()
express.Receive( "MyAddon_InitData", function( data )
processData( data )
end )
end )
We tested Express' performance against two other options:
- Manual Chunking:
- This is a bare-minimum example script that serializes, compresses, and splits the data up across as few net messages as possible. (This is typically what people do in smaller addons.)
- Source
- NetStream:
- This library is very popular. It's the go-to choice for sending large chunks of data. It's currently used by Starfall, PAC3, AdvDupe2, etc.
- Source
Test Setup
Our findings are based on a series of tests where we generated data sets filled with random elements across a range of data types. (string
, int
, float
, bool
, Vector
, Angle
, Color
, Entity
, table
)
We sent this data using each of the options, one at a time.
These test were performed on a moderately-specced laptop. The server was a dedicated base-branch server run in WSL2. The client was base-branch clean-install run on Windows.
For each test, we collected two key metrics:
- Duration: The total time (in seconds) it took to complete each test. This includes compression, serialization, sending, and acknowledgement.
- Message Count: The number of net messages sent during the transfer. Fewer is usually better.
References:
Detailed Test Results
Test 1 (74.75 KB)
:
Summary: This data can fit in only two net messages. In this situation, Express loses out to just sending net messages (by almost a full second).
Data Size | Compressed Size |
---|---|
194.97 KB | 74.75 KB |
Method | Duration (s) | Messages Sent |
---|---|---|
Manual Chunking | 1.265 | 2 |
NetStream | 2.273 | 11 |
Express | 1.909 | 1 |
Test 2 (374.78 KB)
:
Summary: Requiring at least six net messages when sent normally, Express sends the data about 3x faster.
Data Size | Compressed Size |
---|---|
988.2 KB | 374.78 KB |
Method | Duration (s) | Messages Sent |
---|---|---|
Manual Chunking | 6.160 | 6 |
NetStream | 10.303 | 51 |
Express | 2.151 | 1 |
Test 3 (1.5 MB)
:
Summary: After passing the "1 megabyte" mark, Express' advantages bein really shining through, beating the next fastest option by 21 seconds (8x faster!)
Data Size | Compressed Size |
---|---|
3.97 MB | 1.5 MB |
Method | Duration (s) | Messages Sent |
---|---|---|
Manual Chunking | 24.325 | 24 |
NetStream | 40.849 | 200 |
Express | 2.897 | 1 |
Test 4 (11.22 MB)
:
Summary: With a much larger payload, it becomes abundantly clear how slow and prohibitive the built-in net library can be. Express sends this 11mb payload in under 20 seconds, while the net library is nearing 200 seconds.
Data Size | Compressed Size |
---|---|
29.67 MB | 11.22 MB |
Method | Duration (s) | Messages Sent |
---|---|---|
Manual Chunking | 181.491 | 180 |
NetStream | 304.552 | 1,485 |
Express | 18.993 | 1 |
Test 5 (11.96 KB)
:
Summary: Because this payload only requires a single net mesage, Express falls way behind of the pack in terms of transfer speed.
Data Size | Compressed Size |
---|---|
29.79 KB | 11.96 KB |
Method | Duration (s) | Messages Sent |
---|---|---|
Manual Chunking | 0.306 | 1 |
NetStream | 0.833 | 3 |
Express | 1.333 | 1 |
- Express sends data significantly faster than both Manual Chunking and NetStream when the data size exceeds a certain threshold (Roughly whenever 3 or more net messages would be required).
- Express only sends up to 2 net messages per transfer, no matter the size of the data.
- Despite its impressive performance with large data sizes, Express is less efficient than other methods for smaller data sizes.
- (NetStream is surprisingly slow, regardless of data size)
- These results will depend heavily on networking conditions. For some people, lots of smaller messages may actually perform better than one large Express download.
- Anything that uses the built-in net library (like NetStream) will be more reliable than a library like Express, even if they may be slower overall.
- Express caches sends. This means that if you needed to send a dataset to more than one player, Express would only need to upload the data once, saving a significant amount of time and bandwidth. These savings aren't reflected in this test run.
These tests illustrate how Express can significantly improve data transfer speed and efficiency for large or even intermediate-scale data, but may underperform when handling smaller data sizes.
Understanding the trade-offs of Express can help you determine if it's a good fit for your project.
Here's a clip of me spawning a particularly detailed and Prop2Mesh-heavy ACF-3 dupe (both Prop2Mesh and Adv2 use Netstream to transmit their data).
gmod_cL5uWh9hTu.mp4
A few things to note:
- It took ~20 seconds for the dupe to be transferred to the server via Netstream
- It took an additional ~20 seconds for the Prop2Mesh data to be Netstreamed back to me
- On the netgraph, you can see the
in
andout
metrics (and the associated green horizontal progress bar) that shows Netstream sending each chunk - Netstream only processes one request at a time. This is important, because it means while Adv2 or Prop2Mesh are transmitting data, no other player can use any Netstream-based addon until it completes.
Using some custom backport code, I converted Prop2Mesh and Advanced Duplicator 2 to use Express instead of Netstream. Here's me spawning the same tank in the exact same conditions, but using Express instead:
gmod_5RiCPGLfFA.mp4
The entire process took under 15 seconds - that's over 60% faster! My PC actually lagged for a moment because of how quickly all of the meshes downloaded and were available to render.
Even better? This doesn't block any other player from spawning their dupes! Because this is using Express instead of Netstream, other players can freely spawn their dupes, Prop2Mesh, Starfalls, etc. without being blocked and without blocking others.
I had someone who knew more about Prop2Mesh than me create a highly complex controller. Here are the stats:
Nearly 1M triangles across 162 models! If you've ever worked with meshes before, you'll know those are crazy high numbers.
When spawning this dupe in a stock server with Adv2 and Prop2Mesh, it takes nearly 4 minutes! All the while, blocking other players from using any Netstream-based addon. I can't even upload the video here because it's too big. Hopefully this screenshot is informative enough:
Some metrics:
- It took 1 minute and 50 seconds before the dupe was even spawnable (it had to send the full dupe over to the server first)
- After an additional 3 minutes, the meshes were finally downloaded and rendered
- Again, while this was happening, no other player could use Adv2, Prop2Mesh, or Starfall
With that same backport code, forcing Adv2 and Prop2Mesh to use Express, the entire process takes under 30 seconds! That's almost a 90% speed increase.
gmod_3CairQAogv.mp4
A big thanks to @thelastpenguin for his super fast pON encoder that lets Express quickly serialize almost every GMod object into a compact message.