Hardening a live economy

The problem

On a roleplay server, items are the economy. A bug that loses an item costs a player hours of play. A bug that duplicates one devalues everyone else’s. And the server can’t go down while you fix it.

As the only developer, I was also the security team. This is the side of the work players never see: finding the ways people duplicated, teleported or forged things, and closing them on a live server.

How I approach it

• Read every network handler as if the client is hostile. Anything a player’s game sends can be changed, so the server rebuilds or re-checks it against its own data.
• Reserve before you wait. Anything asynchronous, like a database write or an inventory add, is a gap a fast player can squeeze into, so the server takes what it needs first and gives it back if the work fails.
• Check permissions before reading the payload, not after.
• Ship fixes the live server can take. When an exploit is in use, the fix can’t wait for the next big update.

Shipment duplication

Live

Taking an item from a crate started an asynchronous inventory add without reserving the item first. Fast repeated requests could all pass the “is there one left?” check before any of them finished, and each walked away with an item.

I rewrote the take and drop back end. The count is now decremented and the crate locked before the asynchronous work starts, distance and amounts are checked on the server, and the item is refunded if it doesn’t fit. While I was there I fixed an admin inventory reset that could leave slots occupied by items that no longer existed.

netstream.Hook("shpUse", function(client, uniqueID, drop)
    local entity = client.nutShipment
    -- ... per-player and per-crate rate limits omitted

    if (not IsValid(entity) or entity.transactionLockout) then return end   -- one take at a time per crate
    local itemTable = nut.item.list[uniqueID]
    if (not itemTable) then return end

    -- Distance and amount are checked on the server.
    if (entity:GetPos():Distance(client:GetPos()) > 128) then return end
    local amount = entity.items[uniqueID]
    if (not isnumber(amount) or amount <= 0) then return end

    -- Reserve before waiting: take the item off the count and lock the crate
    -- *before* the asynchronous add, so repeated requests can't all pass the check.
    entity.items[uniqueID] = amount - 1
    entity.transactionLockout = true

    local function onError(err)
        if (IsValid(client)) then client:notifyLocalized(tostring(err)) end
        if (not IsValid(entity)) then return end

        -- Didn't fit in the inventory: give the item back to the crate.
        if (err == "noFit") then
            entity.items[uniqueID] = entity.items[uniqueID] + 1
        end
        entity.transactionLockout = nil
    end

    local function onSuccess(res)
        hook.Run("OnShipmentItemSpawned", client, res, entity)
        if (entity:getItemCount() < 1) then
            entity:Remove()   -- empty crates break; the lock stays until they're gone
        else
            entity.transactionLockout = nil
        end
    end

    local add = drop
        and nut.item.spawn(uniqueID, entity:GetPos() + Vector(0, 0, 16))
        or client:getChar():getInv():add(uniqueID, itemTable.maxQuantity, {})

    add:next(function(res)
        if (res.error) then return onError(res.error) end
        onSuccess(res)
    end, onError)
end)

Cheating a live PvP event

Live

Networked variables used by the Target PvP event revealed enough to the client for players to cheat it. I built the fix so it could go onto the live server straight away.

Forged evidence and records

Live

• A fingerprint card’s hidden data could be edited from the client. Edits now only accept known fields, capped in length, and are stored on the server (see forensics).
• Appeals only accept record IDs the server has already shown that player, so changing an ID in the request does nothing (see the justice system).
• When characters move between factions, the server checks the faction ID against its own data instead of accepting the one the client sent.
• The lockpicking minigame has server-side anti-cheat, so a modified client can’t just skip it.

Trunk teleporting

Built, in testing

Donators can spawn more than one vehicle. With two trunks open at once, players could move items between them from anywhere on the map. Only one spawned vehicle’s trunk can be open at a time now. This ships with the shared trunk migration.

And many smaller ones

• A context-menu action that removed weapon recoil.
• A way past the bank limit.
• Switching characters to get around the car limit.
• Duplication and restriction bypasses through gun bags, now isolated inventories.
• Moving and jumping from the console while tied up.
• Disconnecting while handcuffed to drop the cuffs, now kept on the character until they’re removed.
• Staff shooting from noclip and abusing the physics gun, plus logging when staff delete items from storage.
• Inventory syncs large enough to overflow network buffers, now encoded and compressed.

Keeping production safe

The fixes only matter if testing them can’t break the live server. I pushed for a production-like test server that can never touch production data, and guarded the development-only admin tools so they can’t run on production.