sapphireserver / sapphire

A Final Fantasy XIV 4.0+ Server Emulator written in C++

Sapphire - FINAL FANTASY XIV Server Emulator

Sapphire is a FINAL FANTASY XIV Server Emulator currently in development. This branch is specifically for FFXIV version 3.3.

It is a research project to learn how retail servers work and currently not production code; at this time it is insecure (use throwaway passwords for accounts) and you should expect a lot of things unimplemented or broken.

Contributions are always welcome.

Instructions, guides and tips

We provide a wiki with all sorts of information such as details, how to get started, contributing, command lists and development resources in our wiki.

Sapphire Wiki

Client Version

This branch currently targets FFXIV 3.3.

Game: 2016.07.05.0000.0001 EX1: 2016.05.21.0000.0001

A full list of patch names can be seen in the wiki/Patch list.

We are not allowed to distribute the FFXIV client or any related content. Please do not ask for links or binaries related to FFXIV or Square Enix copyrighted content.

Dependencies + Compiling

Sapphire requires the following software:

Name	Windows	Linux
CMake 3.0.2+ and C++17 capable compiler	Visual Studio 2022	gcc 7 and g++ 7 or newer
MySQL Server 5.7	MariaDB 10.7 (x64)	MySQL server from your distribution's package manager

Please check the wiki/Getting Started for detailed installation/build instructions for your OS.

Links

• Sapphire on YouTube

Final Fantasy XIV © 2010-2023 SQUARE ENIX CO., LTD. All Rights Reserved. We are not affiliated with SQUARE ENIX CO., LTD. in any way.

sapphire's Issues

Load GM rank per player

Currently a players GM rank is hardcoded, this should instead be moved to a DB field and loaded accordingly.

Investigate static analysis tools

Just a checklist issue to keep track of what works, what doesn't, and what needs input from others or more experimenting. Goal is to get static analysis and code style checks into CI, ideally on hosted platform but seems most don't support C++ projects.

• clang-check (LLVM)
• clang-format (LLVM)
• clang-tidy (LLVM)
• cppcheck (Google)
• infer (Facebook)
• oclint

4.2 checklists

Server Packets

its just the exd stuff thats broken
i only changed ipcs.h :(((
      PlayerSpawn                = 0x015C, // updated 4.2
      NpcSpawn                   = 0x015D, // updated 4.2
      HateList                   = 0x0165, // updated 4.2
      UpdateClassInfo            = 0x0169, // updated 4.2
      InitUI                     = 0x016B, // updated 4.2
      PlayerStats                = 0x016C, // updated 4.2
      PlayerClassInfo            = 0x016F, // updated 4.2
      ActorControl142            = 0x0142, // updated 4.2
      ActorControl143            = 0x0143, // updated 4.2
      ActorControl144            = 0x0144, // updated 4.2
      Mount                      = 0x01CD, // updated 4.2
      WeatherChange              = 0x01DD, // updated 4.2
      PlayerTitleList            = 0x01EB, // updated 4.2
      Discovery                  = 0x01EC, // updated 4.2
probably correct™(edited)

0FF0 seems like its ServerNotice
00C7 is sending chat message

0xFF blacklist
0xE7 friend's list
0x145 unchanged

Movement is 0x15E

if 0x15d is npc spawn it has 8 extra bytes added to it now

Client Packets

Pos

ClientPos 118 -> 138
ClientPosInstance 157 -> 177

Mount/Orchestrion implementation

Mounts are already present in a basic state, but some things have to still get added:

• Unlocking mounts
• Mount unlock item actions
• Correctly handle mount availability in unlock(index 17) and via player state flags
• Server side checks

Orchestrion needs many of the same things, with slight differences in unlocking.
A few of these issues have been already addressed by me in #191 which isn't yet merged and will need some rework.

Setting Searchmessage is broken

Seems to always be "2". I have not investigated further yet.

Wrong cast time info sent when multiple players are in a zone;

Having multiple players in a zone can screw the cast times on the client, often sending 2000s+ for a cast to complete.

On the server side, the cast is completed in the expected time. However, client keeps on "casting" the already finished action, but unable to interrupt the cast due to it being mostly server-side.

Stateflags are also not cleared on the server-side cast finish, so the player gets stuck with a casting animation/UI.

Occasional crashes when around spawned mobs

This likely is because of incomplete information being set in the spawn packet.

The way we currently store and load mobs should not be concidered final at all though and will potentially be moved to a more script driven approach.

Nontheless, this bug needs to be fixed one way or another.

List of discrepancy between retails

Actor

TODO

Player

• Owner id of player is zero (while retail points to INVALID_GAME_OBJ)
• Actor level stays at zero (this bug again)

TODO

how to contact?

Hi @SapphireMordred is there a way for us to talk about something outside GitHub?

Discord:
Icehunter#5476

Thanks!

Build system refactoring

The current CMake build system is kinda complicated, prone to errors (Boost in VS 2017 on Windows, zlib on Mac/Linux, the recent MySQL path issue...) plus the external dependencies and libs are a bit painful. I'm making this issue to address the known issues plus whatever we find along the way to simplify the build system.

The major proposal is to update CMake minimum version to 3.1, although CI uses 3.5 as a minimum anyway (Ubuntu Trusty), and use the pre-packaged CMake modules for things like Boost, Doxygen, GTest, zlib, or whatever else we currently use or want to use. Additionally, the way external projects are pulled in and built is kinda complex and requires manual work which CMake should be able to manage, for example updating submodules. I'm not sure how that works for tarball builds though.

• CMake minimum version 3.1
• Use CMake's packaged FindBoost
• Add missing explicit dependencies (pthread and zlib - dl is a macro)
• Add CMake packaged FindZlib
• Add CMake packaged FindThread
• Use ExternalProject for SapphireLibs
• Investigate separating SapphireLibs external and internal dependencies
• Add VS2017 generators to CMakeSettings.json
• Add CI for VS2017 targets
• Rename targets ( Server_Zone -> sapphire_zone, etc.... )

Chai script path is hardcoded

With a hardcoded path, the binaries aren't portable. Add a configuration option for the script path and default it to be relative to binary location or via a macro for the project path in CMake.

When merging script update branch shapphirelibs doesn't pull correct tly

when merging script branch it doesn't pull the latest shapphirelibs and you have to manually download the missing files

Social implementation

Group

Here's a basic, "mvp" group system.
It's very generic as far as grouping goes in online games (for parties, rooms etc).

Basic objectives

Implement a system in which a Group, controlled by a GroupManager, satisfies the basics of what we need for all social grouping in the game.

This includes properties such as a list of members, the group leader etc as well as helper functions that can be further overloaded to tailer each group's specifics requirements (such as logging specific errors, handling limits for each group on invite, extended permission abilities ala FCs and LSs).

Error handling

Things such as not being able to invite someone to a specific group if they're already in that group is a given, but extra conditions will show up as more features get added (blacklist etc).

LogMessages found in game data could be helpful for covering all bases, but there may be some unused strings.

Future goals

The system will need to be extended so it supports advanced features. Here's a short list of them:

• Blacklist
• Combat modifiers (limit break, exp etc)
• Companions, pets
• Content finder helpers
• Party finder
• Cross world parties
• Contacts (recently partied members)
• Search tools
• Others

Class/job changes when equipping items

Player class should change automagically when equipping items that are restricted to another class that is currently unlocked

Moogle Post/Mail Implementation

Putting mail implementation details here so it's not lost in conversation

database structure

CREATE TABLE `mooglemail` (
  `letter_id` int(11) NOT NULL DEFAULT '0',
  `mail_body` varchar(200) COLLATE latin1_general_ci NOT NULL,
  `mail_author` varchar(30) COLLATE latin1_general_ci NOT NULL,
  `gil` int(11) NULL DEFAULT '0',
  `attachment_0` int(11) NULL DEFAULT '0',
  `attachment_1` int(11) NULL DEFAULT '0',
  `attachment_2` int(11) NULL DEFAULT '0',
  `attachment_3` int(11) NULL DEFAULT '0',
  `attachment_4` int(11) NULL DEFAULT '0',
  `UPDATE_DATE` DATETIME NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`letter_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;

CREATE TABLE `mooglemailattachment` (
  `attachment_id` int(11) NOT NULL AUTO_INCREMENT,
  `item_id` int(11) NOT NULL,
  `quantity` int(11) NOT NULL,
  `catalog_id` int(11) NOT NULL,
  `UPDATE_DATE` DATETIME NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`attachment_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;

opcodes

CD = MailDeleteRequest
CE = SendMoogleMailLetter
DE = ReqMoogleMailList
DF = ReqMoogleMailLetter
D1 = MailRequestRewardDelivery
E0 = MailDeliveredNotification

Structures

MailDeleteRequest

todo

SendMoogleMailLetter

struct _mail {
    uint unknown_00[10];

    struct _item {
        uint itemId;
        uint quantity;
        uint catalogId;
    } item[5];
    
    uint unknown_64;

    uint gil;
    uint unknown_6C;
    char message[200];
} mail;

ReqMoogleMailList

todo

ReqMoogleMailLetter

todo

MailRequestRewardDelivery

todo

MailDeliveredNotification

struct _mail_notification {
    byte unknown_00[36];
    short unreadCount;
    short flags; // I think...
} notification;

Please add inventory verification

I mistyped //gm item 20956 to //gm item 30956. And a "naught" (just like nothing is there) is added to my inventory and occupied one of my inventory slot. When I want to remove it, I found it's impossible because it's "naught". So I decided to remove the DB record via SQL. I typed in the SQL, run it, no error, no warning. But nothing happened. Then, when I rearrange my items, the "naught" is recorded again in the DB. It seems like no inventory item verification is implemented on the server. Server will record client's message as-is.

Repro:

1. //gm item 30000
2. Delete data (set to 0) related to it in charaiteminventory
3. Move anything (item X) from one slot (slot A) to the slot belong to item 30000 (slot B)
4. Check the charaiteminventory table. What should be removed now appeared again. DB recorded that item 30000 at slot A, and item X at slot B

If level 30+ Game will set you back to 0 on login

Seems to either be a problem with the GM lvl, GM exp command or to do with levels or db and needs to be looked into. Also After about 2 login attempts or zoning the game seems to crash and will keep crashing you on login

Actor movement

• Further map out the packet required for displaying actor movement
• Properly display walking and sprinting animation to other players
• Properly display strafing / backpaddleing to other players
• Figure out how animation speed effects all this

Materia implementation

Improving of loading and saving of player data in server_zone

I am currently not happy with the way player loading and saving is handled it feels hackish in places.

I am not currently sure of how to improve this, but i am open for suggestions.

Gearset functionality

[add more information at some point]

Implementation of /tell chat

Groundwork for this has already been done.

Whats missing is actual handling of the client packet and various lookup helpers for tell receipients along with proper error messages in case a user in unavailable or bound by duty.

Properly track opening sequence state

Currently the opening sequence ( state inside the opening event ) is not tracked properly when logging out during the event.

Idea is to save a flag with the player that indicates what state needs to be restored. ( uint8_t OpeningSequence )

Sapphire Bugs (You know you love em ;)

NOTE: These Are General Things I have Noticed In My Time Playing On The Sapphire Server

Bugs that have been resolved will have a check mark in the box next to them!

Levelup/Exp Bugs

• Leveling Up Past Level 1 Causes The Red Cross To Be Displayed Next To EXP Bar

Command Bugs

• GM Icon Command (Only Does Offline or Unstable Connection Status)

Player/HUD Bugs

• When Attuning Hud Disappears And Only Reappears When Canceling The Event Or Finishing Attuning
• Player HP Does Not Regen Even After Death
• MP Regens at 1 MP (Not Sure If Bug Or Not)
• AFK Status Not Resetting After Put On User From Being AFK

Quest/NPC Bugs

• Archer Quests Seems Needs To Be Done (Does Not Continue Event And Just Loops)
• Mother Miounne After Accepting Close To Home Calls On A Script/Actor Unrelated

Mobs/Actions

• Causes Crash Sometimes After Killing About 5 Mobs When Getting EXP
• Striking Dummy's Are Under The Ground
• All Race Mobs (example. Lalafell) Use The Same Human Model With No Clothes
• Actions Tend To Be Used 2 Times When Pressed Once (Noticed With Lancer True Thrust)
• On Mob Respawn Enemy's Have No HP

More Might Be Coming Soon...

HateList regression after 4.1

HateList seems to be broken after 4.1 hit. Probably a structural change? ID seems to match

To replicate just whack a mob. It won't show up on the aggro list (and mob name color will not change to red)

Possible script/zone-related zone crash

It seems that something is triggering an assert break, possibly related to having 0 scripts loaded in the server.

While running the server with no scripts isn't optimal it's worth looking at.

Emote issues

There are a couple issues with emotes.
Could be looked into, and shouldn't be difficult

• Unchecking Display log message and using an emote only hides the message from selfclient, but still displays message to others.
• Persistent emotes have issues. They need to be stored in the player's class and then be set properly on the appropriate packets (spawn, etc) as well as should be handled when they're cancelled.
• Identify and handle emotes which need weapons drawn previously to being executed to show correctly to other players.

[ZONE] query failed Data too long for column 'HowTo'

[00:37:27][error]       UPDATE sapphire.charadetail SET  HowTo = UNHEX('0C1800000000000000000000000000000000000000000000000000000000000000'),  UPDATE_DATE = NOW() WHERE CharacterId = 2097153;
[00:37:31][debug] [2097153] Handling packet : EventHandler( 011F )
[00:37:33][debug] [2097153] Handling packet : ActionHandler( 0108 )
[00:37:33][debug] [2097153] Incoming action: 0134
param1: 000000000000001D
param2: 00000000
param3: 0000000000000000
[00:37:42][debug] [2097153] Handling packet : EventHandlerReturn( 0128 )
[00:37:49][debug] [2097153] Handling packet : ActionHandler( 0108 )
[00:37:49][debug] [2097153] Incoming action: 0133
param1: 0000000000000050
param2: 00000000
param3: 0000000000000000
[00:37:49][debug] [2097153] Handling packet : EventHandlerReturn( 0128 )
[00:37:49][error] Database: query failed Data too long for column 'HowTo' at row 1
[00:37:49][error]       UPDATE sapphire.charadetail SET  HowTo = UNHEX('0C1800000000000000000100000000000000000000000000000000000000000000'),  QuestCompleteFlags = UNHEX('0000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'),  QuestTracking = UNHEX('FFFFFFFFFFFFFFFFFFFF'),  UPDATE_DATE = NOW() WHERE CharacterId = 2097153;

Glamour implementation

• Store relevant data in db
• Dispelling glamours
• Applying glamours

Inventory improvements

Add missing operations:

• Stacking
• Splitting

Misc:

• Fix max stacks for merging/adding items, currently the max stack size is hardcoded to 999 and isn't read from exd
• Add support for quest item handin
• Remove circular dependency on player
• Items added via //gm item have incorrect data - not sure if fixed

Item trading implementation

Statuseffects Implementation

StatusEffects

The following is a requirement discussion of what i feel is needed from statuseffects.
This is not an implementation guide.

Ways of applying statuseffects include but are not limited to spells, skills, items, area triggers, FC bonuses, certain player conditions ( new player ex bonus ), etc.
This means that statuseffects need to be an autark system.

Conditions Of Livetime Of Effects.

• Timers ( local and FC timers alike, although company actions can be removed prematurly )
• Area ( entering or leaving an area might add/remove an effect )
• Exp received for bonus scrolls ( that means that certain effects need a way to store additional info )
• ...

Persistency Of Effects

Effects need to be persistant through zoning and relogging ( along with any additional information ).
Company actions are not stored with the player but with the FC information

Implications

When zoning and relogging effects have to be reapplied ( although without actually doing the "addStatusEffect" animations )

Complexity Of Statuseffects

The complexity of effects varies greatly. From basic effects such as a sprint to "hot potato" effects like "allagan rot"
A system has to be implemented capable of eanbling scripting of all of them.

Application Of Statuseffects

The way in which an effect is applied varies from effect to effect.
Some effects will overwrite others, some effects will stack, some effects will coexist with identical ones already on the target.
A way needs to be found to ither set those rules manually or ( hopefully ) find that information in the dats.

Script Interface

Statuseffects must be scriptable to allow for easy implementation.
Here is an example of what the script interface might look like.

// lets assume poison has id 12

// possible flags
// REMOVABLE, UNIQUE, PLAYER_UNIQUE, STACKABLE...

class StatusPoison12
{
   def StatusPoison12()
   {
      var flags = REMOVABLE | UNIQUE;
      var baseDuration = 10000;
      var statusID = 12;
   }

   // function being called when the effect is created
   def createEffect( sourceActor, TargetActor )
   {
      // create effect and apply to actor
   }

   def onTick( sourceActor, targetActor, effectObj )
   {
      // whatever happens on tick
   }

   def onRemove( sourceActor, targetActor, effectObj )
   {
      // probably only need target here
   }

   def onExpire( sourceActor, targetActor, effectObj ) 
   {
      // timer of effect actually ran out
   }

   // need to have a way to add additional conditions here, onProximityToWhatever etc...

}

// this can stay global, actual instance is create in onApply or a compareable function
global StatusPoison12Template = StatusPoison12();


class SomeSkillThatAppliesPoison<ID>
{

   def SomeSkillThatAppliesPoison<ID>()
   {
   }

   def onExecute( sourceActor, targetActor )
   {
      StatusPoison12Template.applyTo( sourceActor, targetActor );
   }

}

Dead status not recovering properly

In some situations a dead player doesn't have their status recovered properly, such as:

When logging out while dead, logging back in has you back up and alive.
When another player enters a zone with an already dead player in it, the player is shown as alive to them (but not to the dead player).

Implement Content Finder

Tasks

Sapphire

• Add ContentFinder class (server-wide global variable)
• Implement player queue
• Generate an instance as needed? (Discussion needed)
• Restore previous zone position where left off when he leaves from the duty

Exd

• Figure out how alliance raid is handled in ContentMemberType

Misc.

• Document CF packets

• More TBD

Considerations/Notes

• It has a potential to share some codes with quest instance. (Which, by the way, is required to implement it around around lv5-6 MSQ)
• Opcode 0x78 is more like a union. We have to figure out how to handle this elegantly.

Action EXD Research

Currently we're lacking information on some of the fields for action and action-related tables for the exd.
As an example, the action table has over 60 fields, with over 25 of them being as of now unknown.

Having more of these fields mapped out would greatly help in the future for a new action-handling system for Sapphire, as writing even a prototype implementation and/or a basic design is difficult at this point due to too many unknown things about these values, and how we can correlate data.

Status Effect research may also be needed for a basic action implementation, but in the future.

Proper calculation of stats and attributes

Calculation of stats and attributes need to be improved and in many cases newly implemented.

• Calculation of base stats
• Calculation of HP / MP
• Damage formulars
• GCD calculation
• Spell/Skillspeed
  ... likely more to come ...

casting something on a nonexistant npc crashes zone

1. spawn an npc locally (needs an invalid actorid)
2. cast something with the npc as the target
3. watch the world zone burn

Too many actors triggering client player despawning itself

Basically if there are too many actors, the client despawns itself, effectively locking the player out with no way back in without recreating the session.

This has likely to do with the internal actor table used for ranged actors.

Fix compiler warnings

We currently have too many compiler warnings of various types.
Those need to be fixed.

boost::asio server_zone crash

Happens when sending something in chat and or handling the packet : DiscoveryHandler While outbound/external connection

Call Stack: https://pastebin.com/h5JSVHcQ
Dump: server_zone.zip

Adding of additional data fields for player

We have alot of data for a player that is still not loaded / saved.

Getters / setters / loading and saving aswell as actually sending of data needs to be implemented for various fields.
Some of those include

• Titles
• Mounts
• Minions
• Completed Duties
• Blacklists
• Friendlists
  ... and many more ...
  Expand as you see fit.

4.3 stuff

4.2 -> 4.3

(Unconfirmed)
15C -> 172
15D -> 173
15E -> 174
15F -> 175
160 -> 176
161 -> 177
162 -> 178 (ActorCast)

[goes on until 178 -> 18E]

RP Tag ID Set Wrong

The RP Tag gets set to 47 instead of 22 making it Invis on the player.
This issue is similar to the one in issue #20 to do with GM Commands. Difference is with RP it shows in the Party Member Tab as Role Playing unlike using the GM Command which sets both as offline.

Zoning occasionally taking a very long time

This does not seem to apply when using @set tele XXX but only when using normal teleport / zonelines.
This may be because of some information not being sent properly, needs investigation ( was introduced in 4.0 )

Mobs don't despawn properly

Killed mobs don't despawn properly in certain cases, and reappear with a depleted health bar after a player goes back in range of them. This seems to happen with mobs spawned manually and with our current db solution.

JSON Payload for Lobby issues

Currently, having two or more characters on a single account bring issues to the client.
This is caused by the recent patch's changes to the JSON structure for the lobby.

As for people looking for a workaround, you can create two accounts for now and make a character in each of them

Suggested by @AcedArmy - would be great if someone looked into getting all the details showing properly on the lobby as well.

Client version control

Allow configuration of the supported client version.

Something like an optional value in the XML config (eg. "4.0"). Block sessions/warn somewhere if client doesn't have a supported version and gracefully error the client out.
Client sends its version (apparently can check it from lobby server) but I believe that currently we're not using it.

Inventory

• Add support for stacks

• Merging/Splitting of stacks

• Selecting correct stack when adding stacked items

• General item stuff

• Add support for spiritbinding

• Add item duration

• Add item repair

• Add item coloring

• Add item glamouring

[NOT VERY IMPORTANT] Can't seem to build..... Idk

Like the title mentions, this isn't important and you don't need to bother answering.

Now that that's outta the way, I've installed everything but can't seem to build the solution.

I'm brand new to this, so I have no idea what I'm doing.

If this annoys you or is not worth answering due to it's complexity and/or the fact that I couldn't use the damn thing even if i built it plz feel free to close this issue. Ty for your time.

Item durability

• Decrease item durability
• Add vendor repairing
• Add self repairing
• Remove from stats when durability is 0