shenmue-mods / shenmuedksharp

Shenmue I & II HD Modding SDK for C#

Shenmue Developer Kit for C#

Shenmue I & II Modding SDK for C#

This library was created to give developers an easy access to all the file formats of Shenmue I & II.

How to contribute:

• Feel free to commit if you want to contribute some fixes or features.
• If you found an bug just create an issue on GitHub.
• For feature requests just create an issue on GitHub.

Usage

Model reading example:

using ShenmueDKSharp;
using ShenmueDKSharp.Files.Models;
	 
public void ReadMT7(string filename)
{
  MT7 mt7 = new MT7(filename);
  foreach(ModelNode node in mt7.GetAllNodes())
  {
    foreach(MeshFace face in node.Faces)
    {
      face.GetFloatArray(node, Vertex.VertexFormat.VertexNormalUV);
    }
  }
}

See the github wiki for more examples.

File Formats

For more informations about the file formats see the wulinshu wiki.

Containers

Name	Read	Write	Description	Notes
AFS	✔️	✔️	Archive
IDX	✔️	🔶	AFS Archive Reference Names	Only IDX0 can be created
PKF	✔️	✔️	Archive (mainly textures for PKS)
PKS (IPAC)	✔️	✔️	Archive
SPR	✔️	✔️	Sprite texture Container
GZ	✔️	✔️	GZip
TAD/TAC	✔️	✔️	d3t TAD/TAC container with hash mapping	Filename coverage based on wulinshu hash database

Textures/Images

Name	Read	Write	Description	Notes
PVRT	🔶	🔶	PowerVR Texture	Not all formats working but enough for MT5 and MT7
DDS	✔️	✔️	DirectDraw_Surface
JPEG	✔️	✔️	JPEG format
BMP	✔️	✔️	Bitmap format
PNG	✔️	✔️	PNG format

PVR color formats

Value	Name	Read	Write	Description
0x00	ARGB1555	✔️	✔️	Format consisting of one bit of alpha value and five bits of RGB values.
0x01	RGB565	✔️	✔️	Format without alpha value and consisting of five bits of RB values and six bits of G value.
0x02	ARGB4444	✔️	✔️	Format consisting of four bits of alpha value and four bits of RGB values.
0x03	YUV422	✔️	✔️	YUV422 format
0x04	BUMP	✔️	✔️	Bump map with positiv only normal vectors (S and R direction angles)
0x05	RGB555	✔️	✔️	for PCX compatible only
0x06	ARGB8888	✔️	✔️	Format consisting of 1 byte of alpha value and 1 byte of RGB values. (Palettize only!)
0x80	DDS_RGB24	✔️	✔️	RGB24 format (DXT1)
0x81	DDS_RGBA32	✔️	✔️	RGBA32 format (DXT3)

PVR data formats

Value	Name	Read	Write	Notes
0x01	SQUARE_TWIDDLED	✔️	✔️
0x02	SQUARE_TWIDDLED_MIPMAP	✔️	✔️
0x03	VECTOR_QUANTIZATION	✔️	✔️
0x04	VECTOR_QUANTIZATION_MIPMAP	✔️	✔️
0x05	PALETTIZE_4BIT	✔️	✔️
0x06	PALETTIZE_4BIT_MIPMAP	✔️	✔️
0x07	PALETTIZE_8BIT	✔️	✔️
0x08	PALETTIZE_8BIT_MIPMAP	✔️	✔️
0x09	RECTANGLE	✔️	✔️
0x0A	RECTANGLE_MIPMAP	❌	❌	Reserved: Can't use.
0x0B	RECTANGLE_STRIDE	✔️	✔️
0x0C	RECTANGLE_STRIDE_MIPMAP	❌	❌	Reserved: Can't use.
0x0D	RECTANGLE_TWIDDLED	✔️	✔️	Should not be supported.
0x0E	BMP	❌	❌	No information.
0x0F	BMP_MIPMAP	❌	❌	No information.
0x10	VECTOR_QUANTIZATION_SMALL	✔️	✔️
0x11	VECTOR_QUANTIZATION_SMALL_MIPMAP	✔️	✔️
0x80	DDS	✔️	✔️	DDS format
0x87	DDS	✔️	✔️	DDS format

Models/Animation

Name	Read	Write	Description	Notes
MT5	🔶	🔶	Model Container	Reading/Writing works but still has some unknown stuff.
MT6	❌	❌	Model Container
MT7	🔶	❌	Model Container	Reading works but missing rig and skin weights and some unknown stuff
MOTN	❌	❌	Motion data (Animation sequences)
OBJ	🔶	🔶	Wavefront OBJ	Very basic OBJ implementation

Audio

Name	Read	Write	Description	Notes
SND	❌	❌	Dreamcast sound file
XWMA	❌	❌	Xbox WMA (XAudio2) file

Subtitles/Text

Name	Read	Write	Description	Notes
SRF	❌	❌	Cinematic subtitles file
FONTDEF	❌	❌	Font definition file
SUB	✔️	✔️	Subtitles file
GLYPHS	❌	❌	Font glyph file
FON	❌	❌	Disk font file

Mapinfo

Name	Read	Write	Description	Notes
CHRD	❌	❌
CHRM	❌	❌
COLS	❌	❌	Collisions (sm1)
DOOR	❌	❌	Door portals?
ECAM	❌	❌
FLDD	❌	❌	Collisions (sm2)
LGHT	❌	❌	Lighting data
MAPR	❌	❌
MAPT	❌	❌
SCEX	❌	❌	Cutscenes and maybe other stuff
SNDP	❌	❌	Sound program
WTHR	❌	❌	Weather data

Other

Name	Read	Write	Description	Notes
ATH	❌	❌	Sequence Data
SRL	❌	❌	Scroll Data
IWD	❌	❌	LCD Table
WDT	❌	❌	Weather Data
UI	❌	❌	UI Json
CHR	❌	❌	Character
MVS	❌	❌	MVS data
DYM	❌	❌	Dynamics Info
CRM	❌	❌	Character Model
CHT	❌	❌	Character Properties
CSV	❌	❌	Comma-separated values
EMU	❌	❌	Emulator file

Used by these projects

• ShenmueHDTools - GUI file unpacker/packer and converter for Shenmue I&II file formats.
• wudecon - CLI file unpacker/converter for Shenmue I&II file formats.

Credits

Contributors:

• LemonHaze - SDK development and testing
• DerPlayer - SDK testing

Starting code:

• SHENTRAD Team - Shenmue I & II DC ground work
• ShenmueHDTools - Project where it all started
• Fishbiter - MT5 and MOTN starting code
• hellwig - MT5 incompleted code
• yazgoo - MT5 and MT7 rough starting point
• nickworonekin - PVR reader/writer base code
• KFreon - DDS reader/writer base code

Other:

• Shenmue-Mods - Modding knowledge database
• Raymonf - Wulinshu TAD hash database

shenmuedksharp's Issues

PVR Bump Conversion

Some PVR textures contain the Bump format which is currently not supported yet.

MT5/MT7 Writer

Currently only reading is supported.

DDS ImageClass is broken

DDS class fails to deliver propper images.

Simple Code example (read dds and write as new file):

DDS oldImage = new DDS(fileNode.FullPath);
oldImage.Write(fileNode.RootPath + "\\Dummy\\" + fileNode.RelativPath);

will deliver binary incompatible output, that is unreadable by dds tools.

Same problem with custom BaseImage object creation (read png into baseclass, convert and write to dds format):

object baseimage = Activator.CreateInstance(typeof(PNG), new object[] {  AppDomain.CurrentDomain.BaseDirectory + "Dummy\\test.png" });
BaseImage entry = (BaseImage)baseimage;
DDS image= new DDS(entry);
image.FormatDetails = new DDSFormats.DDSFormatDetails(image.FormatDetails.Format);
image.AlphaSettings = image.AlphaSettings;
image.MipHandling = image.MipHandling;
image.HasTransparency = image.HasTransparency;
image.Write(fileNode.RootPath + "\\Dummy\\" + fileNode.RelativPath);

Microsoft DDS analyse tool (texdiag.exe) fails also to analyse the created dds files:

texdiag.exe analyze test.DDS
Microsoft (R) DirectX Texture Diagnostic Tool
Copyright (C) Microsoft Corp. All rights reserved.

test.DDS FAILED (80070032)

Input file is in this case a 32bit-A8R8G8B8" DDS file (32kb + header)
returned file is much smaller (6kb + header, compression?)
Same also happens with the Texture Converter.

MT7 wrong facing normals or wrong triangle winding

Some models have wrong facing normals or maybe even wrong triangle winding.

Files:

• 1GAS2A2I.MT7

OBJ Reader

For basic model editing a wavefront OBJ reader is needed.

PVR MipMap support

Only the first mipmap is being read at the moment.

MT7 UV mirrored

Some the models have mirrored UVs which get not read out correctly yet.

Models:

• 5GASALNI.MT7 (the face texture is mirrored)

MT5 missing strip color

Most sonic models are missing their color.
Because they have no vertex color strip, the color must be stored somewhere else.

Models:

• GACRSO1G.MT5

FBX Reader/Writer

For conserving the node structure from MT5/MT7 a node based model format is needed.

ModelOverride: Custom "FBX" format from d3t

Find out how this special "fbx" format is structured and implement a reader/writer for this format.

Other "damaged" models

SCENE\01\MODEL\CHARA\BGA_L.MT5 - https://vgy.me/s5So7f.jpg
SCENE\01\MODEL\CHARA\RFD_L.MT5
SCENE\01\OP02_SHD2.PKS_\MAP02.MAPM - https://vgy.me/DGzRIN.jpg

MT5 EndOfStreamException

In the MT5Mesh constructor there is an EndOfStreamException for some files.

Files:

• GACK6HKG.MT5
• GACK6TSG.MT5
• GACK6TZG.MT5
• GACK6XKG.MT5
• GACRDR1G.MT5
• GACRDR2G.MT5
• GACRGBIG.MT5
• GACRMETG.MT5
• SPBK6BRG.MT5
• SPBK6REG.MT5
• SPBK6YEG.MT5

MT5 UV mapping (mirror, repeat, clamp, fan)

MT5 UV mapping attributes are not read out yet.
Only the 0x0200 and/or 0x0300 strip function may contain these UV attributes.

Example Files:
SCENE\01\JU00_SHUKI.PKS_\BIKK3S2G.CHRM - https://vgy.me/AUpK70.png
SCENE\01\JU00_SHUKI.PKS_\BKGS501G.CHRM - https://vgy.me/KCLE86.png
SCENE\01\JU99\MAP.MT5 - https://vgy.me/SVRXpY.jpg (road uv map flip missing)
SCENE\01\MODEL\CHARA\BIKS504G.MT5 - https://vgy.me/WtbMUr.jpg

Extension finder for container files

Currently the extension finder is not being used in the container files.

DDS writing support

For creating PVR texture files DDS would be recommended for performance reasons so we need DDS writing support.