eve-ning / reamberpy

Vertical Scrolling Rhythm Game Mapping Toolbox

Reamber (Py)

📘 Wiki & Getting Started

pip install reamber

---

VSRG Toolbox for data extraction, manipulation & analysis.

Features

• Game Support: osu!mania, StepMania, BMS and partially O2Jam files.
• Algorithms: Map IO, Conversion, Map Image Generation, Pattern Extraction
• Data Architecture: Pandas DataFrame Integration

This is built on pandas DataFrame, thus, if you need more control, you can retrieve the underlying DataFrame via the .df property on many reamber classes.

Status

This is in alpha, names and references will change without notice. We highly recommended to fix version in requirements.

For Developers, By Developers

A growing amount of osu!mania players are interested in programming. The best way to learn is to relate it to something that you're familiar with.

That's why I'm making a library that allows you to tamper with your favorite games, and learn on the way.

Migrating to >v0.1.1 Releases

Migrating to >0.1.1 means spending time updating a lot of names.

Major changes in 0.1.1 include many differences in naming. Only update it if you don't plan mind spending time refactoring.

reamberpy's Issues

Changing rate ignores column 3, outputs everything on column 3 to column 4

Describe the bug
Changing rates to some maps sometime doesn't work and shifts column 3 to 4

Files used (if any)

https://osu.ppy.sh/beatmapsets/1595304#mania/3258332 (stella 1.3)

Play Field Note Snap Coloring

TimingMap sometimes does not snap correctly

Code usable but, sometimes shifts the 1/1 off a bit ?

    def write_string(self) -> List[str]:
        """ Write an exportable String List to be passed to SMMapset for writing.
        :return: Exportable String List
        """

        header = [
            f"//------{self.chart_type}[{self.difficulty_val} {self.difficulty}]------",
            "#NOTES:",
            f"     {self.chart_type}:",
            f"     {self.description}:",
            f"     {self.difficulty}:",
            f"     {self.difficulty_val}:",
            "     " + ",".join(map(str, self.groove_radar)) + ":"
        ]

        def beats(offsets):
            return [SMBpm.mbs_to_beat(*i) for i in tm.snaps(offsets, divisions=(1,2,3,4,5,6,7,8,9,10,12,16,32,48),
                                                            transpose=True)]

        tm = self.bpms.to_timing_map()
        bpm_beats = beats(self.bpms.offset)

        # -------- We will grab all required notes here --------
        # List[Tuple[Beat, Column], Char]]

        notes = \
            [beats(
                [*self.hits.offset,
                 *self.holds.head_offset,
                 *self.holds.tail_offset,
                 *self.rolls.head_offset,
                 *self.rolls.tail_offset,
                 *self.fakes.offset,
                 *self.keysounds.offset,
                 *self.lifts.offset,
                 *self.mines.offset]
            ),
             [*self.hits.column,
              *self.holds.column,
              *self.holds.column,
              *self.rolls.column,
              *self.rolls.column,
              *self.fakes.column,
              *self.keysounds.column,
              *self.lifts.column,
              *self.mines.column],
             [*[SMConst.HIT_STRING] * len(self.hits),
              *[SMConst.HOLD_STRING_HEAD] * len(self.holds),
              *[SMConst.HOLD_STRING_TAIL] * len(self.holds),
              *[SMConst.ROLL_STRING_HEAD] * len(self.rolls),
              *[SMConst.ROLL_STRING_TAIL] * len(self.rolls),
              *[SMConst.FAKE_STRING] * len(self.fakes),
              *[SMConst.KEYSOUND_STRING] * len(self.keysounds),
              *[SMConst.LIFT_STRING] * len(self.lifts),
              *[SMConst.MINE_STRING] * len(self.mines)]
            ]
        notes = pd.DataFrame(list(zip(*notes)), columns=['beat', 'column', 'char'])
        notes['measure'] = notes.beat // DEFAULT_BEAT_PER_MEASURE
        notes['den'] = [i.denominator for i in notes.beat]
        notes['num'] = [i.numerator for i in notes.beat]

        notes.den *= DEFAULT_BEAT_PER_MEASURE
        notes.num %= notes.den

        notes_gb = notes.groupby('measure')
        out = []
        for _, g in notes_gb:
            den_max = g.den.max()
            lines = [['0' for i in range(SMMapChartTypes.get_keys(self.chart_type))] for j in range(den_max)]
            g.num *= den_max / g.den
            g.num = g.num.astype(int)
            g.column = g.column.astype(int)
            for _, note in g.iterrows():
                lines[note.num][note.column] = note.char
            out.append("\n".join(["".join(line) for line in lines]))

        return header + ["\n,\n".join(out)] + [";\n\n"]

OsuHold column dtype unexpectedly described as float

Describe the bug

Add BMS Support

Include as many relevant documents/sites as possible to help me decipher the file format
https://fileformats.fandom.com/wiki/Be-Music_Script

https://hitkey.nekokan.dyndns.info/cmds.htm

Add BMS Metronome Support

Currently I'm leaving out the metronome parsing because it's hard to implement. Will implement if there's a high demand.

Scale Full LN gaps w.r.t. current BPM

As the title suggests, in my opinion having it set to a dataframe would make the algorithm more flexible for chart generation and make FLN only a small piece of a chart. At the moment, the full LN algorithm uses a dataframe anyway for its operation, so I think it should be just a couple of lines to change it. Reference:

reamberPy/reamber/algorithms/generate/full_ln.py

Line 24 in e03d945

df = m.stack((HitList, HoldList))._stacked

main focus of this would be making custom algorithms based on full LN possible eg: having the gap set to a 1/4 of the current BPM of the chart, thing that is not possible (without having a lot of writes and splitting the chart for every bpm change) for multi-bpm charts.

Simple SV normalization algorithm

Describe your algorithm in as few sentences as possible
Normalize the SV of a chart based on multiple BPM points. This is a common algorithm that could be useful for people who map multi bpm charts or for NSV implementation(?)

As per the BMS Timing Fix, other games are to migrate to use TimingMap for consistency

Types to migrate

• SM
• O2JAM

MD using redundant lines

MD will unexpectedly use the full val length instead of just the diff

Writing to BMS may not work as expected

Describe the bug

Files used (if any)

take.bms

The frequent bpm changes causes incorrect offsets for some reason, can crosscheck with ibmsc.

BMS BPM Change Fails on Submeasure Change

Describe the bug

if

xxx03: 0011

it will parse as

xxx03:00
xxx03:11

the 00 will break it.

Files used (if any)

RISH - Take

Conversion algorithm to have __call__ as alternative

class ConversionName:
  def __call__(...): ...
  def convert(...): ...

for ConversionName() as converting api

Map.readFile to be made static

Currently, we use the syntax

m = OsuMap()
m.readFile("path.osu")

it could be

m = OsuMap.readFile("path.osu")

if it's static

Overlapped notes when changing rate

Describe the bug
Changing rate of some maps doesn't fully work and gives overlapped notes as a result

Files used (if any)

https://osu.ppy.sh/beatmapsets/1599290#mania/3266313 changing rate to 1.15

reamber can not convert not only 0% LN quaver map and also 0% Rice map.

When I use it to convert quaver maps, I got this error:

  File "main.py", line 58, in <module>
    main()
  File "main.py", line 53, in main
    convertMap(os.path.join(input_dir, file), output_dir)
  File "main.py", line 16, in convertMap
    quaver_map = QuaMap.read_file(map_file_path)
  File "E:\Anaconda\envs\OSUconvert\lib\site-packages\reamber\quaver\QuaMap.py", line 69, in read_file
    return QuaMap.read(file)
  File "E:\Anaconda\envs\OSUconvert\lib\site-packages\reamber\quaver\QuaMap.py", line 55, in read
    m._read_notes(file.pop('HitObjects'))
  File "E:\Anaconda\envs\OSUconvert\lib\site-packages\reamber\quaver\QuaMap.py", line 110, in _read_notes
    self.holds = QuaHoldList.from_yaml(holds)
  File "E:\Anaconda\envs\OSUconvert\lib\site-packages\reamber\quaver\lists\notes\QuaHoldList.py", line 19, in from_yaml  
    df['EndTime'] -= df['StartTime']
  File "E:\Anaconda\envs\OSUconvert\lib\site-packages\pandas\core\frame.py", line 3761, in __getitem__
    indexer = self.columns.get_loc(key)
  File "E:\Anaconda\envs\OSUconvert\lib\site-packages\pandas\core\indexes\range.py", line 349, in get_loc
    raise KeyError(key)
KeyError: 'EndTime'

After analyzing it, I found the QuaMap._read_notes() method doesn't detect whether the hold and hit is empty. So, it throws the KeyError.

The example error map is this map, it is a Rice map without any LN.

`parse_replays_error` ignores column order

❗ Fixed in #121, please pip install pointing the development 0.1.9.

pip install git+https://github.com/Eve-ning/[email protected]

Description

Currently, parse_replays_error ignores columns, and does minimum distance matching regardless of it.
This resulted in a heavy accuracy overestimate.

Osu Map no load after stacking

problem with casting, will have to check many other formats

0.0x Quaver SVs generate KeyError

Whenever a value is 0, the value is skipped in the yaml generation. If a SV of 0.0x is used, then no Multiplier is given.

Traceback (most recent call last):
  File "qua2osu.py", line 287, in <module>
    main()
  File "qua2osu.py", line 273, in main
    convertQp(file, outputPath, options)
  File "qua2osu.py", line 177, in convertQp
    qua = QuaMap.readFile(filePath)
  File "C:\Users\Ice\AppData\Local\Programs\Python\Python37-32\lib\site-packages\reamber\quaver\QuaMap.py", line 48, in readFile
    self._readSVs(file.pop('SliderVelocities'))
  File "C:\Users\Ice\AppData\Local\Programs\Python\Python37-32\lib\site-packages\reamber\quaver\QuaMap.py", line 78, in _readSVs
    self.svs.append(QuaSv(offset=sv['StartTime'], multiplier=sv['Multiplier']))
KeyError: 'Multiplier'

Simple fix would be to use multiplier=sv.get('Multiplier', 0).

FLN Algorithm

Describe your algorithm in as few sentences as possible

If possible, add your algorithm pseudo or python code

Add O2Jam OJM Support

Documents
O2Jam Wiki
OJN
OJN Notes
OJM

About's Converts

Main issue is that I have no files that exist in o2jam and osu!, which makes it impossible for me to verify if a map is converted correctly.

If anyone has a good source for o2jam and osu maps that are the same, it'll be a good contribution.

SMMapSet should avoid stops as core element

Currently it's possible to mutate stops as with props, if it's possible remove it without breaking everything

Handle unicode characters for Tags, Title, Author etc

Describe the bug
Currently letters like é (LATIN SMALL LETTER E WITH ACUTE) aren't correctly handled and will output \x?? instead of the correct character.

Scroll Speed assumes that BPM doesn't reset SV to 1.0

Describe the bug

Currently, we assumed that scroll_speed BPM takes on the SV of the previous SV line, which is not true.
in osu! behavior, the BPM actually forces the SV to be reset to 1.0, so implicitly the SV is reset.

For example

Normal on 200 BPM

Offset	SV	BPM	Scroll Speed
0	2	100	1.0
1		200	1.0
2	2	200	2.0

reamber incorrectly assumes that offset 1 has double scroll speed.

reamber.osu.OsuMap.OsuMap.rate not working as expected

Describe the bug

When using reamber.osu.OsuMap.OsuMap.rate the output is a corrupted file. Example:

RATE = 1.5
FILE_PATH = "E:\Games\osu!\Songs\992889 4K Chordjack Challenge\Various Artists - 4K Chordjack Challenge (Onta_Bekasi) [BRIGHT - 1 year 2 months 20 days].osu"
m = OsuMap.readFile(FILE_PATH)
m.rate(RATE, True)
m.version = m.version + " {}x".format(RATE)
m.writeFile(file_path[:-4] + " {}.osu".format(RATE))

(the audio file has been externally processed with ffmpeg)
output: https://del.dog/tydeamucyg.txt

replacing last 4 lines from:

192,192,26464,128,0,35453:0:0:0:0:
448,192,26632,128,0,35677:0:0:0:0:
192,192,26800,128,0,35901:0:0:0:0:
64,192,26968,128,0,36125:0:0:0:0:

to:

192,192,26464,1,0,0:0:0:0:
448,192,26632,1,0,0:0:0:0:
192,192,26800,1,0,0:0:0:0:
64,192,26968,1,0,0:0:0:0:

"fixes" it, even though the result won't be an uprated file.

Files used (if any)

https://osu.ppy.sh/beatmapsets/992889#mania/2076376

Quaver Metadata Missing

Describe the bug

IceDynamix: reamberpy quaver metadata format is missing a few things

reamberPy/reamber/quaver/QuaMapMeta.py

Line 32 in 6d19c84

class QuaMapMeta:

BannerFile
Genre
BPMDoesNotAffectScrollVelocity 
InitialScrollVelocity 
HasScratchKey

[5:58 AM] IceDynamix: source: https://github.com/Quaver/Quaver.API/blob/master/Quaver.API/Maps/Qua.cs

Files used (if any)

Missing MD Documentation

requests module not installed on pip install

Hitsound copier page broken

.qua read crashes if no song preview is set

Any map in Quaver that has never had a song preview set before will make reamber raise a KeyError. Similar errors may occur with the other file formats or keys as well.

Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import reamber
>>> qua = reamber.quaver.QuaMap.readFile("test.qua") 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...\env\lib\site-packages\reamber\quaver\QuaMap.py", line 49, in readFile
    self._readMetadata(file)
  File "...\env\lib\site-packages\reamber\quaver\QuaMapMeta.py", line 55, in _readMetadata
    self.songPreviewTime    = d['SongPreviewTime']
KeyError: 'SongPreviewTime'

Simplest solution would be to implement following for that section

self.songPreviewTime = d.get('SongPreviewTime', self.songPreviewTime)

Note Generation

Implement MinaCalc and Star Rating system

Reference:

1. Minacalc:

• https://github.com/etternagame/etterna/tree/develop/src/Etterna/MinaCalc

• https://github.com/kangalioo/Minacalc/tree/b9706e9a1add23127cbdb7c6bc5b3891840c4a41

2. Star Rating:

• https://github.com/ppy/osu/tree/master/osu.Game.Rulesets.Mania/Difficulty

Fix SM conversion offset via referencing Quaver

Typo in the docs

Examples:
https://eve-ning.github.io/reamberPy/reamber/algorithms/convert.html#to-quaver

quaMapSet = SMToOsu.convert(sm) should be quaMapSet = SMToQua.convert(sm)

I don't know if there are others, but might be worth checking.

Add collapsing API for SV ALGO B

Describe your algorithm in as few sentences as possible

if line gaps are smaller than a threshold, show one line only, should be able to handle multiple.

Split tests into Unit & Integration Tests for Parellelism

osu! Storyboard implementation

Describe your algorithm in as few sentences as possible

If possible, add your algorithm pseudo or python code

Quaver map start times should be handled as int, not as float

Describe the bug
Quaver StartTime are currently handled as float, doing so the parser will incorrectly import the map.

To fix it just handle StartTime as int, or at least this is what I believe.

[3:33:59] - RUNTIME - ERROR: YamlDotNet.Core.YamlException: (Line: 28, Col: 14, Idx: 638) - (Line: 28, Col: 31, Idx: 655): Exception during deserialization
 ---> System.FormatException: Input string was not in a correct format.
   at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type)
   at System.Number.ParseUInt64(ReadOnlySpan`1 value, NumberStyles styles, NumberFormatInfo info)
   at System.UInt64.Parse(String s)
   at YamlDotNet.Serialization.NodeDeserializers.ScalarNodeDeserializer.DeserializeIntegerHelper(TypeCode typeCode, String value)
   at YamlDotNet.Serialization.NodeDeserializers.ScalarNodeDeserializer.YamlDotNet.Serialization.INodeDeserializer.Deserialize(IParser parser, Type expectedType, Func`3 nestedObjectDeserializer, Object& value)
   at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   --- End of inner exception stack trace ---
   at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   at YamlDotNet.Serialization.ValueDeserializers.AliasValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.<>c__DisplayClass3_0.<DeserializeValue>b__0(IParser r, Type t)
   at YamlDotNet.Serialization.NodeDeserializers.ObjectNodeDeserializer.YamlDotNet.Serialization.INodeDeserializer.Deserialize(IParser parser, Type expectedType, Func`3 nestedObjectDeserializer, Object& value)
   at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   at YamlDotNet.Serialization.ValueDeserializers.AliasValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.<>c__DisplayClass3_0.<DeserializeValue>b__0(IParser r, Type t)
   at YamlDotNet.Serialization.NodeDeserializers.CollectionNodeDeserializer.DeserializeHelper(Type tItem, IParser parser, Func`3 nestedObjectDeserializer, IList result, Boolean canUpdate)
   at YamlDotNet.Serialization.NodeDeserializers.CollectionNodeDeserializer.YamlDotNet.Serialization.INodeDeserializer.Deserialize(IParser parser, Type expectedType, Func`3 nestedObjectDeserializer, Object& value)
   at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   at YamlDotNet.Serialization.ValueDeserializers.AliasValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.<>c__DisplayClass3_0.<DeserializeValue>b__0(IParser r, Type t)
   at YamlDotNet.Serialization.NodeDeserializers.ObjectNodeDeserializer.YamlDotNet.Serialization.INodeDeserializer.Deserialize(IParser parser, Type expectedType, Func`3 nestedObjectDeserializer, Object& value)
   at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   at YamlDotNet.Serialization.ValueDeserializers.AliasValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
   at YamlDotNet.Serialization.Deserializer.Deserialize(IParser parser, Type type)
   at YamlDotNet.Serialization.Deserializer.Deserialize(TextReader input, Type type)
   at Quaver.API.Maps.Qua.Parse(String path, Boolean checkValidity)
   at Quaver.Shared.Database.Maps.MapsetImporter.InsertAndUpdateSelectedMap(String extractDirectory)

Fix Osu <-> SM Offset issue

Currently, it's very off, will run a few tests to try to fix this