balisujohn / tortoise.cpp

A ggml (C++) re-implementation of tortoise-tts. Under construction and seeking contributors.

tortoise.cpp: GGML implementation of tortoise-tts, under construction

Implementation status:

Tokenization seems to work, but doesn't exactly match the tokenization tortoise-tts performs, needs work.

Voice latent is hardcoded for now.

Text embedding/ text position embedding reconstruction complete.

Mel embedding reconstruction complete.

Autoregressive model(gpt-2) reconstruction complete.

Contrastive Language-Voice Pretrained Transformer reconstruction pending.

Diffusion model reconstruction pending.

Compiling

For now, cuda only. To compile:

mkdir build
cd build
cmake ..
make

This is tested with Ubuntu 22.04 and cuda 12.0 and a 1070ti

Running

You will need to place ggml-model.bin and ggml-diffusion-model.bin in the models directory to run tortoise.cpp. You can generate the model yourself following the instructions in this tortoise-tts reverse engineering fork here https://github.com/balisujohn/tortoise-reverse-engineering, or download it here https://huggingface.co/balisujohn/tortoise-ggml.

From the build directory, run:

./bin/tortoise

Contributing

If you want to contribute, please make an issue stating what you want to work on. I'll make a discord to manage contributors if there is a lot of interest. You can email me questions at <mylastname>u<myfirstname>@gmail.com. I am happy to help get people get started with contributing!

I am also making available a fork of tortoise-tts which has my reverse engineering annotations, and also the export script for the autoregressive model.

License

This is released with an MIT License.

MIT License

Copyright (c) 2024 John Balis

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Derived from tortoise-tts and ggml.

tortoise-tts:

Apache 2.0 License James Betker https://github.com/neonbjb/tortoise-tts/blob/main/LICENSE

GGML

MIT License

Copyright (c) 2022 Georgi Gerganov

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

tortoise.cpp's Issues

AMD GPUs

Hi,
I saw this project currently only supports Cuda. I was wondering if it might be possible to use HIPIFY to make it work on AMD GPUs. Do you know if this would be possible?
Thank you!

why not vulkan

that way you dont waste time implementing cpu,cuda,amd,metal,intel

Add test for gpt-2 module to prevent regressions

This project is a fork of ggml so it inherits ggml's tests. They are unfortunately somewhat out of date with ggml's current test battery.

You can run the tests with the following commands

cd build
make test

This task is to add a test to make sure that the gpt-2 model outputs the current output for the current seed and input phrase, to prevent regressions from being introduced from subsequent edits to the forward pass.

The input is the tokens:

 std::vector<gpt_vocab::id> tokens = ::parse_tokens_from_string("255,42,2,97,60,49,2,63,48,61,2,26,163,2,149,2,68,161,33,2,42,2,33,77,78,16,32,2,58,2,42,2,50,18,125,9,2,80,43,32,2,127,2,106,29,57,33,159,7,2,55,2,204,32,9,2,16,31,54,54,2,26,213,61,2,60,2,136,26,29,242,2,51,2,22,20,95,46,2,42,2,36,54,18,2,46,63,31,137,192,2,73,2,26,245,2,26,50,2,19,46,18,9,2,0,0", ',');

and there should be a batch of 4 resulting token sequences that look like

[8192, 8, 964, 2961, 896, 978, 4542, 4134, 887, 878, 7051, 1143, 2608, 670, 3014, 99, 795, 1265, 2228, 6370, 611, 329, 5529, 1813, 616, 4153, 157, 917, 2932, 1919, 2967, 2256, 1815, 37, 6542, 2681, 725, 7135, 6893, 1901, 226, 4182, 3484, 2231, 4191, 6344, 2099, 7415, 6893, 2528, 1265, 3546, 4137, 380, 2770, 5560, 1548, 3020, 2362, 7474, 4062, 6921, 6268, 3225, 6693, 5047, 5805, 1613, 2081, 83, 45, 8, 7406, 1134, 4769, 1702, 2813, 70, 5576, 989, 1730, 184, 369, 4387, 3690, 2617, 500, 2978, 5902, 5478, 2797, 2825, 1209, 315, 5033, 1580, 20, 45, 7406, 6560, 6842, 4518, 804, 6288, 6041, 6490, 7677, 3894, 1227, 2489, 936, 1613, 3415, 2214, 716, 1580, 20, 83, 45, 7005, 964, 4074, 184, 2662, 2731, 670, 6433, 4767, 1488, 2296, 4411, 5759, 1265, 3455, 826, 8142, 7014, 6893, 4084, 8158, 1369, 555, 1539, 2636, 1460, 2930, 2893, 1527, 4136, 4961, 3888, 6009, 2127, 1322, 2770, 2775, 2154, 3879, 4447, 579, 1715, 1875, 7229, 2075, 3996, 1940, 4259, 211, 1660, 7886, 224, 226, 1272, 3734, 1298, 4669, 2371, 2735, 779, 19, 4336, 6964, 5406, 1364, 4062, 3633, 1539, 2976, 186, 4277, 1170, 2286, 2797, 1516, 388, 937, 1425, 3323, 3300, 3894, 8193, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 45, 45, 248, 8193]
[8192, 45, 913, 2261, 448, 4244, 3570, 1451, 799, 2080, 5823, 3188, 37, 2919, 372, 8075, 1265, 4879, 7456, 6893, 1554, 6079, 1328, 3213, 6563, 22, 4036, 485, 2075, 323, 2926, 4989, 907, 2015, 2218, 1032, 4039, 6893, 7992, 6871, 489, 3085, 6456, 5961, 6084, 837, 849, 7815, 1539, 4718, 3415, 4627, 3447, 991, 3020, 7382, 7856, 6659, 3635, 6490, 2290, 3225, 3894, 5047, 5805, 152, 5090, 20, 45, 299, 897, 3419, 2524, 2465, 2954, 6680, 8136, 2099, 2978, 2439, 3214, 3387, 4257, 1277, 6, 184, 7251, 5446, 4090, 5, 7403, 3689, 1580, 7005, 212, 7474, 1692, 5116, 5216, 5874, 5226, 7677, 4754, 1227, 2324, 2354, 134, 432, 1580, 20, 45, 83, 45, 7005, 3005, 4074, 1212, 887, 1588, 3610, 1244, 564, 8051, 456, 3958, 2882, 944, 5960, 1325, 2114, 3099, 4275, 6893, 3250, 6784, 918, 1558, 3546, 37, 6407, 4120, 3385, 2546, 2724, 6599, 3947, 1728, 2247, 4475, 951, 3224, 1174, 16, 758, 4277, 5623, 4221, 3291, 1746, 1215, 2075, 364, 1729, 8100, 224, 2429, 1326, 3628, 134, 283, 7410, 1499, 5199, 3487, 5114, 99, 134, 372, 6106, 5860, 7079, 1364, 4837, 3633, 1820, 611, 3327, 6772, 6848, 645, 2797, 283, 303, 2098, 787, 630, 3300, 3905, 8193, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 45, 45, 248, 8193]
[8192, 20, 11, 7571, 2416, 2524, 2437, 6011, 616, 1451, 4889, 560, 2156, 456, 826, 4967, 2369, 2228, 4080, 3606, 1067, 5191, 4098, 5056, 438, 1363, 442, 2490, 1624, 6672, 1508, 837, 7921, 3761, 7229, 7394, 37, 964, 523, 1539, 2495, 5375, 8072, 942, 6084, 849, 824, 7136, 4264, 2724, 329, 4928, 6284, 350, 5046, 4120, 7020, 6046, 3741, 7607, 4099, 3894, 3905, 297, 5311, 11, 1580, 20, 83, 45, 7406, 282, 3646, 360, 1105, 1813, 7258, 2932, 492, 107, 7464, 4264, 6661, 1181, 3302, 1730, 5902, 4275, 1640, 3616, 1227, 913, 6996, 3848, 4685, 3412, 5961, 1692, 6142, 4940, 748, 6204, 5329, 6490, 6131, 2290, 4099, 3225, 4724, 913, 670, 1356, 1580, 20, 45, 83, 45, 7005, 1906, 4939, 295, 1601, 5606, 1982, 1940, 5336, 1244, 412, 2015, 2061, 1600, 826, 5293, 8022, 6893, 5276, 7965, 4025, 2150, 4992, 5617, 7282, 4120, 3385, 4391, 1753, 4629, 4518, 2247, 2282, 3635, 1670, 3522, 3670, 4374, 6945, 2797, 378, 4091, 1814, 6490, 8100, 22, 2168, 630, 2478, 119, 913, 6308, 1499, 4837, 6834, 2737, 361, 7403, 6570, 8082, 627, 7020, 3633, 2247, 647, 897, 4431, 6511, 7020, 2797, 3223, 388, 283, 5497, 2351, 4389, 297, 3725, 8193, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 45, 45, 248, 8193]
[8192, 8, 913, 2012, 1105, 2465, 6011, 416, 2662, 3613, 4335, 4726, 754, 1488, 372, 7642, 2020, 2853, 4626, 611, 1554, 2279, 1326, 8184, 456, 5288, 3322, 2490, 460, 6534, 2793, 1349, 7039, 2681, 6536, 6111, 5952, 7261, 1763, 3607, 5954, 6084, 1995, 1539, 6893, 6523, 2362, 1355, 826, 7910, 5859, 458, 1460, 6834, 7224, 5085, 4277, 1907, 7020, 2797, 5502, 4389, 6693, 5311, 7005, 45, 83, 45, 299, 2564, 5837, 634, 4432, 5078, 6887, 1236, 3780, 1442, 2704, 5444, 3455, 1833, 4962, 6637, 2672, 6652, 6852, 7356, 7322, 2801, 5952, 1580, 20, 7005, 1059, 5085, 1692, 5625, 7596, 7224, 6490, 6842, 5864, 4099, 5695, 4099, 2112, 134, 2214, 20, 45, 83, 45, 7005, 824, 1833, 1442, 1541, 2219, 3610, 2682, 4767, 6008, 456, 5058, 2301, 944, 5418, 2117, 4928, 6022, 837, 2484, 6233, 2662, 3309, 4264, 1290, 1988, 2131, 555, 2976, 3921, 2138, 3968, 5331, 4810, 558, 1167, 2190, 364, 7576, 2895, 984, 3283, 1174, 2530, 2075, 1601, 7578, 265, 6458, 4547, 670, 1953, 647, 7010, 1351, 4062, 2957, 3828, 913, 19, 2460, 6596, 7079, 3895, 4837, 3633, 4039, 861, 23, 4277, 6511, 4526, 2797, 3123, 1126, 937, 787, 900, 7020, 3300, 4169, 8193, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 45, 45, 248, 8193]

and the autoregressive latent tensor should match the values it currently produces(the first and last 3 values are printed out here)

NAME:
cur
TYPE
0
SHAPE:
1024
500
4
1
DATA:
0.703938
0.134067
0.577487
-0.891981
-0.0644716
0.370529

So this test will require saving the 1024 * 500 * 4 tensor to disk and uploading it to the repo.

Feel free to ask any questions on this issue or in the discord!

Weights

Note that the pre loaded ggml-model.bin saved in this repository does not have all the tensors currently loaded by the program, since the version with a lot of tensors is too big to put in version control. Feel free to make an issue asking for the conversion script if you want me to hurry up in posting it.

Hi,
Thank you so much for working on this! I was wondering if you might be able to post the conversion scripts, and also to post the full weights on Hugging Face.
Thank you!

Make the tokenizer match the tortoise-tts Tokenizer exactly

If people are interested in contributing to tortoise.cpp, a great first task would be getting the tokenizer to always match the tokenization tortoise-tts uses. The tokenizer I'm using in tortoise.cpp seems to be able to load the tokenizer vocab but the regex had issues with some of the special chars which I bandaided at least for spaces, but more perplexingly, the tortoise-tts tokenizer isn't greedy with respect to always choosing the longest possible next token, while the default tokenizer I copied from ggml gpt-2 seems to be greedy. So the task would be studying the tokenizer tortoise-tts uses, and modifying the tokenizer in tortoise,cpp to exactly match this behavior. Please reply here to claim this task, and feel free to ask any questions here if you need help getting set up with development.

Part of solving this would be coming up with a number of test phrases to make sure the tortoise.cpp tokenizer finds matching tokenizations for all of them.

Implement the CLVP module

CLVP is a non-essential component of tortoise-tts for filtering for latents that will will yield good generation quality if given to the diffusion model.

This task is to export CLVP from tortoise-tts to a ggml format, then load it tortoise.cpp and reconstruct the forward pass.

This will be relatively involved; as in the Discord if you want to attempt this and I'll go into more detail.

Profile the gpt-2 module and the tortoise-tts gpt-2 module and try to improve the gpt-2 module's performance

This task is blocked by the gpt-2 forward pass test being added since this could introduce regressions. #5

The task is as follows:

measure the runtime of the autoregressive model all the way from inputs to the full sequence of tokens and last layer latents being generated (as checked by the test in #5 ), and the time taken for the corresponding batch of 4 token sequences and final layer latents in tortoise-tts.

Then, try improving the efficiency of the tortoise.cpp forward pass. Some suggestions are as follows:

try removing seemingly redundant ops
change ops to in place where possible

Feel free to ask questions here or in the discord.